Data + AI in 2024: Three Problems for Now and for the Future

Although they are closely related, the data and AI communities have historically operated in parallel, each advancing in its own right — but often separately.

As we continue into 2024, this trend has dramatically changed. We're witnessing a convergence that is not only reshaping existing frameworks, but also forging new pathways for technological innovation.

After working in data for almost 10 years — and learning in AI for some time — I want to share my thoughts, mainly around three big problems in data and AI.

Disclaimer : I speak mostly for myself in this blog post. Some places might be biased toward SingleStore, but in general I want to discuss the bigger picture.

There is just not enough 'valuable' textual data: it costs nothing to store them, in memory or even in GPU. So hosting the data — either as raw data or in vector form — would be a much easier problem than correctly extracting info and embedding the data. Who does the best job extracting and embedding? Model providers.

OpenAI Assistants API already provides a built-in solution for up to 10,000 files, but I don't see any real limitations there.

Leave a Comment