OpenAI’s lead over other AI companies has largely vanished, ‘State of AI’ report finds

Hello and welcome to Eye on AI. In this edition…AI's fast-falling cost…Google goes nuclear…LLMs may be dumber than you think…and a filmmaker burned by genAI backlash.

Every year for the past seven, Nathan Benaich, the founder and solo general partner at the early-stage AI investment firm Air Street Capital, has produced a magisterial “State of AI” report. Benaich and his collaborators marshal an impressive array of data to provide a great snapshot of the technology’s evolving capabilities, the landscape of companies developing it, a survey of how AI is being deployed, and a critical examination of the challenges still facing the field.

One of the big takeaways from this year’s report, which was published late last week, is that OpenAI’s lead over other AI labs has largely eroded. Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5, X’s Grok 2, and even Meta’s open-source Llama 3.1 405 B model have equaled, or narrowly surpassed on some benchmarks, OpenAI’s GPT-4o.

But, on the other hand, OpenAI still retains an edge for the moment on reasoning tasks with the release of its o1 “Strawberry” model—which Air Street’s report rightly characterized as a weird mix of incredibly strong logical abilities for some tasks, and surprisingly weak ones for others. (For more on the fragility of o1’s reasoning abilities, see the “Research” section below.)

Leave a Comment