I've spent a lot of time recently with Anthropic's Computer Use capability. The results are fascinating from an engineering perspective. We

The Brittle Lesson: Claude & Computer Use — Anjney Midha

submited by

Style Pass

2024-10-25 02:30:02

I've spent a lot of time recently with Anthropic's Computer Use capability. The results are fascinating from an engineering perspective. We're seeing a fundamental tension between clean abstractions and messy reality that reminds me of the early days of 3D engines.

The core insight here is actually pretty profound: Computer vision is becoming our universal API layer, despite being theoretically inferior to structured interfaces in almost every way. This feels wrong to anyone who cares about clean engineering, but the empirical evidence is becoming hard to ignore.

It's objectively worse in terms of computation, latency, and reliability. We're taking perfectly good structured data, rendering it to pixels, then trying to reconstruct the structure through ML. It's the kind of thing that would make any serious systems programmer cringe.

But here's the thing: it works. Not perfectly, not reliably, but it works on literally everything that can render to a screen. The abstraction is leaky as hell, but the universality is compelling.