Two big announcements from Anthropic today: a new Claude 3.5 Sonnet model and a new API mode that they are calling computer use. Unlike OpenAI’s Cod

Simon Willison’s Weblog

submited by

Style Pass

2024-10-22 18:00:22

Two big announcements from Anthropic today: a new Claude 3.5 Sonnet model and a new API mode that they are calling computer use.

Unlike OpenAI’s Code Interpreter mode, Anthropic are not providing hosted virtual machine computers for the model to interact with. You call the Claude models as usual, sending it both text and screenshots of the current state of the computer you have tasked it with controlling. It sends back commands about what you should do next.

The quickest way to get started is to use the new anthropic-quickstarts/computer-use-demo repository. Anthropic released that this morning and it provides a one-liner Docker command which spins up an Ubuntu 22.04 container preconfigured with a bunch of software and a VNC server.

I’ve tried this and it works exactly as advertised. It starts the container with a web server listening on http://localhost:8080/—visiting that in a browser provides a web UI for chatting with the model and a large [noVNC](https://github.com/novnc/noVNC) panel showing you exactly what is going on.

This has very obvious safety and security concerns, which Anthropic warn about with a big red “Caution” box in both new API documentation and the computer-use-demo README, which includes a specific callout about the threat of prompt injection: