What ChatGPT and Claude Can See on Your Screen
We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.
Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.
Pasting screenshots into ChatGPT, and now Claude, has become a regular practice for me. As we first saw in my post about an LLM-backed Datasette plugin, the ability of LLMs to read text in images has torn down the barrier that once separated data from pictures of data. That’s a big deal, but their screen-reading power extends far beyond just reading text.
In this case, while debugging SQL, I found it easier to provide LLMs with screenshots of Postgres output than to copy text from the terminal.
That picture is worth quite a few words. It says that we’re operating in the context of Python’s Google API client and that we’ve authenticated to the service with some kind of valid credential, but the document ID is wrong or a necessary scope wasn’t granted (or requested by the app), or perhaps there’s a different problem. Using words to transmit that context to another person would be tedious — that’s why we screenshare. Using the same words to prompt a language model would be just as tedious. The ability to show rather than tell is a gamechanger.
