Welcome back: we’re discussing how to write an AI coding / copilot service named The Owl. This is the post I have been most looking forward to w

Building a Copilot #2: Parsing & Converting On The Fly - Parnassus

submited by

Style Pass

2024-11-27 14:00:08

Welcome back: we’re discussing how to write an AI coding / copilot service named The Owl. This is the post I have been most looking forward to writing, because it describes a logical but semi-crazy sequence of incremental streamed parsing layers.

At the end of Part 1, we had a Python server that streamed text to a client using Server Sent Events. In the client, this let us display text as it arrived word by word:

AIs can easily return Markdown-formatted text, but the server response from the AI server is not just the text but a complex object (ie, JSON.) Code reading the AI’s response needs to handle partially complete data, and process it as it arrives piece by piece.

But this is not the only layer of parsing and streaming on the fly. In fact, the Owl has four layers of parsing incomplete streams of data:

This is four different levels of data arriving via a stream and being parsed and handled on the fly, token by token as it arrives. Of them all, the most complex to implement (to me) was the Markdown parsing.