Concurrency For Starlette Apps (e.g FastAPI / FastHTML)
Motivation: we often want to call LLMs in Starlette based apps (FastHTML, FastAPI, etc.) apps, and we don’t want to block the server on network calls to APIs.
This post documents my explorations of various approaches that can run tasks in the background without blocking the main process.
You are often already using a database for your web application, and if you need to process items in that database with some kind queue, its convenient to use the database itself!
Next, we want perform proceessing on items from the queue, but do so in the background. We can use the ThreadPoolExecutor from Python’s concurrent.futures module to process items in a thread pool without blocking the main process. Here’s how we can modify our implementation to achieve this:
On a completely separate note, we can use async processing, which is very similar to threads. The main benefit of async over threads is that async is easier to debug (stacktrace, breakpoints, etc).
