Spatio-temporal indexing the BlueSky firehose
I recently added a "spatial feed" to Aurora, my map of Bluesky. Now, in addition to seeing community clusters laid out on a giant map, you can also see a real-time of posts from just the accounts currently in view. This works smoothly at all scales — you can see the most recent posts from the entire network when zoomed all the way out, and local posts from any neighborhood when zoomed in.
This is actually the first (and only) backend service that I've had to deploy for this project. To compute the clustering and layout for the map, I index the follow graph in a SQLite database that only lives on my home server, do all the data processing locally, and just push static assets to a Cloudflare R2 bucket at the end that the web client fetches directly.
But adding spatial feeds means having the web client make constant queries for post URIs in arbitrary map areas, which it can then "hydrate" into post content from the Bluesky API directly. I didn't want to expose public ports from my home server, so that means deploying a firehose consumer to the cloud.
