How we over-engineered our data pipeline...

When we launched SneakersAPI.dev, our goal was simplicity. It was a proof of concept - a quick experiment to gauge interest in our product. At that stage, our setup was straightforward: a data scraper, a modest server, and Meilisearch for full-text search. We ran the scraper daily, updated the Meilisearch index, and that was it.

The initial setup lasted for about eight months, without any plans for scaling it into the robust solution that SneakersAPI.dev would eventually become. I had almost forgotten about this small side-project, but then one day, I checked the RapidAPI dashboard and noticed that multiple users were accessing the API. This resulted in approximately 50,000 daily requests.

This was a wake-up call, as the data quality was subpar, the server was located in Europe while most users were in the USA (leading to high latency), and we were the only free provider for this type of data. Even today, there is no other free API for sneakers and fashion data, except ours.

It was clear that it was time to build something more substantial. The old website was a mess, hosted on a personal domain, and the API was only available on RapidAPI.

Leave a Comment