Launching 0xFast Stream: 100x Faster Ethereum Block Downloads

submited by

Style Pass

2022-09-22 16:00:22

As I’ve ventured into the web3 space, I’ve been astounded by how unintuitive getting your hands on web3 data is. The first thing I wanted to do was to download the blockchain data and play with it. Seems simple enough, right? Not quite.

While there are several web3 API services available, they can be excruciatingly slow. To test this out, I wrote a simple Go script to fetch an Ethereum block, all its transactions, its receipts, and logs. And I ran it over these services to see what kind of throughput and latency I could get.

Ethereum mainnet has 15M+ blocks. At these speeds, it would take forever to download blockchain data. Despite a wide array of services, this approach is a no-go.

After this, I ran Geth node locally, used its JSON-RPC interface, then GraphQL interface, and the ethereum exporter — even those didn’t provide a good download speed. Ultimately, I gave up on all API methods and wrote a script to parse Geth’s underlying storage format. I then put together the various pieces needed to form an entire block worth of data. This allowed me to download 15.3M Ethereum blocks — within 12 hours.

To say that the path leading up to this was time-consuming and cumbersome would be an understatement. Being able to query or download entire block data is fundamental to building any indexing or analytics system. Having a way to quickly download any block, a bunch of blocks, or even the entire blockchain should be easy, but it’s not. That changes today.