An alternative to cursor pagination

submited by

Style Pass

2024-04-29 08:00:03

I need to first give context. I have been part of a multi-year project that splits an existing SaaS product from one AWS region into separate instances in several AWS regions, to improve availability and reliability through improved isolation. This has been greatly celebrated internally. The migration’s main motivation is consolidation of infrastructure improvements. Part of migrating infrastructure is moving existing data and I’ve been primarily working on moving elasticsearch data in the past 6 months.

Based on the limitation of 10,000 docs per API request, it’s required that some version of pagination is needed. Most of our customers’ data consists of more than 10,000 ES documents.

There is one big benefit to cursor pagination. Everyone could understand cursor pagination in a few minutes to an hour. I definitely under-estimated how an easy to understand algorithm would benefit everyone. If ops could understand it, they are more able to optimize the machine to the usage pattern. If engineers could understand it, you can get a buy-in quicker.

4 months into the project, an idea emerges from reading Merkle tree wikipedia page (Seeing the illustration in the wikipedia page helps me grasp the data structure concept).