Design patterns for extracting from REST APIs
But often, that's not sufficient. When it's not, you need to perform an extraction process to get data out of the API first. With the data extracted, then you can build your integration.
Extractions are tricky, though. Most extractions I’ve seen in the wild have major flaws. If your extraction process has a flaw, it will cause holes in your data, and those can be hard to track down.
The two primary requirements for an extraction are accuracy and efficiency. You need your extraction to be 100% accurate. But you also want it to be efficient. Ideally, you don't need your extraction process to paginate an API from front-to-back every time it runs. (Both expensive and slow.)
But extracting is also a good way to create a boundary between an API and your system. By extracting API data into systems like Kafka and Postgres, you decouple the rest of your system from the quirks and limitations of the API. You can design schemas that fit your use case and optimize for your access patterns. And you gain the ability to trigger side effects based on the data flowing through the extraction process.
For example, take Stripe. A record would be a JSON representation of a Stripe subscription. An event would be a JSON object that indicated that a subscription was canceled or crossed some threshold of value.
