Iterating through all records from an API
Transferring data from an API is a common task. On the surface the task sounds simple enough, but you'll need to consider these following issues when building an Extract, Transform and Load (ETL) pipeline.
In this guide we'll be using HelpScout's Conversation API as an example of processing a large amount of records. This would be helpful to extract the data needed to train a chatbot on all past conversations with customers.
You can also accomplish task this using a custom Source. Sources are serverless functions in Pipedream that includes polling, deduplication, and triggering subscribed workflows. But for this guide we'll use workflows, which can be built, tested and deployed without leaving the Pipedream dashboard.
But also consider that list or search API endpoints typically don't return all records in single query. You typically need to paginate through records with multiple API requests. So the 12.5 minute maximum won't be an issue.
In a new step in your workflow, search for and select the Help Scout integration, then choose the Build any Help Scout API request action:
