Black box testing hundreds of data connectors

Collectively, organizations use thousands of platforms, databases, file systems, and other data silos to run their businesses. Every silo contains a partial view into the business, making it nearly impossible to make data driven decisions without integrating the data sources into a single view. Given the breadth of possible integrations today and in the future, this presents a monumental engineering challenge. 

Airbyte was created to solve this problem. Our founding thesis is that the only way to fully solve data movement is via an extensible, decentralized, open source approach. Therefore, we see our role as that of enabling the building and maintenance of thousands of data connectors with minimal effort. To facilitate creating and maintaining connectors we have created both a Connector Development Kit (CDK), so that developers write less code for each connector, and a suite of Connector Acceptance Tests (CAT). Connector Acceptance Tests are the focus of this article. CATs are black box, platform-agnostic test suites that validate a connector “behaves correctly” and follows best practices. 

The Airbyte Specification describes the rules and behaviors which allow any pair of data connectors to “talk to each other” via Airbyte. As a quick recap, there are two types of connectors in Airbyte: 

Leave a Comment