Declarative Data Pipelines

submited by
Style Pass
2022-01-15 14:30:08

Declarative data pipelines allow for more reliable, resilient, and reproducible deployments and faster iterations in development.

As stated in CNCF’s definition of cloud-native technologies, declarative APIs are a cornerstone of cloud-native technology. Among APIs, we also have tools like Hashicorp Terraform, AWS CloudFormation, Kubernetes Manifests, and others that commoditize Infrastructure as Code and enable developers to manage their infrastructure in a declarative manner.

A declarative approach allows for more reliable, resilient, and reproducible deployments and faster iterations in development. To allow the declarative definition of data pipelines, the underlying platform needs to fulfill a couple of prerequisites.

A common pattern in cloud-native environments is the outsourcing of the application state, e.g. moving session affinity to load balancers, implementing cache management with key-value stores, etc. State-managing technologies are consumed as services, which are declared as dependencies and either configured during runtime or at deployment time.

Leave a Comment