Search code, repositories, users, issues, pull requests...
Pyper is a flexible framework for concurrent and parallel data-processing, based on functional programming patterns. Used for 🔀 ETL Systems, ⚙️ Data Microservices, and 🌐 Data Collection
Pyper provides an elegant abstraction of the execution of each task, allowing you to focus on building out the logical functions of your program. In the main function:
Each task has workers that submit outputs to the next task within the pipeline via queue-based data structures; this is the mechanism underpinning how concurrency and parallelism are achieved. See the docs for a breakdown of what a pipeline looks like under the hood.
A pipeline consisting of at least one asynchronous function becomes an AsyncPipeline, which exposes the same usage API, provided async and await syntax in the obvious places. This makes it effortless to combine synchronously defined and asynchronously defined functions where need be.
Pyper is implemented in pure Python, with no sub-dependencies. It is built on top of the well-established built-in Python modules:
