The library heavily relies on goroutines and channels, but this complexity is hidden from the user behind a simple API. Basically, this is the main purpose why I implemented this library in the first place.
Please, note that the library is still in the early development stage, so the API might change in the future. There still might be some bugs, so please, feel free to report them.
Pipeline is a sequence of stages. It is the first and the key object in the library. Its API provides a possibility to interrupt the entire pipeline.
The user can see the pipeline current status by accessing the Status field. Please, note that the status is updated asynchronously, so it may not be up-to-date right away after the change - eventual consistency.
It is possible to configure the pipeline by using the configs.PipelineConfig struct, which contains the following config options:
If you pipeline performs any network calls within its transformation/aggregation logic, I'd suggest configuring the maximum number of goroutines to prevent the possible DDoS attack on the target server or reaching the maximum number of open files on the client machine.