Airflow is not your data platform
Every data team knows this story: Some team member wrote some Python script that moves some data. You run it once and it works great! Now you want to run it on a schedule. Maybe every day? So someone sets up Airflow.
Then the real fun begins! You need to figure out how to monitor Airflow so you know if it’s still working. And you need some observability to understand why your scripts are slow. Also, you need to get secrets into the scripts in a secure way. And you need to figure out how to get code into production in an automated way. Oh ya, you also need a staging environment!
After much gnashing of teeth, you’ve managed to cobble together either home-grown or purchased services to solve all these problems. Congratulations! You’ve got a data platform! Kind of.
Airflow is just a single piece of the overall puzzle. It’s a workflow orchestrator lacking all the key features you need to do real work.
