Airflow

Airflow's task-based DAG model is straightforward: define tasks, define dependencies between them, schedule the DAG, and let Airflow handle execution, retries, and monitoring. For pipelines that follow a predictable pattern (extract, load, transform, refresh BI tool), this model works well. The web UI gives operators visibility into task status, run history, and logs. When something fails, you can see what failed, re-run from the failure point, and trace the issue.

The pre-built operator ecosystem is Airflow's strongest practical advantage. Need to trigger a dbt run? There is an operator. Need to check if an S3 file exists before processing? There is a sensor. Need to call the Snowflake API? Covered. This breadth means teams spend less time writing glue code and more time defining pipeline logic.

For an NBA team, we stabilized an Airflow environment that was failing multiple times per week. After rebuilding critical DAGs with proper error handling, retry logic, and monitoring, the failure rate dropped to one incident in four months.

Airflow is built around a task-based execution model: it tracks that task B depends on task A. When teams also want a data-asset view, knowing that task B produces a table that task C reads, this is something you layer on rather than get from the scheduler itself. Pairing Airflow with a lineage or data-observability layer gives you stale-asset detection, change-aware execution, and column-level lineage alongside the orchestration Airflow already handles well.

For teams starting fresh that want asset-level observability and orchestration combined in one model, Dagster is also worth evaluating. Where Airflow is already in place, the asset layer is a complement rather than a replacement.

For local development, running a full Airflow environment locally involves a scheduler, metadata database, and web server. We set teams up with containerized local environments and CI-based validation so DAGs get thorough testing before they reach production.

The most common Airflow problem we encounter is DAGs written without adequate error handling. Tasks fail, the failure is logged, but nobody is notified. Data goes stale for days before a stakeholder notices. We add structured alerting (Slack, PagerDuty, email) with severity-based routing so the right person knows within minutes.

The second common issue is DAGs that were written for an earlier data volume. A DAG that ran in 10 minutes two years ago now runs for 3 hours because the data grew and nobody adjusted the task logic, timeouts, or parallelism. We audit DAG performance and restructure where the pipeline's requirements have outgrown its configuration.

As the number of DAGs and tasks grows, the scheduler benefits from tuning to keep pace. We tune the scheduler configuration, executor settings, and worker scaling to handle the current workload, and set up monitoring so you know when it is time to tune again.