Apache Airflow

What is Apache Airflow?

Apache Airflow is an open source platform for authoring, scheduling and monitoring workflows in Python. It is maintained by the Apache Software Foundation and represents workflows as directed acyclic graphs, or DAGs, made of tasks. The problem it solves is dependable automation: teams replace fragile cron jobs and scripts with workflows that have dependencies, retries, logs and a central UI.

Key features

• Python-defined workflows. DAGs are written in Python, so teams can use normal code, libraries and tests instead of a separate configuration language.
• Scheduling and dependencies. The scheduler runs workflows on a timetable and respects task dependencies, backfills and catchup settings.
• Rich web interface. The UI shows DAG runs, task states, logs, durations and retries for debugging.
• Operators and providers. Hundreds of operators and provider packages connect Airflow to cloud services, databases, warehouses and SaaS tools.
• Dynamic task mapping. Workflows can create tasks at run time based on upstream data, which suits variable batch sizes.
• Retries and alerts. Tasks can retry on failure, and alerts can be sent when a run or task fails.
• Managed options. Cloud providers and vendors offer managed Airflow services, while the core project remains free and self-hosted.

Who is Apache Airflow for?

Data engineers and platform teams building ETL, ELT, machine learning and batch workflows. It suits organisations with many scheduled jobs that need dependencies and visibility, teams already comfortable with Python, and companies that want an open source orchestrator they can self-host or run through a managed service. It is less suited to simple one-off scripts or teams that want a fully managed product with no infrastructure.

Use cases

• Scheduling nightly ETL jobs between databases and a warehouse.
• Running machine learning training and batch scoring pipelines.
• Orchestrating data quality checks and downstream reports.
• Backfilling historical data after a pipeline change.
• Coordinating cloud services, APIs and file transfers.

Pricing

Apache Airflow is free and open source under the Apache License 2.0. The project does not sell a paid plan, and teams can self-host it at no licence cost. Managed Airflow services are sold separately by cloud providers and vendors, with their own pricing. See the official Apache Airflow site.

Getting started

1. Install Airflow in a Python environment.
2. Start the standalone server for local development.
3. Create a DAG file with tasks and dependencies.
4. Open the web UI, trigger the DAG and inspect the run.

pip install apache-airflow
airflow standalone

Alternatives

• Prefect is a Python workflow orchestration platform with a managed cloud option.
• Dagster is a data orchestrator built around software-defined assets.
• Temporal is a durable execution platform for long-running workflows.

FAQ

Is Apache Airflow free?

Yes. Apache Airflow is open source under the Apache License 2.0 and has no licence fee. Managed Airflow services from cloud providers are priced separately.

What is Apache Airflow used for?

Airflow is used to schedule and monitor workflows such as ETL, data processing, machine learning and batch jobs. Workflows are defined as Python DAGs with task dependencies.

Airflow vs Prefect?

Airflow represents workflows as DAGs and has a large operator ecosystem and broad enterprise adoption. Prefect lets teams write native Python functions with decorators and focuses on dynamic workflows and developer experience. The right choice depends on existing skills and how dynamic your pipelines are.

Can I self-host Airflow?

Yes. Airflow is designed to be self-hosted on your own infrastructure, including containers and Kubernetes. Managed services are optional.

Does Airflow support cloud services?

Yes. Provider packages and operators connect Airflow to AWS, Google Cloud, Azure, databases, warehouses and many SaaS tools.