Apache Airflow

What it is

The most widely used open-source workflow orchestrator for data pipelines: workflows are Python code (DAGs) that are scheduled, retried, monitored and backfilled.

Maker, ownership and history

Created by Maxime Beauchemin at Airbnb in October 2014; Apache Incubator March 2016; top-level Apache project January 2019 (Wikipedia, secondary). Governed by the Apache Software Foundation; Apache-2.0. Astronomer is the best-known commercial vendor (not researched here).

Editions and deployment

Free open source (self-hosted, Helm/Kubernetes). Managed: Google Cloud Composer, Amazon MWAA, Azure Apache Airflow Job (Wikipedia), plus vendor offerings. Latest release found: 3.3.2 (2026-09-17, PyPI/release notes); 3.3.0 on 2026-07-06.

Core architecture

Scheduler, DAG processor, API server, workers/executors, metadata database. Airflow 3.0.0 (April 2025) added a React UI, a Task Execution API and Task SDK, scheduler-managed backfills and DAG versioning. 3.3.0 (July 2026) added asset partitioning, a task and asset state store, pluggable retry policies and Java and Go task SDKs (release notes).

Role in an enterprise AI rollout

The glue that triggers ingestion (fivetran, airbyte), dbt runs, embedding refreshes and evaluation jobs on a schedule or on asset events. The batch layer; streaming belongs to Flink. See data-pipelines-for-ai.

AI features as of October 2026

apache-airflow-providers-common-ai runs LLM calls and tool-using agents (pydantic-ai) as Airflow tasks: portable LLM operators across vendors (OpenAI, Anthropic, Google, Bedrock, self-hosted), MCP integration, an LLMBatchOperator for batch APIs, and human approval gates. Requires Airflow 3.0.0 or later. Provider maturity (GA vs early) is not stated on the page.

Integrations

Provider packages for Snowflake, Databricks, dbt, Fivetran, Airbyte, Kafka, cloud services. Peers: dagster, prefect, kestra, temporal.

Strengths and weaknesses (opinion)

Strong: ubiquity, hiring pool, provider ecosystem, managed options in every cloud. Weak: task-centric model historically less ergonomic for asset lineage than Dagster; operational weight when self-hosted; 3.x migration effort from 2.x.

Self-learning

• Official tutorial: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/ (free)
• Release notes: https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html (free)
• Common AI provider: https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html
• Certification: not verified (Astronomer academy URLs did not load).

Sources (fetched 2026-10-07)

• https://en.wikipedia.org/wiki/Apache_Airflow (secondary); release notes; PyPI and GitHub API (apache-airflow 3.3.2)

Open items

• Astronomer details, certification names, adoption figures.