Airflow Data Lineage: OpenLineage, Marquez, and What They Do Not Capture

Apache Airflow collects data lineage through OpenLineage, an open standard, using the native apache-airflow-providers-openlineage package that has shipped inside Airflow since version 2.7 and continues in Airflow 3. The provider listens to task runs, extracts metadata per operator, and emits lineage events to a backend such as Marquez, the reference OpenLineage store and UI. For SQL operators it can produce column-level lineage by parsing the executed SQL. The coverage is only as complete as your instrumented operators: un-instrumented tasks, work run outside Airflow, and the BI layer downstream of the warehouse do not appear, which is why many teams pair it with warehouse-native lineage.

If you run Apache Airflow and want to know how data actually flows through your pipelines, this is how you get lineage out of it, what you can realistically expect, and where the picture stops.

Airflow does not carry lineage: OpenLineage does

Airflow orchestrates tasks; it does not, by itself, maintain a graph of how datasets connect. The standard way to get one is OpenLineage, an open, vendor-neutral specification for lineage events, wired into Airflow through the native apache-airflow-providers-openlineage provider. That provider has been part of Airflow since 2.7 and carries forward into Airflow 3, superseding the older standalone openlineage-airflow package you may still see referenced in older tutorials.

Mechanically, the provider registers listener hooks into Airflow's task lifecycle. When a task runs, an operator-specific extractor pulls metadata about the datasets it read and wrote, and the provider emits an OpenLineage event describing that run. Those events conform to the OpenLineage spec, which is the same standard used by the Spark and dbt integrations, so Airflow is one producer in a broader ecosystem rather than a closed system.

Marquez: where the events go

Emitting events is only half the job; they need somewhere to land. The reference backend is Marquez, an open-source OpenLineage store with a metadata repository, a query API, and a graph UI. Point the provider at a Marquez instance by setting the OpenLineage transport (typically an HTTP endpoint via an environment variable or the provider config), run your DAGs, and Marquez accumulates a catalog of historical runs plus the lineage graph you can browse.

Marquez is the reference implementation, not the only option: any OpenLineage-compatible backend can receive the events, and several commercial and open tools consume the OpenLineage format. But if you are getting started, standing up Marquez is the usual first step to actually see your Airflow lineage drawn rather than just emitted into the void.

What about column-level lineage?

OpenLineage does support column-level lineage, and Airflow's provider can produce it, with an important qualifier: it works well for SQL-based operators. When a task runs SQL against a supported database, the provider parses that SQL to resolve which input columns produced which output columns, so an INSERT ... SELECT or a CREATE TABLE AS yields column-to-column edges, not just table-to-table ones. The column-lineage capability has matured over recent releases and now tracks derivations more completely than early versions did.

The qualifier is the operator. A PythonOperator that reshapes a dataframe in memory has no SQL to parse, so unless a custom extractor emits column facets explicitly, you get dataset-level lineage at best and often nothing. The general rule: the more of your transformation logic is SQL against instrumented operators, the better your column-level coverage; the more of it is arbitrary Python or shell work, the thinner it gets.

The coverage gap: what Airflow lineage cannot see

This is the part worth understanding before you rely on Airflow lineage as your source of truth. OpenLineage in Airflow is a push model, so a task shows up in the graph only if its operator emits an event. That creates three predictable blind spots.

Un-instrumented tasks. SQL operators emit rich lineage; custom operators and in-memory Python transforms may emit little or none. The graph gaps out precisely where the interesting transformation happened, which is the opposite of what you want.

Everything that did not run through Airflow. An analyst running a CREATE TABLE AS by hand, a Fivetran or Airbyte load, a scheduled query in the warehouse console, a reverse-ETL sync triggered elsewhere: none of it flows through Airflow, so none of it appears in Airflow lineage. Most warehouses accumulate a lot of this out-of-band activity, and an orchestration-scoped view is blind to all of it.

The BI layer. OpenLineage traces to the tables Airflow writes. The Looker explore, the Tableau workbook, or the reverse-ETL destination that reads those tables sits one hop past the orchestrator, so the graph usually ends at the warehouse boundary, which is exactly one hop short of where a broken column actually gets noticed.

There is also an operational cost worth naming honestly: you run and maintain Marquez (or another backend), keep the provider and extractors current as your operators evolve, and accept that lineage quality tracks instrumentation quality. Running that infrastructure alongside your warehouse compute is real spend, and teams that watch where their cloud and data-platform costs actually go tend to weigh the build-and-operate cost of a self-hosted lineage stack against a managed one.

Filling the gap with warehouse-native lineage

The complementary approach is to collect lineage from the warehouse side instead of the orchestrator side. Rather than depending on each task to emit an event, a warehouse-native tool connects to Snowflake, BigQuery, Redshift, Databricks, or Postgres with a read-only role and parses the query history the warehouse already records. Because the source is the SQL that actually executed, the column-level lineage covers every statement that touched the warehouse: Airflow tasks, ad hoc queries, other schedulers, and un-instrumented jobs alike. There is nothing to instrument and no per-operator extractor to write.

This is the approach Datatrail takes alongside Airflow. The graph extends past the warehouse into dbt models and the BI layer, impact analysis names every downstream model and dashboard that reads a column before you change it, and schema change alerts catch upstream changes that would land unannounced. The two are complementary rather than competing: OpenLineage gives you run-aware, orchestration-level lineage inside Airflow, and warehouse-native lineage gives you complete, persistent column lineage out to the dashboards. Many teams keep OpenLineage for pipeline observability and add warehouse-native lineage for the full warehouse-to-dashboard picture.

The short version

Get lineage out of Airflow with the native OpenLineage provider, store and visualize it in Marquez, and expect solid column-level lineage for SQL operators and gaps everywhere else. If your pipelines are mostly SQL and mostly run through Airflow, that may be enough. If a lot of work happens outside Airflow or in Python, or you need to trace a column all the way to the dashboard that shows it wrong, pair Airflow lineage with a warehouse-native graph. You can see that graph on your own warehouse in a few minutes from the product, and our comparison of data lineage tools lays out where each option fits.