Airflow Data Lineage With OpenLineage, and Where It Stops
Apache Airflow can emit lineage through the OpenLineage provider, and Marquez stores and draws it. That gives you orchestration-scoped lineage for the tasks you instrument. Datatrail reads the warehouse query history directly, so the graph covers everything that ran, not just what Airflow emitted.
Read-only connection. Datatrail never moves or mutates your data.
In short
Airflow data lineage is usually collected with OpenLineage, an open standard for lineage events. The native apache-airflow-providers-openlineage package, included with Airflow since 2.7 and carried forward in Airflow 3, listens to task runs and emits lineage events to a backend such as Marquez, the reference OpenLineage store and UI. For SQL operators it can produce column-level lineage by parsing the executed SQL. The coverage is only as complete as the operators you instrument: tasks with no extractor, ad hoc queries run outside Airflow, and the BI layer downstream of the warehouse do not appear. Datatrail takes the complementary approach, connecting to the warehouse read-only and parsing its query history into a persistent column-level graph, so the lineage covers every statement that touched the warehouse regardless of whether Airflow orchestrated it.
Why it fits
Airflow teams who want lineage beyond what instrumented tasks emit, covering ad hoc queries and the BI layer.
OpenLineage is the standard
Airflow emits lineage through the native OpenLineage provider (built in since 2.7, carried into Airflow 3), and Marquez is the reference backend that stores and visualizes it. Datatrail is not trying to replace that standard.
Instrumentation-scoped by design
OpenLineage sees a task only if that operator has an extractor. Un-instrumented operators, PythonOperator work, and queries run outside Airflow leave gaps in the graph.
Datatrail fills the rest read-only
By reading warehouse query history directly, Datatrail captures the columns every statement touched, then follows them out to dbt and the BI dashboards Airflow lineage stops short of.
How Airflow produces lineage: OpenLineage and Marquez
Airflow does not carry a lineage graph of its own. The standard way to get one is OpenLineage, an open specification for lineage events, wired in through the native apache-airflow-providers-openlineage package. That provider has shipped inside Airflow since version 2.7 and continues into Airflow 3, replacing the older standalone openlineage-airflow integration. It registers listener hooks, watches task runs, extracts metadata through per-operator extractors, and emits OpenLineage events describing the datasets each task read and wrote.
Those events have to go somewhere. The reference backend is Marquez, an open-source OpenLineage store with a query API and a graph UI; it is the tool most teams stand up first to see their Airflow lineage drawn. For SQL-based operators, the provider can resolve column-level lineage by parsing the executed SQL, so an INSERT ... SELECT shows which input columns produced which outputs, not just which tables were involved. This is a genuinely good, open, vendor-neutral way to collect lineage, and if you are standardizing on OpenLineage it is the right foundation.
Where Airflow lineage stops: the coverage gap
The limit is structural, not a bug. OpenLineage in Airflow is a push model: a task appears in the graph only if the operator running it has an extractor that emits an event. That produces three predictable gaps.
Un-instrumented tasks. SQL operators against supported databases emit rich lineage; a PythonOperator that reshapes data with pandas, a bash task shelling out to a script, or a custom operator with no extractor emits little or nothing. The graph shows a gap exactly where the interesting transformation happened.
Everything that did not run through Airflow. An analyst who runs a CREATE TABLE AS by hand, a reverse-ETL sync triggered by another system, a Fivetran load, a scheduled query in the warehouse console: none of it flows through Airflow, so none of it appears in Airflow lineage. Most warehouses have a lot of this, and it is invisible to an orchestration-scoped view.
The BI layer. OpenLineage traces to the tables Airflow writes. The Looker explore, the Tableau workbook, and the reverse-ETL destination that read those tables sit downstream of the orchestrator, so the graph typically ends at the warehouse boundary, which is one hop short of where a broken column is actually noticed.
There is also the operational cost: you run and maintain a Marquez deployment (or another OpenLineage backend), keep provider versions and extractors current as your operators change, and accept that lineage quality tracks how well each operator is instrumented.
How Datatrail complements Airflow lineage
Datatrail approaches the same problem from the warehouse side rather than the orchestrator side. Instead of relying on each task to emit an event, it connects to Snowflake, BigQuery, Redshift, Databricks, or Postgres with a read-only role and parses the query history the warehouse already records. Because the source is the SQL that actually executed, the column-level lineage covers every statement that touched the warehouse: Airflow tasks, ad hoc queries, other schedulers, and un-instrumented jobs alike. There is nothing to instrument and no extractor to write for each operator.
From there the graph extends past the warehouse the way it does on our other platforms, out into the dbt models and the Looker, Tableau, and reverse-ETL layer where a break is felt. Impact analysis then names every downstream model and dashboard that reads a column before you change it, and schema change alerts catch upstream changes that would otherwise land unannounced. The two approaches are complementary: OpenLineage gives you run-level, orchestration-aware lineage inside Airflow, and Datatrail gives you complete, persistent, warehouse-native column lineage out to the dashboards. If you run more than one warehouse, Snowflake, BigQuery, and Redshift land in the same graph, and the wider field is in our comparison of data lineage tools.
Questions people ask
Airflow lineage, answered
Does Apache Airflow have built-in data lineage?
Not on its own, but it ships the native OpenLineage provider that collects it. The apache-airflow-providers-openlineage package, included since Airflow 2.7 and carried into Airflow 3, listens to task runs and emits OpenLineage events to a backend such as Marquez, which stores and visualizes the graph. So Airflow produces lineage through OpenLineage rather than carrying its own graph, and the completeness depends on which operators are instrumented.
What is OpenLineage and how does it relate to Airflow?
OpenLineage is an open, vendor-neutral standard for collecting lineage events from data tools. Airflow implements it through the native OpenLineage provider, which hooks into task execution, extracts metadata per operator, and emits events describing the datasets each task read and wrote. Marquez is the reference backend that receives those events and draws the graph. OpenLineage also has integrations for Spark and dbt, so it is a broader standard that Airflow is one producer for.
Does Airflow OpenLineage support column-level lineage?
Yes, for SQL-based operators. When a task runs SQL against a supported database, the OpenLineage provider can parse that SQL to resolve which input columns produced which output columns, giving column-level detail rather than only table-to-table edges. Coverage is weaker for operators that do not run SQL, such as a PythonOperator transforming data in memory, because there is no SQL to parse and the extractor may emit only dataset-level information or nothing at all.
Why does Airflow lineage miss part of my pipeline?
Because OpenLineage in Airflow only sees a task when that operator has an extractor emitting events. Un-instrumented operators, custom Python tasks, and anything that runs outside Airflow entirely, such as an analyst query, a Fivetran load, or a reverse-ETL sync, never produce an Airflow lineage event, so they leave gaps. The graph also typically ends at the warehouse tables Airflow writes, one hop short of the BI dashboards that read them. Reading warehouse query history directly closes those gaps.
Can I use Datatrail alongside OpenLineage and Airflow?
Yes, and they complement each other well. OpenLineage gives you run-aware lineage inside Airflow, tied to specific task executions. Datatrail connects to the warehouse read-only and parses query history into a persistent column-level graph that covers every statement, including work that never ran through Airflow, and extends it out to dbt and the BI layer. Teams often keep OpenLineage for orchestration observability and use Datatrail for complete warehouse-to-dashboard lineage and impact analysis.
More use cases
Related features
Map your lineage, end to end
Connect your warehouse read-only and see your lineage map in minutes. Datatrail never moves or mutates your data.