OpenLineage: How the Open Standard Tracks Spark, dbt and Airflow Lineage

OpenLineage is an open standard for collecting data lineage at runtime. Instead of scanning your warehouse after the fact, instrumented tools such as Apache Spark, Airflow, dbt and Flink emit JSON events while a job is running, describing which datasets went in, which came out, and what happened in between. It is an LF AI and Data Foundation Graduate project under the Apache 2.0 license, and it defines the format only. You still need a backend to receive and store the events.

That last sentence is the part most introductions skip, and it is the one that decides whether adopting OpenLineage will actually give you a lineage graph. The specification is a contract between producers and consumers. Turning it on emits events. Something else has to catch them.

What is OpenLineage?

The project describes itself as "an open framework for data lineage collection and analysis" with "an extensible specification that systems can use to interoperate with lineage metadata." It started at WeWork in October 2020, is now governed under the LF AI and Data Foundation, and the reference repository sits at OpenLineage/OpenLineage with roughly 2,600 stars and an Apache 2.0 license. Development is genuinely active rather than nominally alive: release 1.52.0 shipped on 23 July 2026, following 1.51.0 in early July and 1.50.0 in June, and the repository saw commits this week.

The data model is deliberately small. Four concepts carry everything:

Facets are where the real detail lives, and where the standard gets its extensibility. The current specification ships around 39 of them in spec/facets, covering schema, SQL text, source code location, ownership, data quality metrics and assertions, error messages, processing engine, job type, and column lineage. If you need something the standard does not model, you define a custom facet rather than fork the spec.

Does OpenLineage support column-level lineage?

Yes, through the ColumnLineageDatasetFacet, currently at schema version 1-2-0 against core spec 2-0-2. It maps each output field to the input fields used to evaluate it, recording the input dataset namespace, dataset name and field name for every dependency.

The detail worth knowing is that each dependency carries a typed transformation. The allowed values for type are DIRECT and INDIRECT, plus an optional subtype and a free-text description. DIRECT means the input value actually flowed into the output column. INDIRECT means the input influenced the row without appearing in the value, which is what a column in a WHERE, GROUP BY or JOIN clause does. That distinction matters more than it sounds: a filter column is part of how a number was produced, and lineage that ignores it will understate your blast radius during an impact review.

There is also a masking boolean on each transformation, flagging that the value was obscured on its way through. That is a small field with real audit value, because it is machine-readable evidence of where sensitive columns were protected in transit, which is the kind of artifact you end up hunting for by hand when mapping controls to SOC 2 or HIPAA.

The catch is coverage. The facet exists in the specification, but whether it is populated depends entirely on the integration doing the emitting. The Spark integration produces detailed column lineage because it can read the query plan. Others emit dataset-level lineage and leave the facet empty. Support is per-integration, not guaranteed by the standard.

Which tools emit OpenLineage events?

The project maintains official integrations for Apache Airflow, Apache Spark, Apache Flink, Apache Hive, dbt, Trino, SQL, Feast and Great Expectations. Beyond those, a growing list of vendors emit natively, including Dagster, Snowflake and Google Cloud.

Two of these are worth calling out because they are what most teams start with:

• Airflow. Lineage is delivered by the apache-airflow-providers-openlineage package, which is one of the most actively maintained providers in the ecosystem: version 2.20.0 was published on 8 August 2026. Operators that support it emit lineage automatically once the provider is configured. Our guide to Airflow data lineage covers the setup and where the coverage stops.
• dbt. The openlineage-dbt package, also on 1.52.0, wraps a dbt run and converts the manifest and run results into OpenLineage events.

Events travel over a configurable transport. HTTP to a collector endpoint is the common choice, Kafka is supported for higher volume, and there is a file transport that is genuinely useful for testing what your pipeline is actually emitting before you point it anywhere.

Where do OpenLineage events go?

A producer without a consumer gets you nothing but outbound HTTP calls. The consumer side is the decision that determines what you can actually see.

Marquez is the natural starting point because it was built as the reference consumer and it is free, but running it is running a service: an API, a web UI and a Postgres database that you keep alive and back up. Commercial catalogs remove that operational burden and add search, ownership and governance on top. Our data lineage tools roundup compares the field, and the IBM Manta alternative page covers the vendor that pairs an OpenLineage scanner with the deepest legacy code parsing in the market.

What OpenLineage does not give you

This is the honest limitation, and understanding it before you invest a quarter in instrumentation will save you a rebuild.

OpenLineage only sees jobs you have instrumented. It is a push model. A pipeline emits an event because someone installed a provider and configured a transport. Everything else is invisible. In practice that means the analyst who created a table from the Snowflake console leaves no trace. The scheduled query someone set up in BigQuery two years ago emits nothing. The Python script on a VM, the reverse-ETL tool, the Looker PDT, the one-off backfill: none of them appear unless a maintained integration exists and is switched on.

The gap is not marginal. In most warehouses a substantial share of tables were last written by something that is not Airflow or dbt, and those are frequently the tables that break. A lineage graph that is confidently complete about the instrumented half of your estate and silent about the rest is a graph you cannot trust for impact analysis, which is the use case that justified building it.

Two smaller points follow from the same design. Coverage of the column-lineage facet varies by integration, so granularity is uneven across your graph. And because events are tied to runs, you get lineage from the moment you turned it on, not history. There is no backfill for pipelines that already ran.

The complementary approach is to read what the warehouse already recorded. Every query that ran against Snowflake, BigQuery, Databricks or Redshift is in the query history, which means the lineage is derivable after the fact, for every writer, with no instrumentation and no gaps by tool. That is how Datatrail builds column-level lineage, and it is why impact analysis can answer what a schema change breaks rather than what it breaks among the jobs you remembered to instrument. The two models are not rivals. Runtime events tell you what a job intended; query history tells you what actually happened.

Is OpenLineage production ready?

By the usual signals, yes. It is a Graduate project at the LF AI and Data Foundation rather than an incubating one, it is Apache 2.0 licensed with no vendor holding the copyright, releases ship roughly every two to three weeks, and the largest cloud providers have built consumers against it. Google, AWS and IBM all document OpenLineage ingestion, which is about as strong a signal of standard stability as this category produces.

The realistic assessment is that OpenLineage has won the argument about format and is still winning the one about coverage. Adopt it for the pipelines you own and control, because a shared format is genuinely better than every tool inventing its own. Do not assume it gives you a complete picture of your warehouse, because by construction it cannot.

The short version

• OpenLineage is a specification for emitting lineage events at runtime, not a lineage product.
• The model is Dataset, Job, Run and Facet, with about 39 facets in the current spec.
• Column-level lineage exists via ColumnLineageDatasetFacet, with DIRECT and INDIRECT transformation types and a masking flag, but population varies by integration.
• Official producers include Spark, Airflow, dbt, Flink, Hive, Trino and Great Expectations. Airflow support ships as a first-party provider.
• You need a consumer. Marquez is the free reference, though its release cadence stalled in 2025; DataHub, OpenMetadata, Google, AWS and IBM all ingest events. See what running Marquez actually costs for the comparison.
• It only sees instrumented jobs, so console SQL, scheduled queries and ad hoc scripts stay invisible, and there is no history before you enable it.