Databricks Data Lineage: Unity Catalog Lineage, Extended to Every Column and Dashboard
Unity Catalog already captures a lot of lineage inside Databricks. Datatrail reads that graph, fills the holes it documents, and follows the column out to the models and dashboards that break when you change it.
Read-only connection. Datatrail never moves or mutates your data.
In short
Databricks data lineage is captured natively by Unity Catalog, which tracks tables, views, notebooks, jobs, and dashboards down to the column level for queries run on Databricks, aggregated across every workspace on the metastore. It has documented gaps: column lineage is not captured for path-based references such as delta."s3://bucket/path", user-defined functions can obscure the column mapping, lineage is not preserved when a catalog, schema, table, or column is renamed, and the lineage system tables keep only a rolling one-year window. Datatrail reads Unity Catalog lineage and your query history together, resolves the column-level edges Unity Catalog leaves out, and extends the graph downstream to the dbt models and BI dashboards that depend on each field, so impact analysis covers the whole path and not just the Databricks part of it.
Why it fits
Databricks and Lakehouse teams on Unity Catalog who need column-level lineage and impact analysis that reaches past the workspace.
Fills the gaps Unity Catalog documents
Path-based reads, UDF-heavy transforms, and renamed objects are exactly where native column lineage drops out. Datatrail parses the SQL and notebook queries themselves, so the column edge survives a rename or a UDF.
Beyond the one-year window
The Unity Catalog lineage system tables hold a rolling one-year window. Datatrail keeps its own graph, so lineage for a table that only rebuilds quarterly does not quietly age out of the picture.
Out to the dashboards
Native lineage is strongest inside Databricks. Datatrail carries the column onward into dbt models and the BI dashboards that read the gold layer, so you see the real blast radius before you ship.
Start with what Unity Catalog gives you free
Before you evaluate anything, turn on Unity Catalog lineage and look at it. It is automatic, it costs nothing, and it is better than most teams assume. It captures lineage for queries run on Databricks down to the column level, aggregated across every workspace attached to the metastore, and it covers more than tables: notebooks, jobs, pipelines, queries, dashboards, and ML model versions all appear in the graph. Lineage in Catalog Explorer is retained indefinitely for data captured after September 1, 2024, and the system tables system.access.table_lineage and system.access.column_lineage let you query it directly.
For a lot of lakehouse teams that is enough, and we would rather you know it. The question is only what happens when you hit one of its edges.
The four gaps that send teams looking
Databricks documents these, to its credit, but they are easy to walk into.
- Path-based reads lose the column mapping. Column lineage is captured only when source and target are referenced by table name. A query like
select * from delta."s3://bucket/path"cannot be resolved to columns, so a bronze layer that still reads raw files by path has holes exactly where data enters the platform. - UDFs obscure derivations. When a column comes out of a user-defined function, the clean column-to-column mapping gets fuzzy. Heavily UDF-driven transforms produce lineage that is technically present and practically vague.
- Renames break the chain. Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns. Since renaming is one of the main reasons you go looking for lineage, this one stings.
- The system tables hold one year. Lineage system tables maintain a rolling one-year window, so a dependency that only fires during an annual or quarterly rebuild can age out and disappear.
There is a fifth, softer limit: the Catalog Explorer column view shows one level up and one level down. Impact is transitive by definition, so one hop cannot answer what breaks.
What Datatrail adds on top
Datatrail reads Unity Catalog lineage and your query history together, parses the SQL itself so a column edge survives a rename or a UDF, and keeps a persistent graph so a quarterly dependency does not roll out of a one-year window. Then it walks the chain transitively, out past the lakehouse into the dbt models and BI dashboards that read your gold layer.
The result is impact analysis that answers the question you actually have: if I drop this field, which dashboards go wrong and who opens them, computed before you merge rather than after someone notices a wrong number. The connection is read-only. Datatrail never moves, copies, or mutates a row in your lakehouse. For the full breakdown of what native lineage covers, read Unity Catalog lineage: what Databricks captures and what it misses.
Questions people ask
Databricks lineage, answered
Does Databricks have data lineage?
Yes. Unity Catalog captures data lineage automatically for all queries run on Databricks, down to the column level, with no agent to install. It covers tables, views, notebooks, jobs, pipelines, queries, dashboards, and ML model versions, and lineage is aggregated across every workspace attached to the same metastore. It is available in Catalog Explorer and queryable through the lineage system tables.
Does Unity Catalog support column-level lineage?
Yes, Unity Catalog captures column-level lineage wherever it can resolve the mapping, which requires both source and target to be referenced by table name. Column lineage is not captured for path-based references such as delta."s3://bucket/path", user-defined functions can obscure the column mapping, and lineage is not preserved across renames. The Catalog Explorer view also extends only one level up and one level down from the selected column.
How long does Databricks keep lineage data?
Lineage shown in Catalog Explorer is retained indefinitely for data captured after September 1, 2024. The lineage system tables, system.access.table_lineage and system.access.column_lineage, maintain a rolling one-year window. That matters if you have dependencies that only fire on a quarterly or annual rebuild, because those edges can age out of the system tables you are querying.
Why do I need a lineage tool if Unity Catalog is free?
Most teams do not, at first. You need one when you hit a documented gap: a rename that breaks the chain, a UDF-heavy transform that blurs the column mapping, a path-based bronze read with no column lineage, or the moment you need transitive impact out to the BI dashboards Unity Catalog cannot see. Start free, and add a tool when the gap costs you an incident.
More use cases
Related features
Map your lineage, end to end
Connect your warehouse read-only and see your lineage map in minutes. Datatrail never moves or mutates your data.