Unity Catalog Lineage: What Databricks Data Lineage Captures and What It Misses

Unity Catalog captures data lineage automatically for anything that runs on Databricks, down to the column level, across every workspace attached to the metastore. It covers tables, views, notebooks, jobs, pipelines, queries, dashboards, and ML model versions, and it is free. It also has documented limits that catch teams by surprise: column lineage drops out for path-based reads and user-defined functions, lineage is not preserved when you rename anything, and the lineage system tables keep only a rolling one-year window.

Here is what the native graph actually gives you, exactly where it stops, and what to do about the gaps.

What Unity Catalog lineage captures

Turn on Unity Catalog and lineage capture is automatic. There is no agent to install and no manifest to maintain. When a query runs, Databricks records which objects it read and which it wrote, and it does so at the column level wherever it can resolve the mapping.

The coverage is broader than most people expect:

• Tables and views, including the column-to-column edges between them.
• Notebooks, jobs, pipelines, and queries, so you can see not just that gold.fct_orders depends on silver.orders, but which job did the transforming.
• Dashboards that read the tables, which is a real advantage over lineage that stops at the warehouse edge.
• ML model versions, external assets, and file paths.
• Cross-workspace: lineage is aggregated across every workspace attached to the same Unity Catalog metastore, subject to permissions, so a table transformed in one workspace and consumed in another still connects.

Lineage in Catalog Explorer is retained indefinitely for data captured after September 1, 2024, and it is queryable programmatically through the system tables system.access.table_lineage and system.access.column_lineage. If you want to build something on top of it, those two tables are the API.

For a lot of teams this is genuinely enough, and the honest advice is to start here before you buy anything. It costs nothing and it is already on.

Where Unity Catalog lineage stops

The limits are published by Databricks, which is to its credit, but they are easy to miss until one of them bites you.

Column lineage needs table names, not paths

Column-level lineage is captured only when both the source and the target are referenced by table name, as in select * from catalog.schema.table. If a query reads a path directly, such as select * from delta."s3://bucket/path", the column mapping cannot be resolved. Any lakehouse with a bronze layer that still reads raw files by path will have holes at exactly the point where the data enters the platform.

User-defined functions obscure the mapping

A UDF is a black box to the lineage parser. If revenue comes out of calculate_net(gross, refunds), Databricks can see that the query read those source columns, but the clean column-to-column derivation gets fuzzy. Heavily UDF-driven transforms produce lineage that is technically present and practically vague.

Renames break the chain

This is the one that surprises people. Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns. Rename orders_v2 to orders and the history connecting it to everything upstream does not follow. Given that renaming things is one of the most common reasons you would go looking for lineage in the first place, it is an awkward gap.

The one-year window

Lineage system tables maintain a rolling one-year window of data. If a table is rebuilt annually, or a dependency only fires during a quarterly close, that edge can age out of the system tables and simply stop being visible to anything you built on top of them.

The visualization shows one hop

In the Catalog Explorer UI, the column-level view extends one level up and one level down from the column you selected. That is fine for orientation and useless for impact analysis, which is transitive by definition: if A feeds B and B feeds C, changing A puts C at risk, and one hop will never show you C.

It ends at the edge of Databricks

Unity Catalog knows about Databricks. It does not know about the dbt project that runs against it from CI, the reverse-ETL job syncing a gold table to Salesforce, or the Tableau workbook the CFO opens on Monday. Lineage that does not reach the consumer cannot tell you what a change will actually break for a human being.

Does Unity Catalog give you column-level lineage?

Yes, Unity Catalog captures column-level lineage automatically for queries run on Databricks, as long as both source and target are referenced by table name. It is real column lineage, not table lineage with a column tab. The caveats are the ones above: path-based reads, UDFs, and renames all degrade or break the column mapping, and the UI visualization only shows one level up and one level down from the selected column. Streaming lineage between Delta tables requires Databricks Runtime 11.3 LTS or above, and column lineage for Lakeflow pipeline workloads requires 13.3 LTS or above.

How to fill the gaps

You have three realistic options, and they are not mutually exclusive.

Build on the system tables. system.access.column_lineage is queryable, so you can write recursive SQL to walk the graph more than one hop and materialize your own snapshot before the one-year window rolls over. This works. It also means you now own a piece of internal data infrastructure, and the person who wrote it will eventually leave. Worth doing if your needs are narrow and your team is large.

Fix the source patterns. Some of the gaps are self-inflicted. Reading by table name instead of by path, preferring SQL expressions over UDFs in transformation logic, and avoiding renames in favor of new columns with a deprecation window will all materially improve what Unity Catalog can capture. This is good hygiene regardless of what tooling you buy.

Layer a lineage tool on top. The gaps that remain (renames, transitive impact, and everything downstream of Databricks) are structural, and no amount of hygiene closes them. This is where a dedicated lineage tool earns its keep, and it is worth comparing the available data lineage tools rather than defaulting to the biggest name.

What Datatrail adds on Databricks

Datatrail reads Unity Catalog lineage and your query history together, with a read-only connection that never moves or mutates a row. It parses the SQL itself, so a column edge survives a rename or a UDF that the native graph loses. It keeps its own persistent graph, so a quarterly dependency does not age out of a rolling window. And it walks the chain transitively, out past the lakehouse into the dbt models and BI dashboards that read your gold layer.

The payoff is impact analysis that answers the real question. Not "what is one hop downstream of this column" but "if I drop this field, which dashboards go wrong and who opens them", computed before you merge the change rather than after someone notices a wrong number. On top of the same graph sit freshness monitoring and schema change alerts, so an upstream change that lands on you unannounced pages you with the affected assets attached.

One practical note while you are in the lakehouse: lineage and spend are closely related, because the tables nobody reads are usually the ones quietly costing you the most compute. A dependency graph is the fastest way to find them, and pairing it with a view of where your cloud spend actually goes tends to pay for itself before the reliability benefits do.

To see the whole picture for Databricks, read the Databricks data lineage page, or go deeper on how field-level tracing works in column-level lineage explained.