BigQuery Data Lineage, Column Level and Kept Longer Than 30 Days

What BigQuery already does well, and what it is called now

Start with the correction, because most pages on this topic are wrong. BigQuery native lineage is column-level. It went generally available on 29 September 2025 and covers CREATE TABLE, CREATE TABLE COPY, INSERT, UPDATE, MERGE, DELETE, and SELECT with a destination table. If a comparison page tells you BigQuery only does table-level lineage, it was written before that date and nobody updated it.

The naming is genuinely confusing right now, which is its own reason people struggle to find the docs. Data Catalog was deprecated in February 2025 and scheduled for discontinuation on 1 June 2026. The catalog role moved to Dataplex, which became Dataplex Universal Catalog, which as of 10 April 2026 is called Knowledge Catalog. Through all of it the API and IAM identifiers stayed put: you enable datalineage.googleapis.com and grant roles/datalineage.viewer, no matter what the console header says.

Turn it on and lineage is captured automatically per project for query jobs, copy jobs, and load jobs from Cloud Storage, plus Dataflow, Spark, Composer, and Looker. For a team living entirely inside Google Cloud that needs a governance view, this may be all you need, and we would rather say that than pretend a free built-in feature does not exist.

The five documented limits that send teams looking

These are all from Google's own documentation, not our opinion of the product.

• 30-day retention. The docs are unambiguous: lineage information is retained for 30 days. A model rebuilt monthly or quarterly has no lineage most of the year. This is usually the one that hurts, because lineage is most valuable precisely for the pipelines nobody has touched recently and nobody remembers.
• The 1,500-link ceiling. Column-level lineage is not collected if a job creates more than 1,500 column-level links. Read that again, because the failure mode is the problem: it does not warn you, it does not partially resolve, it just stops producing column detail on your largest transformations. The wide fact table assembled by your most complex job is the most likely to cross the line and the one you most wanted resolved.
• Up to 24 hours of latency. Lineage appears within 24 hours of the job finishing. That is fine for an audit and useless for a check you run before merging a pull request.
• Top-level columns only. Nested fields inside STRUCT or JSON types are not tracked, and external tables get no upstream column-level lineage.
• Capture gaps. Recurring load jobs from the BigQuery Data Transfer Service are not automatically recorded, and neither is direct lineage for BigQuery routines, so stored procedures are a blind spot.

And it is billed, not free. Lineage runs on the Dataplex premium processing SKU at $0.089 per DCU-hour plus $2 per GiB of metadata storage, and Google states the free tier applies only to the standard processing SKU, not the premium one. Their own worked example puts 100 DCU-hours plus 1 GiB at $10.90. That is not expensive, but no one should plan around it being free.

The DIY route, and what it costs you

The other native path is to build lineage yourself from INFORMATION_SCHEMA.JOBS, which keeps job history for 180 days, six times the lineage API window. Each row gives you referenced_tables, the destination table, the job type, and the full SQL text. Query it with a region qualifier and you can reconstruct a table-level graph of everything that ran.

That gets you durable table-level lineage for the cost of a scheduled query. Getting from there to the column means writing a SQL parser that handles joins, aliases, CTEs, window functions, CASE expressions, and SELECT * expansion against the schema as it existed when the query ran, across every dialect quirk in your codebase. Teams do build this. It is a real engineering project with a real maintenance tail, and it is worth being clear-eyed about that before you start.

Datatrail is that parser, kept current, plus the graph on top. It connects to BigQuery with a read-only role, parses query history into column-level lineage with no link ceiling, keeps the graph rather than expiring it, and follows the column downstream into the BI layer. Then impact analysis lists every model, exposure, and dashboard that reads a field before you change it. The step-by-step version of the native options is in our BigQuery data lineage guide, and if you run more than one warehouse, Snowflake and Databricks land in the same graph.