Redshift Data Lineage, Column-Level and Kept Past Seven Days
Redshift records every query that ran, then deletes the log after about a week. Datatrail reads that history read-only, parses it into column-level lineage, and keeps the graph so it is still there when you actually need it.
Read-only connection. Datatrail never moves or mutates your data.
In short
Redshift data lineage maps how data moves between Redshift tables and columns, from raw ingestion through each transformation to the views that feed reporting. Redshift itself does not build a lineage graph: it exposes query history in SYS_QUERY_HISTORY and the older STL_QUERY system tables, but STL logs are retained for only about seven days, and turning that log into a column-level map is left to you or to Amazon SageMaker and DataZone, which run daily lineage captures as part of a broader governance platform. Datatrail connects to Redshift read-only, parses the query history into a persistent column-level graph, and follows each column out to the dbt models and BI dashboards that depend on it, so impact analysis covers the whole path.
Why it fits
Redshift teams who need column-level lineage that outlives the seven-day log and reaches the dashboards.
Before the log expires
Redshift STL query logs are kept for only about seven days, so the pipeline you need to trace during an incident may already be gone. Datatrail keeps the graph as long as you do.
Column-level, from the SQL
Datatrail parses the SQL in SYS_QUERY_HISTORY into column-level edges, so you trace an individual field through each CTAS and view, not just table to table.
Past the edge of AWS
Native AWS lineage lives inside SageMaker and DataZone. Datatrail follows the column into dbt, Looker, Tableau, and the reverse-ETL syncs where a break is actually noticed.
What Redshift gives you natively, and what it does not
Redshift keeps a detailed record of what ran. Every statement lands in the system tables, and the modern place to read it is SYS_QUERY_HISTORY, the unified monitoring view that carries the query text, timing, and the user and database for each query. The older STL_QUERY and the STL family expose similar detail. So the raw material for lineage, the SQL of everything that touched the warehouse, is there.
Two things it does not give you. First, it is a log, not a graph. Redshift does not assemble those queries into a lineage map, resolve columns through views, or walk dependencies transitively. Second, and this is the one that catches people, the STL logs are retained for only about seven days, and often closer to two to five depending on cluster activity. The lineage you most want to trace is usually for a pipeline nobody has touched in weeks, which is exactly the lineage Redshift has already deleted.
The AWS answer: SageMaker and DataZone lineage
AWS does have a native lineage story, it just does not live in Redshift itself. Data lineage went generally available in Amazon DataZone and the next generation of Amazon SageMaker in December 2024, and it automatically captures lineage from AWS Glue and Amazon Redshift. In SageMaker Unified Studio the lineage runs execute queries against the Redshift system tables on a schedule, commonly daily, and store the resulting events in the catalog, with column-level detail shown where it can be resolved.
If you are standardizing on SageMaker Unified Studio or DataZone as your governance layer, that is a reasonable path and it is built in. The trade is what you are adopting to get it: a broader catalog and governance platform, a daily capture cadence rather than something you can check against a pull request in the moment, and a graph scoped to the AWS services it integrates with. For a team that just wants to know what a Redshift column change will break, that can be a lot of platform to take on for one answer.
What Datatrail does with Redshift query history
Datatrail connects to Redshift with a read-only role, reads SYS_QUERY_HISTORY and the STL views, and parses the SQL into column-level lineage before the seven-day window erases it. It keeps its own persistent graph, so a table that rebuilds monthly still has lineage when you go looking, and it reconciles the result against your dbt models and exposures.
From there the graph extends the way it does on our other warehouses: out past Redshift into the dbt models and the Looker, Tableau, and reverse-ETL layer where a broken column is actually noticed. Impact analysis then lists every downstream model and dashboard that reads a field before you alter it, and schema change alerts catch the upstream changes that would otherwise land unannounced. Nothing is written back to Redshift. If you run more than one warehouse, Snowflake and BigQuery land in the same graph, and the wider field is in our comparison of data lineage tools.
Questions people ask
Redshift lineage, answered
Does Amazon Redshift have built-in data lineage?
Not as a graph. Redshift records every query that ran in SYS_QUERY_HISTORY and the older STL system tables, which is the raw material for lineage, but it does not assemble that into a browsable column-level map or follow dependencies downstream. AWS provides native lineage through Amazon SageMaker Unified Studio and DataZone, which run scheduled captures against the Redshift system tables. Redshift on its own gives you the logs, not the lineage.
How long does Redshift keep query history?
The STL system views retain about seven days of log history, and under heavy cluster activity it can be closer to two to five days. SYS_QUERY_HISTORY gives you a cleaner unified view of recent query activity, but it is still a monitoring window, not a long-term archive. Because lineage is most valuable for pipelines nobody has touched recently, teams often parse and persist the history externally so the graph outlives the log.
How do I trace a column back to its source in Redshift?
Pull the queries that wrote to your target table from SYS_QUERY_HISTORY or STL_QUERY, read their SQL, and follow the derivation upstream. For a single column one or two hops back, that works. It stops scaling when the derivation is several hops deep, involves views or CTEs, or the query that built the table has already aged out of the roughly seven-day retention window. Automated column-level lineage keeps a persistent graph so the trace is one click instead of an archaeology project.
What is the difference between Redshift lineage in DataZone and a dedicated tool?
Amazon DataZone and SageMaker Unified Studio capture Redshift lineage as part of a broader governance and catalog platform, on a scheduled daily cadence, scoped to the AWS services they integrate with. A dedicated tool like Datatrail focuses on the lineage and impact question: it connects read-only, parses query history into a persistent column-level graph, and extends it past AWS into dbt and the BI layer, so you can check the blast radius of a change against a pull request rather than waiting for the next daily run.
More use cases
Related features
Map your lineage, end to end
Connect your warehouse read-only and see your lineage map in minutes. Datatrail never moves or mutates your data.