Data Lineage Documentation: Best Practices and BCBS 239 Requirements

Data lineage documentation is the written, reviewable record of where each data element comes from, what happens to it on the way, and which reports or models consume it. For an audit it has to do three things: identify a single authoritative source per data element, explain every transformation between that source and the final number, and name the owner accountable for each hop. The part teams underestimate is currency. Documentation that was accurate at the time of writing and has not been refreshed since is the finding, not the evidence.

If you landed here because someone asked you to produce lineage documentation for BCBS 239, start with a fact that will save you an argument: the 2013 Basel Committee text everybody cites does not contain the word "lineage" anywhere in its 28 pages. What it asks for is documentation, reconciliation and a single authoritative source. Lineage is how you satisfy those, not a line item you can point at. The section below quotes the actual paragraphs.

What is data lineage documentation?

Lineage documentation is the human-readable layer on top of a lineage graph. The graph is the technical fact: this column was built from those columns by that query. The documentation is what makes the graph defensible to somebody who was not in the room, and it typically adds four things the graph alone does not carry.

• Business meaning. That fct_exposure.ead_usd is exposure at default in US dollars, not just a numeric column.
• Ownership. Who is accountable for the definition, and who is accountable for the pipeline.
• Transformation logic in words. Not the SQL, but what the SQL is doing and why, including the business rules and any exclusions.
• Known gaps. The manual steps, the spreadsheet in the middle, the feed that arrives a day late. Auditors respond far better to a documented weakness with a remediation plan than to a diagram that implies everything is automated.

That last point is worth sitting with, because it inverts how most teams approach the exercise. The goal is not a clean picture. It is an accurate one.

What BCBS 239 actually says about data lineage

BCBS 239 is the Basel Committee on Banking Supervision's Principles for effective risk data aggregation and risk reporting, published in January 2013. It sets out 14 principles across governance and infrastructure, risk data aggregation capabilities, risk reporting practices, and supervisory review. Banks designated as global systemically important by the FSB in November 2011 or November 2012 were required to meet the principles by January 2016, with later designations given three years from designation.

Read the document and you will not find the word "lineage". You will find these, which are what the lineage work is actually for:

• Paragraph 39, the documentation mandate. "Supervisors expect banks to document and explain all of their risk data aggregation processes whether automated or manual (judgement based or otherwise)." It continues: "Documentation should include an explanation of the appropriateness of any manual workarounds, a description of their criticality to the accuracy of risk data aggregation and proposed actions to reduce the impact."
• Paragraph 36(c), reconciliation. "Risk data should be reconciled with bank's sources, including accounting data where appropriate, to ensure that the risk data is accurate." A footnote defines the term: "reconciliation means the process of comparing items or outcomes and explaining the differences."
• Paragraph 36(d), the single source. "A bank should strive towards a single authoritative source for risk data per each type of risk."
• Paragraph 37, the dictionary. "As a precondition, a bank should have a 'dictionary' of the concepts used, such that data is defined consistently across an organization."
• Paragraph 33, metadata. Banks should establish "integrated data taxonomies and architecture across the banking group, which includes information on the characteristics of the data (metadata), as well as use of single identifiers and/or unified naming conventions".
• Paragraphs 35 and 38, the automation bias. "Data should be aggregated on a largely automated basis so as to minimise the probability of errors", tempered by "There should be an appropriate balance between automated and manual systems" where professional judgement is genuinely required.

So where did "BCBS 239 requires data lineage" come from? From supervisors, later, and from vendors repeating them. The Basel Committee's own newsletter of 6 January 2026 puts it directly: "Data lineage, or the traceability of data from its origin to its final use, is important for confirming data quality. This remains a challenging component of BCBS 239 for banks." The same newsletter lists data lineage as one of five recurring themes from its industry outreach sessions, and it is candid about why banks struggle: "Legacy systems, distributed data estates and the dynamic nature of data lineage complicate banks' efforts to confirm end-to-end data traceability. Finding appropriate vendor solutions and the resource-intensive nature of identifying and maintaining data lineage can further hinder progress."

Two honest caveats. That newsletter states explicitly that it "is for informational purposes only and does not constitute new supervisory guidance or expectations". And nobody at the Basel Committee has specified column-level granularity, despite what a lot of marketing pages assert. Where column-level detail becomes necessary is practical rather than regulatory: you cannot explain a specific number in a specific report at table granularity.

The newsletter also hands you your business case, almost verbatim: "demonstrating the business benefits of data lineage, such as cost reduction and improved efficiency, can help secure investment in automated tools."

How to document data lineage

Work backwards from the report, not forwards from the source systems. Every team that starts by cataloging its warehouse produces hundreds of pages nobody reads. Every team that starts from the twelve numbers on the regulatory return produces something defensible in a few weeks.

1. Fix the scope to specific reported figures. Name the report, then name the fields on it. Scope creep in a lineage exercise is what turns three months into three years.
2. Trace each figure upstream, hop by hop, and record the actual query. Not the intended design. The lineage graph the warehouse can generate is the ground truth here, and it will disagree with your architecture diagram.
3. Stop at the authoritative source and say so. Paragraph 36(d) wants one per risk type. If you find three, that is a finding you should document rather than a decision to make quietly.
4. Write the transformation in plain language beside the code. "Excludes intercompany positions and restates to USD at month-end WM/Reuters rates" is what a reviewer needs. The SQL is the evidence, not the explanation.
5. Flag every manual step explicitly. The spreadsheet, the emailed adjustment, the analyst override. Paragraph 39 asks for exactly this, plus the remediation plan.
6. Name an owner per hop, and record the reconciliation control. What is compared against what, how often, and who investigates a break.
7. Decide how it stays current before you publish. This is the step that gets skipped, and it is the one that determines whether the document is worth anything in eighteen months.

The organizational half of this is a separate exercise from mapping columns. The Basel Committee notes that a data-driven culture "remains a work in progress" at some banks and that periodic self-assessment helps, and plenty of teams run a structured assessment of process and culture maturity in parallel with the technical tracing work, because a perfect diagram cannot fix fragmented accountability.

Data lineage documentation template: what belongs in it

Most lineage documentation templates circulating as spreadsheets are close to right. A workable row-per-hop structure looks like this:

That final row is the one auditors read first and the one teams leave blank.

Data lineage documentation example

A single hop, filled in, so the shape is concrete. Suppose a credit risk report shows total exposure at default by counterparty.

Data element: risk_marts.credit.fct_exposure.ead_usd, "Exposure at Default (USD)". Business definition: expected gross exposure on a facility at the point of counterparty default, converted to USD. Upstream source: raw.loaniq.facility_balance.committed_amt from LoanIQ, the system of record for corporate lending. Authoritative: yes for corporate lending; retail exposures arrive from a separate source and are documented on their own row. Transformation: committed amount plus accrued fees, less eligible collateral per the netting rules in int_collateral_netting, converted at month-end rates; excludes intercompany facilities. Mechanism: automated dbt model, runs 02:15 ET daily. Owner: Head of Credit Risk Analytics (business), Risk Data Engineering (technical). Reconciliation: daily total compared against the general ledger corporate lending balance, tolerance 0.05 percent, breaks investigated by Risk Data Engineering within one business day. Known limitation: collateral valuations are received T+1, so intraday figures understate netting; remediation targeted for the next quarter. Last verified: reviewed and confirmed against the running pipeline this month.

Notice how much of that is not a diagram. Boxes and arrows are the least valuable part of lineage documentation, which is why an exported picture rarely satisfies a reviewer on its own. If you do need the visual, our lineage map covers what to put in one.

Why lineage documentation goes stale, and what to do about it

Hand-maintained lineage documentation decays for a structural reason: it is a snapshot of a system that changes every deployment. A model gets refactored, a column gets renamed, a new source is added, and none of those events update a spreadsheet. Within a couple of quarters the document describes a warehouse that no longer exists, and the more thorough the original effort was, the more expensive it is to re-verify.

The Basel Committee is describing precisely this when it calls out "the dynamic nature of data lineage" and "the resource-intensive nature of identifying and maintaining data lineage".

The way out is to split the artifact by how fast each part changes. The structural facts, which column came from which column through which query, should be generated from the warehouse itself, because that is the only version that cannot drift out of date. The interpretive layer, business definitions, ownership, reconciliation controls and known limitations, is genuinely human and changes slowly, so maintaining it by hand is reasonable. Teams who automate the first half and hand-write the second half end up with documentation that survives. Teams who hand-write both end up re-doing the exercise before every examination.

In practical terms that means your reconciliation controls and owner assignments live in a document you curate, while the hop-by-hop map is pulled from query history on demand, and the "last verified" date becomes a real check rather than a promise. For the mechanics of producing that map, how to build data lineage compares the manual, dbt and automated routes, and data lineage vs data catalog untangles which system should hold the business definitions.

Data lineage documentation tools

Tooling for this splits into three groups, and buying from the wrong one is a common and expensive mistake. Governance suites such as Collibra and Alation are built around the human layer, the glossary, the policies and the workflow, and they are what large regulated programs typically standardize on. Code-parsing lineage engines, chiefly IBM Manta and Cloudera Octopai, read stored procedures and legacy ETL, which is why banks with mainframe and on-premises estates keep buying them. Warehouse-native lineage from Snowflake, BigQuery and Databricks Unity Catalog is accurate and free of extra licensing, but each stops at the edge of its own platform and every one carries a retention floor, so none of them can evidence a period that predates the feature.

Whichever you pick, the documentation obligation in paragraph 39 does not go away. No tool writes your reconciliation controls or explains why a manual workaround is appropriate. Our comparison of data lineage tools covers the field in detail, and metadata management tools covers the wider governance layer.

Keeping the map current without re-doing the exercise

This is the part Datatrail is built for. It connects to your warehouse with a read-only role and parses query history along with your dbt manifest into a column-level lineage graph, so the structural half of your documentation is derived from what actually ran rather than from what somebody believed was running. Because it reads execution rather than repository code, it also catches the ad hoc queries and scheduled jobs that live outside dbt, which is usually where undocumented manual workarounds hide.

From the same graph it computes downstream impact, so before a column changes you can see every model, exposure and dashboard that reads it, which is the reconciliation question asked in advance. Schema change alerts tell you when a definition moved underneath your documentation, which is the event that silently invalidates a spreadsheet. Pricing is published, the connection never moves or mutates data, and you can see your own lineage the same afternoon.