Data Incident Management: Detect, Trace, and Resolve Faster

What counts as a data incident

The word covers more ground than most runbooks admit, and the categories fail differently enough to be worth separating.

• The pipeline did not run. Loudest and easiest. Orchestrator failed, job errored, somebody gets an alert from a system that already exists.
• The pipeline ran and did nothing. The job succeeded, the extract was empty, the table is stale and green. No error anywhere.
• The data changed shape. An upstream vendor renamed a field or changed a type. Your pipeline is fine. The model reading it is not.
• The values are wrong. Everything loaded, on time, in the right shape, and a currency conversion has been applied twice. This is the worst category because nothing technical is failing at all.

Only the first shows up in ordinary infrastructure monitoring. The other three are silent by construction, which is why data teams so often learn about incidents from a person rather than a system.

The four stages of data incident response

Borrowed from software incident practice and adapted, because the failure modes are genuinely different.

1. Detect. A check fires, or somebody notices. The gap between those two is the single most informative number about a data platform.
2. Scope. What does this affect? Which dashboards, which models, which downstream teams, and is anything customer-facing. This is where the hours go without a dependency graph, and it is the stage most runbooks skip entirely.
3. Diagnose. Walk backward to the change that caused it. The symptom is almost never where the cause is: a wrong number on a dashboard usually originates four models upstream, and often in a table nobody on the call has opened before.
4. Resolve and prevent. Backfill, correct, communicate, then add the check that would have caught it. Skipping the last step is how the same incident recurs quarterly.

Stages two and three are both lineage questions, which is why teams with a real dependency graph resolve incidents in a fraction of the time regardless of which alerting tool they bought. Tracing backward is covered in more depth in our guide to root cause analysis for data pipelines.

Measuring whether incident response is actually improving

Two numbers are worth tracking, and neither is the count of incidents. Incident count mostly measures how much you are looking.

Mean time to detection is the gap between when the data broke and when anyone knew. If the honest answer is "a stakeholder told us", that is your detection mechanism and it is worth being blunt about it in a review. Mean time to resolution is detection to fix, and it decomposes neatly into the four stages above, which is useful because it shows where the time actually goes. For most teams that adds a dependency graph, the stage that collapses is scoping rather than diagnosis.

A useful third measure is the share of incidents found by a check rather than by a human. Moving that ratio is a better goal than driving incident count to zero, which mostly encourages people to stop reporting things. The broader question of how to score data health is covered in data quality dimensions and metrics, and the cost of not measuring it in what data downtime is.

Datatrail shortens detection with freshness and schema change alerts that fire on the upstream table that broke, and shortens scoping with column-level lineage that lists the affected models and dashboards in the alert itself.