What Is Data Observability? The Five Pillars Explained

Data observability is the practice of continuously measuring the health of your data so you can detect, diagnose, and resolve problems before they reach a dashboard or a stakeholder. It is usually broken into five pillars: freshness, volume, schema, distribution, and lineage. Together they answer one question that traditional infrastructure monitoring cannot: is the data itself correct, current, and complete, not just is the pipeline green?

Why monitoring is not enough

Classic monitoring watches systems. It tells you that a job finished, that CPU is fine, that the warehouse is up, and that the dbt run exited zero. None of that tells you whether the numbers are right. A pipeline can succeed perfectly while loading half the rows it should, or while silently dropping a renamed column, or while delivering yesterday's data because an upstream source never refreshed.

Data observability looks at the data itself. The pipeline can be green and the data can still be wrong, and observability is the discipline of catching that gap. The five pillars are the standard way to organize what you measure.

Pillar 1: Freshness

Freshness asks: how recently was this table updated, and is that within the window we promised? Every important table has an implicit or explicit freshness SLA. A table loaded hourly should not be six hours stale; a daily mart should land before the morning standup.

You measure freshness from the warehouse metadata. In Snowflake you can read the last load time from INFORMATION_SCHEMA.TABLES or infer it from QUERY_HISTORY; BigQuery exposes last-modified timestamps in INFORMATION_SCHEMA; dbt source freshness checks compare a loaded_at column to a threshold. When raw.stripe.charges has not loaded since midnight and it normally lands hourly, that is a freshness incident. Freshness monitoring is often the highest-value pillar because stale data is the most common cause of a wrong dashboard.

Pillar 2: Volume

Volume asks: did we get roughly the number of rows we expected? A table that normally loads 200,000 rows a day and suddenly loads 4,000, or 4,000,000, is signaling a problem upstream. Row-count anomalies catch partial loads, duplicate loads, broken filters, and source outages that freshness alone misses, because a partial load can still be recent.

Good volume monitoring is not a fixed threshold. It learns the normal range, including weekly seasonality, so a quiet Sunday does not page anyone but a real 90 percent drop on a Tuesday does. This is where anomaly detection on row counts earns its keep.

Pillar 3: Schema

Schema asks: did the structure of the data change? Columns get added, dropped, renamed, or retyped, often by an upstream team that has no idea who depends on them. A column rename from email to email_address can silently null out a downstream join. A type change from INTEGER to STRING can break a cast in a staging model.

Schema observability compares the current structure of each table against its last known structure and flags schema drift: new columns, removed columns, renamed columns, and type changes. The real value comes from pairing the alert with lineage, so a schema change alert tells you not only that customers.email was renamed but exactly which models and dashboards reference it.

Pillar 4: Distribution

Distribution asks: do the values inside the columns look right? This is the most data-aware pillar. Even when a table is fresh, full, and structurally unchanged, the values can go wrong:

• A null rate that jumps from 1 percent to 40 percent on orders.customer_id.
• A revenue column that suddenly contains negative values or zeros.
• A currency field where amount shifts by a factor of 100 because someone changed cents-to-dollars handling.
• A categorical column where a new, unexpected value appears.

Distribution checks track statistics per column over time, such as null percentage, min and max, mean, and cardinality, and flag when they move outside their normal band. This is where you catch the subtle data-quality bugs that no pipeline status would ever reveal. Many teams formalize these as data-quality checks tied to specific columns.

Pillar 5: Lineage

Lineage is the pillar that ties the other four together. The first four tell you that something is wrong; lineage tells you what it affects and where it came from. A freshness incident on raw.stripe.charges is just noise until lineage shows that fct_orders and the revenue dashboard depend on it.

Lineage maps every asset to its upstream sources and downstream consumers, at the table and ideally the column level. With it, every alert arrives with context: the blast radius, the affected dashboards, and the chain back to the original source. Without it, your team triages alerts blind. This is why observability platforms treat the lineage map as the spine of the whole system rather than a nice-to-have.

How the pillars work together

In practice the pillars cascade. A source outage causes a freshness miss. A partial recovery causes a volume drop. An upstream team's fix introduces a schema change. That schema change shifts a distribution. And lineage connects all of it to the three dashboards your finance team checks every morning. A real observability practice watches all five continuously so that the first signal triggers a response instead of a stakeholder noticing a broken chart hours later.

Observability without writing to your warehouse

None of these checks require touching your data. Freshness, volume, schema, and distribution are all measured by reading warehouse metadata, system tables, and lightweight aggregate queries, while lineage is reconstructed from query history and dbt metadata. A proper observability tool connects read-only by design and never moves or mutates a single row.

Datatrail implements all five pillars against a read-only connection to Snowflake, BigQuery, Redshift, or Postgres, with lineage at the center so every freshness, volume, schema, and distribution signal arrives with its downstream impact already mapped. If you want to see what your warehouse's health looks like across the five pillars, you can explore how it connects and what it surfaces against your own data, or compare the main data observability tools before you commit to one.