Data Observability: The Engineering You're Probably Missing

Your pipeline ran successfully. Your data is wrong.

This happens more often than anyone admits. A source system changes a column type. A timezone shift silently doubles records. A third-party feed goes quiet for six hours and nobody notices until the CFO asks why revenue is down 40% in a dashboard.

The gap between "pipeline health" and "data health" is where most enterprise data platforms fail silently. Your orchestrator says everything is green. Your warehouse is full of garbage. I've seen this pattern destroy trust in data teams faster than any technical failure.

Data observability is the engineering discipline that closes this gap. Not a tool. Not a dashboard. A discipline.

The Five Pillars#

Every production data platform needs monitoring across five dimensions. These aren't theoretical categories; they're the things that actually break.

Freshness. Is the data arriving on time? If your finance table usually refreshes by 06:00 UTC and it's 08:30 with no update, something is wrong. Freshness is the easiest signal to implement and the most impactful. Most data incidents are discovered because someone noticed stale data in a report.

Volume. Are you getting the expected amount of data? A table that normally receives 500K rows per day suddenly receiving 50K is a problem. So is receiving 5M. Volume anomalies catch issues that schema validation and pipeline health checks miss entirely.

Distribution. Do the values look right? If a country_code column that's historically 85% US suddenly becomes 60% US, something changed upstream. Distribution checks catch the subtle corruptions that volume checks miss; the row count is fine, but the data inside is wrong.

Schema. Has the structure changed? Added columns are usually harmless. Dropped columns break things. Type changes break things quietly. Schema monitoring should block on breaking changes and alert on additive ones.

Lineage. Where did this data come from, and what depends on it? Lineage isn't a monitoring signal; it's the map you need when something goes wrong. Without it, every incident becomes a manual investigation of "what else did this break?"

The Monitoring Stack That Works#

Don't try to monitor everything equally. Tier your tables by business criticality.

• Tier 1: Revenue, compliance, customer-facing. If this data is wrong, someone gets a phone call. Full observability on all five pillars.
• Tier 2: Operational analytics, internal reporting. If this data is wrong, teams make bad decisions. Freshness, volume, and schema monitoring.
• Tier 3: Experimental, exploratory, staging. If this data is wrong, someone might notice eventually. Basic freshness checks.

Start with Tier 1. Get it right. Then expand.

A Practical Starting Point#

Freshness monitoring doesn't require a vendor. Here's a pattern I've used with Airflow and a simple metadata table:

-- Track when each table was last updated
CREATE TABLE IF NOT EXISTS data_observability.table_freshness (
    table_name      VARCHAR(256),
    last_updated_at TIMESTAMP,
    expected_by     TIMESTAMP,
    check_time      TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Run this as an Airflow sensor before downstream dependencies
-- If the table hasn't been updated within its SLA window, fail the sensor
SELECT
    table_name,
    last_updated_at,
    expected_by,
    DATEDIFF('minute', expected_by, CURRENT_TIMESTAMP) AS minutes_overdue
FROM data_observability.table_freshness
WHERE last_updated_at < expected_by
  AND DATEDIFF('minute', expected_by, CURRENT_TIMESTAMP) > 30;

Wire this into an Airflow sensor that gates downstream DAGs. If your Tier 1 tables are late, nothing downstream runs. This is blunt, but it prevents the most common failure mode: stale data flowing into dashboards without anyone knowing.

Layer in volume and distribution checks as you mature. But freshness and schema on Tier 1 tables will catch 70% of incidents on day one.

Alert Strategy That Doesn't Burn Out Your Team#

Alert fatigue is the number one killer of observability programmes. I've watched teams build beautiful monitoring that generates 200 alerts per week, and within three months every alert gets ignored.

Classification#

• P1: Tier 1 data is missing, wrong, or stale. Pages someone immediately. Response SLA: 15 minutes.
• P2: Tier 2 data issues or Tier 1 anomalies that haven't breached SLA yet. Sends a Slack message. Response SLA: 2 hours.
• P3: Tier 3 issues, minor anomalies, informational. Goes to a channel that gets reviewed daily.

Rules That Keep Alerts Useful#

• Every alert must have a runbook. If there's no documented response, it's not an alert; it's noise. Before creating any alert, write the three-step action someone should take when it fires.
• Use composite alerts. A single metric breaching a threshold is often a false positive. Two or three signals firing together (freshness delay + volume drop) is almost certainly real. Composite alerts reduce noise dramatically.
• Assign ownership. Every alert has a team or individual owner. Unowned alerts are unanswered alerts.
• Review monthly. Every month, look at which alerts fired, which were actionable, and which were false positives. Tune or remove the noisy ones. If an alert hasn't been actionable in three months, delete it.

The target is fewer than 10 actionable alerts per week for a mid-sized platform. If you're above that, you're either not tiering properly or your thresholds need tuning.

Cost Observability#

The forgotten pillar. Most teams have no idea what their data platform actually costs per team, per pipeline, or per query.

Cloud warehouses charge by compute. The database and warehouse you choose determines the cost model, but without attribution, you can't answer basic questions: Which team is driving 60% of our Snowflake bill? Is that new dashboard costing us $3,000/month in queries? Should we materialise this view or keep it virtual?

Practical Implementation#

In Snowflake, tag every query with a comment containing the team and purpose:

-- Add to every scheduled query
ALTER SESSION SET QUERY_TAG = '{"team": "finance", "pipeline": "daily_revenue", "tier": "1"}';

Then build a daily roll-up of costs by tag. Snowflake's QUERY_HISTORY view gives you credits consumed per query, which maps directly to dollars.

In Redshift, use query groups and WLM queues to separate workloads, then join STL_QUERY with STL_QUERY_METRICS for attribution.

In BigQuery, use labels on scheduled queries and reservation assignments for cost allocation.

The goal isn't to police spend. It's to make costs visible so teams can make informed trade-offs. Show teams their weekly cost trends and you'll be surprised how quickly they start optimising their own queries.

The Incident Response Playbook#

When data breaks (and it will) you need a process, not heroics. After years of handling data incidents, this is the playbook that works.

1. Detection. Ideally automated via your monitoring stack. Worst case, a business user reports incorrect numbers. Track your detection method for every incident; the ratio of automated to manual detection is your observability maturity score.

2. Containment. Stop the bleeding. Pause downstream pipelines that depend on the affected data. Use lineage to assess blast radius: if your orders table is corrupted, what dashboards, ML models, and downstream tables are affected? Communicate early; a brief "we're aware and investigating" message buys you time and trust.

3. Root cause analysis. What changed? Common culprits: schema change from source, data volume spike/drop from upstream, timezone or encoding issues, infrastructure failures. Lineage and audit logs are your friends here.

4. Resolution. Fix the issue. Backfill affected data. Validate that downstream systems are consistent. Don't skip the validation step; partial fixes that leave inconsistencies across tables create worse problems than the original incident.

5. Post-mortem. Within 48 hours, document what happened, the timeline, root cause, and what you'll change. Blameless. Focus on systemic improvements: could we have detected this earlier? Should this table be Tier 1? Do we need a new check?

Every post-mortem should produce at least one concrete improvement to your observability stack. Otherwise, you'll have the same incident again.

The Tool Landscape (Honest Assessment)#

There's no shortage of tools. Here's how I think about the decision.

| Approach | Best For | Limitations | |---|---|---| | dbt tests | Teams already using dbt. Schema, uniqueness, not-null, accepted values. | No freshness, no anomaly detection, no lineage visualisation. Baseline, not sufficient alone. | | Elementary | dbt-native shops wanting more. Adds anomaly detection and lineage on top of dbt. | Tied to dbt. Won't cover sources outside your dbt project. | | Great Expectations | Complex validation logic, non-dbt pipelines, Python-heavy teams. | More setup overhead. You're building and maintaining the framework. | | Monte Carlo | Enterprise teams wanting managed observability. Coverage across the full stack. | Expensive. Vendor lock-in risk. You're paying for convenience. | | Custom built | Specific needs that tools don't cover, or teams with strong platform engineering. | You own the maintenance burden forever. |

My recommendation: start with dbt tests as your baseline. Add Elementary or Monte Carlo depending on budget and team size. Build custom only for gaps that no tool covers, usually cost attribution and domain-specific business rule validation.

Don't buy a tool before you've defined what you're monitoring and why. A well-configured set of dbt tests with clear ownership will outperform an expensive observability platform that nobody has tuned.

Starting Point, Not End State#

Data observability isn't a tool purchase. It's an engineering discipline that matures alongside your data architecture. It is one piece of a larger data architecture puzzle that also includes pipeline engineering, database selection, and governance.

Start with your Tier 1 tables. Implement freshness and schema monitoring. Write runbooks for every alert. Run post-mortems for every incident. Build outward from there.

The goal isn't zero incidents; that's impossible. The goal is fast detection and fast recovery. The best data teams I've worked with don't have fewer incidents than average. They detect them in minutes instead of days, contain them in hours instead of weeks, and learn from every one.

There's an emerging discipline forming around this: data reliability engineering. It borrows from site reliability engineering (SLOs, error budgets, incident response) and applies it to data systems. If you're a data engineer thinking about where to invest your career, this is a space worth watching.

Build the boring fundamentals first. The monitoring, the alerts, the runbooks, the post-mortems. They're not glamorous, but they're the difference between a data platform that people trust and one they work around.

Frequently Asked Questions#

What is the difference between data observability and data monitoring?#

Data monitoring checks whether pipelines ran successfully, while data observability checks whether the data itself is correct. Your orchestrator can show all green while your warehouse is full of garbage. Data observability covers five pillars: freshness (is data arriving on time), volume (is the expected amount arriving), distribution (do values look right), schema (has structure changed), and lineage (where did data come from and what depends on it).

What are the most important data observability metrics to track?#

Start with freshness and schema monitoring on your Tier 1 (business-critical) tables, as these two checks alone catch roughly 70% of data incidents on day one. Freshness monitoring detects stale data before it reaches dashboards, and schema monitoring catches breaking structural changes from upstream systems. Layer in volume anomaly detection and distribution checks as your observability practice matures.

How do you implement data observability without expensive tools?#

Start with dbt tests as your baseline for schema validation, uniqueness, null checks, and accepted values. Add freshness monitoring using a simple metadata table in your warehouse combined with Airflow sensors that gate downstream DAGs when Tier 1 tables are late. Build cost attribution by tagging queries with team and pipeline metadata. Only consider commercial tools like Monte Carlo or Elementary after you have defined what you are monitoring and why.