Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Gray failure in the data plane

How does a data job succeed with wrong output, and what can detect it?

In ordinary performance work, is seen as a performance phenomenon. In a data system the more dangerous form is a correctness phenomenon with the same epistemics: the job that succeeds and produces incomplete or wrong output.

Examples, once you name them:

  • A source connector silently drops records after a schema mismatch and exits with code zero.
  • A partition-pruning filter on a mistyped column matches nothing, so the query returns quickly, correctly and empty.
  • An upstream partition arrives late, so the job reads what exists and writes a fact table that is missing a region.
  • A join key changes cardinality (how many rows match each key) and rows are silently duplicated.
  • A permissive parser turns failures into null.

In every case: exit code zero, no exception, orchestrator green, downstream consumers served. This is exactly , moved to data. The job's self-report ("I completed") measures the process. The consumer's experience measures the data. The gap is the same gap, and it is worse here for a specific reason: the channel at least degrades continuously and can be caught by a tail-latency alert, whereas incorrect data is identical in form to correct data and carries no signal at all.

Detection cannot come from the executing job, because the job's idea of success is control-flow completion. It must come from a comparison against something external: expected row counts, source-to-target reconciliation, volume and distribution baselines, freshness. That is why data validation and freshness monitoring are not quality niceties but the only available for this class. It is worth carrying this reframing: data-quality checks are the health checks of the , and a pipeline without them has no failure detector at all.

-- A reconciliation check: the nightly job is "healthy" only if every city is present
-- and the row count is no more than 20% below the same weekday last week.
SELECT city
FROM expected_cities e
LEFT JOIN revenue_report r USING (city, report_date)
WHERE r.city IS NULL                      -- a missing city is an alert
   OR r.trip_count < 0.8 * (SELECT trip_count FROM revenue_report p
                            WHERE p.city = e.city
                              AND p.report_date = r.report_date - 7);