Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Correct per arrival, and progress

Why are datasets correct per arrival rather than per request, and why are ETAs so often wrong?

Services are correct per request; datasets are correct per arrival

A service fails per request: one request in a thousand returns an error. A is correct per arrival: each time new data should arrive, it either does or does not, and either arrives right or wrong. A pipeline can have 100% availability and still deliver stale or wrong data. So we need observables that measure the data, not the job. The main ones are progress, , lag, volume and .

Progress and ETA

. Both terms are estimates, and each is wrong in its own way.

  1. Remaining work is unknowable for anything data-dependent. You know the input bytes, not the output rows. A filter's selectivity (the fraction of rows it keeps) decides how long the next stage takes, and you learn it only by running the filter.
  2. The rate changes across stage boundaries. If stage 1 reads 1 GB per second, linear extrapolation says stage 2 will too. But stage 2 may cost 50 times more per record. The estimate is wrong by the ratio of per-record costs between stages.
  3. Completion is the maximum of the task times, not the mean. A stage with 1,000 tasks ends when the slowest task ends. (This is the tail-at-scale effect: when many parallel parts must all finish, the slowest part decides, and the chance that at least one part is slow grows with the number of parts.) Any estimate driven by the median task under-predicts. This is why progress bars race to 95% and then sit there: the last 5% of tasks are the slow ones.

The honest report is: percent of tasks complete plus the running time of the current longest-running task. The second number tells you whether the 95% is moving. applies to progress too, but only in a steady state, and a staged DAG (a job with stages that depend on each other) is never in a steady state.