Consumer lag and skew

Why watch lag's derivative per partition, and why does skew cost the maximum rather than the mean?

Consumer lag

Lag has two units, and the cheap one is misleading.

Offset lag = newest offset − consumer's offset (a count of records). It is free to compute, but it does not convert to time unless throughput is constant.
Time lag = the age of the oldest unconsumed record. This is what a human needs ("we are 40 minutes behind").

RelationshipCatching up after an outage →

time to drain = \frac{lag}{consume rate - produce rate}

$M$: the system's maximum throughput
$a$: normal arrival rate
$T$: how long the outage lasted

Stated in 4 places →Principle: Queues explode near full load →

So the useful signal is the sign and size of the derivative (the rate of change), not the level:

a large, flat lag is a backlog being worked down: calm;
a small, rising lag is an outage in forty minutes: urgent.

Look at lag per partition, for the same reason that averaging p99 values fails: a sum hides a single stuck partition, because the quantity that hurts is the maximum. If 99 partitions have lag 0 and one partition is stuck at 4,000,000, the total (4,000,000) looks like a smooth number, and the mean (40,000) looks tiny. This is Nadia's "40,000" graph. Aggregation destroys the shape that contains the fault.

One stuck partition inside a calm total

Skew detection

Skew is variance across partitions. Its cost is the maximum, not the mean, because the job waits for the slowest partition. (Observing skew is the visible side of the design trade between ordering and parallelism: keeping all records of one key together forces them into one partition, and popular keys make big partitions.)

Two measurements:

The partition size distribution. Cheap, and available after the fact from task metrics.
The key frequency distribution. To know which key is heavy, you need a heavy-hitters sketch, because computing it exactly is a full aggregation over the data you wanted to avoid scanning. The sketches Space-Saving (keeps a small table of the currently most frequent keys, replacing the least frequent when full) and Count-Min (a small grid of counters addressed by several hash functions, which only over-counts and never under-counts) reappear here as monitoring primitives.

The number to watch is max / median partition size. Beyond about 5×, you are paying for parallelism you do not receive.

There is an asymmetry: skew is cheap to see after the run and expensive to see before. This is exactly why adaptive execution (the engine splits big partitions during the run) exists: the only affordable place to observe is mid-flight.

One line for the family: services are correct per request, datasets are correct per arrival, so the observables are freshness, lag, volume and skew, and each is a distribution whose maximum is the quantity that hurts.

Case, resolvedConstructed case · Nadia at Brightcart

A green job over empty input

The nightly job showed green while daily_revenue was hours stale, a progress bar sat at 95% for forty minutes, total lag looked fine while one partition was stuck, and an average hid a hot partition. Where it started: Data-state observables →

False green. She replaces "last successful run" by two checks on the data: freshness against event time (now − max(event_time) must be under 3 hours on business days) and a volume check (today's row count must be within a band of the same weekday's usual count). Tuesday night's empty input would have failed both within the first hour.
Progress. The team stops using the percentage bar. They report "tasks done" plus "longest-running task: 31 min". It shows the slow tail, which is the real status.
Lag. The graph now shows lag per partition and its derivative. One partition is stuck. The cause is a single customer sending 30% of all events: skew. max/median is 18×. She enables adaptive splitting of the hot key.

She also wrote down the expectation behind each absence check ("the feed arrives by 02:00") in the table's metadata, with an owner, and set a quarterly reminder to review it.

Consumer lag and skew

Consumer lag

Per partition, not as a sum

Skew detection