Labs

Barriers multiply variance

Reproduce E[max]/E[mean] ≈ 1 + c·√(2 ln n) and the BSP rule that the penalty multiplies by the number of rounds.

Measure the cost of a boundary: tuple-at-a-time against batches

See the "fixed cost per control transfer" with your own numbers.

Find the best batch size on your machine

Check the claim "batch size is chosen so the working set stays in cache, about L1 size ÷ row width".

Compute a compilation break-even

Turn "compilation has a break-even data volume" into a measured number.

Per-row UDF versus vectorized UDF

Score 5 million rows with x * 0.3 + 2.0 per row and vectorized in blocks of 100 to 1,000,000; and the time to read and write the data.

Whole-query restart against task retry

Check that expected whole-query attempts grow like e^(nTλ).

The Zipf ceiling on data-parallel speedup

Turn "the largest partition is your serial fraction" into a number.

Triage plot, salting and the join product

Classify three kinds of long tail, then fix skew by salting, and see that join skew multiplies.

Many small reads against few large reads: the transfer floor

Measure the minimum efficient transfer size on your own disk, then compute a shuffle width ceiling from it.

When does shuffle compression pay? And a non-deterministic partition function

Measure codec throughput and ratio on shuffle-like data, then apply C > BW · r/(r−1).