Workload realism and scale ladders

Which features must a synthetic workload get right, and how do you measure across scale?

Workload realism

The workload is half the system. Every mechanism inside a database depends on the data, so the generated data silently selects which mechanisms get exercised.

Uniform keys defeat every skew-handling path. You measure a system whose worst case is structurally unreachable. (Real seat sales are skewed: the front rows of the famous band are clicked thousands of times, the back rows rarely.)
Random, high-entropy values defeat compression entirely, because nothing compresses. Naively repeated values compress absurdly well. Both are wrong, in opposite directions.

The features that actually change which mechanism runs are: key skew, cardinality relative to the memory grant, value compressibility, null density, row width, update-to-insert ratio, and inter-query locality (how often queries touch the same data).

Cardinality (the number of distinct values) deserves special attention because its effect is a cliff, not a slope. When a hash table (a structure for grouping or joining) stops fitting in memory, the engine switches to spilling (writing parts to disk), which is a different algorithm with a different cost curve. A generator parameter set carelessly can move your experiment across that boundary and invert the conclusion.

The deepest failure is correlation. Generators emit independent columns. Real data has functional dependencies (city determines country) and correlated predicates. A cost-based query optimizer estimates how many rows a filter will match, and it usually assumes the columns are independent. So synthetic data flatters exactly the component whose real-world weakness is that assumption. Here the false-independence problem produces a bias with a known direction, not just noise.

Perfect realism is unobtainable and is not the goal. The achievable standard is: the distributional features that the mechanism under test is sensitive to must match reality. This means you must know what the mechanism keys on before you generate the data. Realism is a claim about the mechanism, not about the data.

Generating it

The mechanics are harder than the requirement suggests.

Generating each column independently is easy and wrong, because independence is false in real data. Correlated generation is what you need, and it is truly difficult. Preserving all of these at the same time has no general solution, and each approximation breaks something:

joint distributions (how columns relate to each other);
functional dependencies (city determines country);
referential integrity (every order points to an existing customer) across tables;
realistic key skew (a few keys are very popular).

There are three families of generators, each with a different failure:

Family	What it does	How it fails
Parameterized generators	Produce known, tunable distributions	Unrealistic joint structure
Sampling real data	Preserves everything, including correlations	Privacy problems, and a scale ceiling: you cannot sample your way to 10× your current volume
Model-fitted generation	Preserves single-column distributions and pairwise structure	Smooths away the rare combinations, and rare combinations are often what breaks things

Scaling a dataset is usually done wrong, so it deserves its own name. Worked example. Suppose you have 10 million seat-hold rows over 100,000 distinct seats, so there are 10,000,000 / 100,000 = 100 rows per seat. To make a dataset 10× bigger you have two choices:

Multiply the rows by 10 and keep 100,000 seats: now there are 1,000 rows per seat. That is a different workload (hotter keys), not the same workload at scale.
Keep 100 rows per seat and have 1,000,000 seats: now every hash structure and index has 10× more keys, and its memory behaviour changes.

There is no neutral way to scale a dataset. The scale factor is a design decision that selects which mechanism your ladder is actually exercising.

Scale ladders

The mathematics is almost trivial once stated: one point determines no function, two points determine a line, and real systems are not lines. Amdahl's Law bends the curve (the serial fraction of work limits speed-up). The Universal Scalability Law (USL) adds a coherency term (the cost of coordination between workers) that can turn the curve downward. And every buffer, cache and rate limit puts a cliff in it.

So measure geometrically: 1×, 3×, 10×, 30×. Interesting behaviour changes at orders of magnitude, not at fixed increments. The output you want is the shape: linear, sublinear with a knee, or rising and then falling. Each shape implies a different mechanism and a different remedy. A single point cannot distinguish "fast" from "not yet saturated."

Scale has more than one dimension, and each answers a different question:

What you grow	What you hold fixed	Name	What it tests
Data size	Cluster	Strong scaling	Amdahl's serial fraction
Data and cluster together	—	Weak scaling	Gustafson's view: bigger problems keep large clusters busy
Concurrency	Data	—	Contention
Key cardinality	Data	—	Memory cliffs

These are four different curves, and a result on one says nothing about the others.

What this article forbids

A causal claim from a single configuration.
A tuning recommendation applied off the OVAT path where it was measured.
Any result whose configuration is unknown, including to its own author.
A comparison of runs taken in fixed order on hardware that has memory.
Extrapolating a curve from one point, or across a scale dimension you did not vary.
An average over a warm-up trajectory.
A conclusion from synthetic data without knowing which distributional feature the mechanism keys on.

Escapes and their prices. Every escape moves a cost and does not remove it.

Escape	What it buys	What it costs
Fractional factorial	Fewer runs	Aliasing: main effects confounded with interactions
Randomized run order	No bias from drift	More variance, so more runs
Shared infrastructure plus an interleaved control workload	Access to cheap machines	A noise floor that bounds the smallest effect you can detect
Modelling instead of measuring	Speed	The model's assumptions (fine if you then check the model against two points, not one)

One principle

Case, resolvedConstructed case · Ingrid at Bellwether Tickets

40% faster on Monday, slower on Thursday

Tessera DB was 40% faster on Monday and 5% faster on Wednesday on a fresh instance, a hold-writing test claimed 9 million holds per second, and a tuning guide's batch size did nothing in production. Where it started: Experiment design →

The 40% / 5% / slower puzzle. She had run on credit-based instances in a fixed order (A first, then B), with no recorded configuration. The credit balance and thermal state were hidden variables, so run order was a variable. Now she records an environment dump with every run, interleaves a fixed baseline workload as a noise-floor check, and runs A and B in alternating, randomized order.

The 9,000,000 holds/s. A prediction (36 GB/s, 5–12× above one drive) would have flagged it at once. She now writes down the expected range before each run.

The tuning guide. "Batch size 512" came from varying one knob at a time on someone else's starting point. She screens seven factors in 16 runs, then runs a full factorial over the three that matter.

Before the next run she also writes the decision rule on a page: "Ship Tessera only if on-sale p99 improves by at least 15% at the same monthly cost, on skewed seat-hold data, at 1×, 3×, 10× and 30× load."