Vectorized and compiled execution

What do batch-at-a-time and compiled execution buy over a row at a time, and what does each cost?

Vectorized (batch-at-a-time) execution

Vectorization applies the fixed-cost principle to the boundary. If a boundary has a fixed cost F and you send N tuples across it, the per-tuple overhead is F/N. With F ≈ 40 cycles and N = 1,024, the overhead is 40 ÷ 1,024 ≈ 0.04 cycles per tuple. It stops existing as a line item. Nothing about the direction of the boundary changed. That is the whole mechanism, and it is why vectorization works in both pull and push engines.

# one call per batch, not per tuple; the loop inside is a plain array loop
def filter_batch(batch):                 # batch = arrays of up to 1024 values
    out = []
    for i in range(len(batch.status)):   # no per-tuple virtual call here
        if batch.status[i] == OK and batch.zone[i] == N:
            out.append(i)                # a "selection vector": indexes that passed
    return out

The second advantage is not amortization but what a batch makes possible. Inside an operator, a batch is a loop over an array with no per-tuple polymorphism. That allows:

Branch-free predicate evaluation with selection vectors (a list of the positions that passed). The branch misprediction cost is paid once at the vector level instead of once per tuple.
SIMD (one instruction handles several values).
And the underrated one: many independent cache misses in flight at once. A vectorized hash probe issues N independent lookups whose waiting times overlap. Memory-level parallelism rises, and Little's Law does the work: in-flight misses = miss rate × latency. Only a batch can generate enough independent misses to fill that.

The size derivation is the important part, because the answer is not "bigger is better." The batch's working set must stay in cache. If it does not, each operator writes N tuples to memory and the next reads them back, which brings back memory traffic proportional to the pipeline depth. With k live columns of 8 bytes each (plus intermediates), the working set is roughly N × k × 8 bytes. Set that against a 32 KB L1 cache. If we say the equivalent of about 4 live 8-byte columns:

N ≈ 32,768 ÷ (4 × 8) ≈ 1,024

Hence the canonical batch size of 1,000 to 2,000. Too small, and F/N comes back. Too large, and you turn a free in-cache hand-off into a round trip to memory. The batch size is a granularity chosen so that materializing between operators costs nothing. That is the "match the unit to the substrate" principle in its clean form: matched granularity, no amplification.

The honest costs of vectorization:

Latency to the first row rises by one batch.
Each operator's code must handle vectors and selection vectors instead of tuples. This is a real but bounded engineering cost.
Expression interpretation still happens once per batch per operator. A query with fifty tiny expressions still pays fifty dispatches per batch. That is where compilation keeps its remaining margin.

Vectorized UDFs: a boundary between two runtimes

The same rule decides what a UDF (user-defined function: a function you write and the engine calls) costs when it runs in a different runtime from the engine, for example a Python model called from a JVM engine. Every call crosses a language boundary, and row-at-a-time invocation pays serialization and a cross-language call for each row. That fixed per-call cost is often microseconds, while the per-row computation is nanoseconds.

Batching amortizes the boundary: the engine hands the function a block of columnar data, the function runs over the whole block, and a block comes back. That is a vectorized UDF. If both sides also agree on an in-memory layout, the crossing costs a pointer instead of a serialize-and-deserialize round trip, so the boundary is deleted rather than amortized. The price moves onto the agreement: both sides commit to one layout.

One warning carries over from batch scoring of models: the call is usually the wrong thing to optimize. Reading the inputs, joining them and writing results typically dominate, and teams buy accelerators for a workload limited by I/O.

Compiled query execution

Amortization has an alternative: elimination. Generate code for this one query, with the operator structure resolved at compile time. The boundary disappears instead of being divided. Data-centric compilation pushes a tuple as far through the pipeline as it can go before a pipeline breaker. The resulting code has no operator interface at all: types are known, field offsets are constants, predicates are inlined, and the tuple never leaves the registers.

The cost is relocated, not removed, and it lands somewhere new. Compilation is a fixed cost paid per query, not per tuple. So it has a break-even data volume.

This is the fixed-cost principle again, but at a level it had not reached before: the fixed cost moved from the tuple boundary to the query boundary. So it derives a natural minimum query size instead of a natural batch size.

That threshold forces the mechanism that every engine converged on: tiering. Start interpreting at once, compile in the background, and switch when the compiled code is ready and the query is still running. This is JVM JIT tiering (JIT: just-in-time compilation of hot code while the program runs), Umbra's adaptive compilation and Spark's code-generation fallbacks, reached independently. That several teams arrived at tiering from unrelated starting points is evidence that the threshold problem is structural and not an implementation detail.

A correction to one-sided folklore: compilation does not uniformly beat vectorization. A well-known 2018 comparison took both models seriously. It found that compiled execution was ahead on calculation-heavy queries (many expressions, arithmetic, tight predicates), and vectorized execution was ahead on memory-bound, data-heavy queries, because vectorized primitives keep more independent cache misses in flight than a compiled tuple-at-a-time loop does. The two models are good at different halves of the roofline picture (compute-bound versus memory-bound work), and the overall gap is far smaller than either camp's marketing.

The stronger argument for vectorization is not performance at all. It is observability, portability, debuggability and not needing a compiler inside your process. That is why nearly every engine designed since about 2019 chose push + vectorized + hand-written primitives. When several independent teams land on the same cell of the cube, it tells you about the best balance of engineering cost, not about peak throughput.