Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Adaptive query execution

Which decisions can be revised at a barrier, and why can adaptivity never fix the first stage?

This is the true centre of the article, and its central move is often mis-stated.

The founding constraint of this mechanism was that intermediate cannot be observed before execution. (AQE) does not improve the estimate. It moves the decision to a point where the quantity is measured and not estimated. This is a different kind of answer from the earlier articles, and it is why this field's research moved from search (join order), to estimation (), to adaptivity. Each shift admitted that the previous one was optimizing the tractable problem and not the one.

What can be revised at a , from easy to hard:

  1. Sizing of an operator already chosen: merging small partitions after a shuffle, changing parallelism, swapping the build side of a . No plan change, only parameters. Cheap and reliable, and in practice most of AQE's real-world value.
  2. Operator substitution: a sort-merge join becoming a hash join, now that one side is known to be small.
  3. Skew handling: splitting an oversized partition. This needs per-partition statistics, which the barrier supplies for free.
  4. Real re-planning of the rest of the plan: changing the join order downstream. The hardest, and rarely attempted.

The limits of adaptivity

AQE can revise only decisions whose alternatives are still reachable. After you have shuffled on key k, switching to a broadcast join is no longer cheap, because the shuffle already happened. So a barrier's adaptive value is proportional to how many decisions remain downstream of it. barriers are valuable. Late ones buy almost nothing. This is why the biggest AQE wins cluster at the first shuffle, and why re-planning near the end of a plan is mostly for show.

The first stage is never adaptive. Everything before the first barrier runs on estimates, exactly as before. So the folklore "AQE fixes bad statistics" is wrong in a specific, diagnosable way. It fixes bad estimates downstream of a barrier. It does not fix a wrong first access path, a missing partition filter, a non-sargable predicate or a failed , because those errors are upstream of every barrier by construction. When someone switches adaptive execution on and sees no improvement, the error was almost always before the first barrier, which sends you back to the earlier articles.

Costs

  • Barrier latency, which may be zero if the barrier existed anyway.
  • Re-planning time, which is small.
  • Loss of determinism and explainability. The plan you examined is not the plan that ran. Two runs of the same query may differ. Plan-regression analysis (checking that a change did not make plans worse) gets much harder. This is the variance principle arriving inside the : plan shape, the very thing that operators had been buying hints (manual instructions to the ) to remove. Both mechanisms are reasonable and they are in direct tension. A system can have predictability or adaptivity as its default, not both.

A placement note: adaptive execution arrived in Spark in version 3.0. On a 2.4-era engine none of the above is available. The remaining substitutes are all manual early-binding moves: broadcast hints, precomputed statistics, explicit repartitioning to fix skew. In other words, the escape from an estimation problem that you cannot defer is to supply the answer yourself.

In one sentence: adaptivity replaces estimation with measurement for decisions downstream of a barrier, is bounded by which alternatives remain reachable, cannot touch the first stage, and pays in plan determinism.