Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Control plane and data plane

Why must the control plane stay off the data path and scale with changes, not requests?

Control plane and data plane: the separation and its two rules

Agreement and dissemination gave the ratio: kilobytes of facts go through , petabytes of data go around it. (The makes and stores decisions such as membership, config and leadership. The serves the actual requests.)

How the data plane hears a decision

The control plane decides; the data plane must find out. Asking on every request would put the control plane back on the data path, so the data plane holds a copy of the decision and is told when it changes. The usual form is a watch: a node reads the current state together with its version, then asks to be notified of any change after that version. Three rules make it safe.

  1. A notification is a hint, not the decision. It says "something changed after version v", and the node re-reads the state and its new version. Acting on a notification's payload alone risks acting on a change that a later one has already overtaken.
  2. Watch from the version you read. A watch registered without a version can miss a change made between the read and the registration. Starting from the version closes that gap.
  3. A missed notification must be harmless. Connections drop and notifications are lost. A node that reconnects re-reads from its last version, or the whole state if that version is gone. Until it does, it acts on a stale copy, which is why receivers still check the version on arrival (fences, reject and correct).

A watch costs the control plane one notification per watcher per change, not one read per request. The separation itself has two rules, and each is derived, not a matter of style.

Rule 1: If serving a request requires a call to the consensus-backed component, then control-plane availability is data-plane availability. And the control plane is the least available part of the system by construction, because it needs a majority up continuously while the data plane needs only itself. The only design that makes a low-availability control plane acceptable is fail-static: the data plane the control plane's output and keeps running on the last known configuration when the control plane cannot be reached.

Static configuration is stale configuration. It is safe exactly when its staleness costs a retry, and unsafe when staleness costs an invariant. So and fencing are complements, not alternatives. This is also the "undeclared dependency" idea handled correctly: the dependency on the control plane is explicitly declared and its failure mode designed, instead of being discovered when etcd goes down and every dependent system that never listed it stops.

Rule 2: control-plane load must scale with changes, not with requests. Membership changes, config pushes and transitions are rare. Requests are not. Any design that writes a per-request or per-file fact through the coordination layer has turned consensus into a data path. The symptom is always the same: a single metastore becomes the throughput ceiling of a system whose storage otherwise scales horizontally. The small-files problem (known as an I/O issue and as a pricing issue) appears here as a coordination issue, which is its most basic form: a million files means a million facts stored in a component sized for thousands of changes.

Speculative execution (a second copy of a slow task) deliberately creates two workers writing one logical output, the same two-writers problem as a stale leader, and it is resolved the same way: one designated commit makes exactly one attempt's output visible (commit arbitration).

In short: keep the control plane off the data path, let the data plane fail static, and never let a per-request fact become a control-plane write.