Savepoints and state schema evolution

Why must new code read old state, and why is the state schema a public interface?

Deploying new code against existing state requires the new program to read the old state. This is the practical dominant operational problem of streaming.

A savepoint is a deliberately taken, format-stable, self-contained checkpoint that you own, rather than one that the retention policy owns and may delete. You take one before an upgrade. To restore a savepoint into modified code, three things must line up:

Stable operator identity. The engine matches saved state to operators by an ID. Auto-generated IDs change when the topology changes, so you must assign explicit IDs (often called UIDs).
Compatible state serializers. The code that writes and reads the state bytes must stay compatible. Adding a field with a default value is fine. Changing a field's type, or removing a key field, is not.
A topology shape that tolerates the change. Removed state must be possible to drop, and added state must be able to start empty without corrupting the meaning of results.

Now notice what this list is. It is compatibility modes over a schema registry (a service that stores each schema version and checks backward and forward compatibility), relocated into the runtime.

The escape hatches and their prices

Escape	How	Price
Parallel run and cutover	start a new job from an earlier source offset, run both, then switch	double compute during the replay window; the source must still retain that window; sinks must be idempotent or partitioned so the overlap is harmless
Offline state migration	a program that reads the savepoint and rewrites it	a bespoke program against an internal format
Accept state loss, restart empty	begin with no state	for a 30-day window, 30 days of degraded output

Every change is a migration

Case, resolvedConstructed case · Ana at Ridelo

A 14-minute rescale, and state that will not load

Scaling the fraud job from 8 to 16 instances took 14 minutes, the autoscaler oscillated, and adding one field to the state made the new version refuse to start. Where it started: Rescaling and upgrades →

The 14-minute rescale. She takes the rescale out of the critical path: she picks max_parallelism of 720 for new jobs (divisible by many numbers: 8, 10, 12, 16, 20 and more), schedules scale-ups before the holiday in a quiet hour, and keeps headroom so that a catch-up after the rescale takes minutes, not hours. She decides not to automate scale-down at all.
The oscillating autoscaler. She adds a cooldown of 30 minutes, scales up at 60% and down at 30%, and never scales during a backlog. The test settles.
The new field. She had not assigned explicit IDs to her operators, so the new topology generated different IDs. She takes a savepoint of the old version, adds explicit IDs and the new field with a default value, and restores. For the future, she writes a rule in the team's checklist: a state schema change is reviewed like an API change. For one risky change, she uses the parallel-run approach: a second job replays from a 7-day-old offset into a separate idempotent table, and she cuts over when it catches up.