Message keys and key-based ordering

Do you need per-key order at all?

Here is the ordering guarantee, stated precisely. Imprecision here is the source of a whole category of bugs.

Every clause carries weight. If any one is broken, the guarantee is void.

Where it breaks in practice

1. The partition count changed. Keys are remapped. The records of one key now exist in two partitions, with no defined order between them. This is why increasing the number of partitions is a semantic event, not only a scaling event.
2. Producer retries with pipelining. Records 1 to 5 are in flight at once. Record 1 fails, records 2–5 succeed, and then record 1 is retried and lands after them. The fix requires an idempotent producer: the broker tracks a sequence number per producer and partition, and rejects out-of-sequence writes. In Kafka this is enable.idempotence=true, which keeps order with up to 5 requests in flight.
3. Consumer concurrency. You partitioned for order, and then added a thread pool inside the consumer. The broker's guarantee ended at your process boundary. This is an extremely common self-inflicted wound.
4. Asynchronous side effects. The consumer fires off writes without waiting for them to finish. The broker order is preserved, but the order at the destination is arbitrary.
5. Dead-letter queue re-injection. Covered on the next page. This is Ravi's second problem.
6. Several producer instances writing the same key with no coordination. Their relative order is whatever the network decided.
7. The key changed. The entity's routing key was changed in the middle of its life.

The guarantee also has a price that is paid when nothing breaks. A partition is a total order by construction, so a slow consumer of one key blocks every key in that partition (head-of-line blocking). The standard mitigations are more partitions, parallel processing by key inside the consumer, and moving bad records aside. All of them break the sharing; none of them is a fix to the log.

The prior question: do you need order at all?

Before you build machinery to preserve order, ask whether you need order. The test is commutativity: does the result stay the same if the operations are applied in a different order?

• Full-state upserts with a version number (or log sequence number, LSN: a number that grows with each change in the source database) are commutative under last-writer-wins (when two versions arrive, keep the one with the higher version). Order is not needed.
• Non-additive deltas and appends are non-commutative. Order is needed.
• Insert and delete interleavings are non-commutative, and a delete that arrives before its insert resurrects a deleted row. Order is needed, or you need versioned tombstones (a delete marker that carries a version, so a late insert with a lower version is ignored).

Delta events:      +1, +1, -1            order irrelevant here, but a lost or doubled one is a wrong total
Full-state upsert: {order 4711, status SHIPPED, version 7}   the highest version always wins

Converting deltas into full state at the source is usually cheaper than preserving order from end to end. This is the "buy-out", and it is underused because it looks like more data on the wire. It is more data. It is also a whole class of bugs that you never have.

A queue can keep per-key order too: SQS FIFO's MessageGroupId is the clean version, with order within a group and parallelism across groups.

Where you cannot buy out, the general fallback is sequence numbers plus re-sequencing at the sink. The consumer holds out-of-order records in a buffer until the gap fills, and uses a timeout policy for gaps that never fill. You have reintroduced a buffer, a timeout and a decision about what to do on a permanent gap. That is why this is the fallback and not the default.