Choosing N

How many partitions should a stream have, and why is changing the number later a project?

N is the number that sets your parallelism ceiling and your ordering scope at the same time. It deserves its own section because it is the one decision in this article that you make once.

Too low: parallelism is capped below what you need. Adding consumers does nothing. A single hot partition cannot be fixed.

Too high:

more files and smaller batches downstream (the small-files problem);
more metadata;
more per-partition overhead (memory, file handles, replication traffic);
longer rebalances and longer leader elections (a rebalance reassigns partitions among consumers; a leader election picks the replica that accepts writes for a partition);
more coordination on failure;
and more variance: with many partitions and moderate key cardinality, partition skew gets worse, not better.

Heuristics

N ≥ target_throughput / per_consumer_throughput, then multiply by 2–3 for headroom.
N ≥ max_consumers_you_will_ever_want. This is the ceiling, and it is hard to raise.
Choose N so that the number of distinct keys per partition is comfortably above 1. Otherwise partition skew dominates.
Prefer a highly composite number (12, 24, 60, 120, 360), meaning a number with many divisors. If you ever reassign partitions to consumers, a composite N divides evenly among many consumer counts. It is a small thing that occasionally saves you.
Consider overpartitioning at the start and running several partitions per consumer. Consumers can scale up later without a repartition. It costs some per-partition overhead now, and buys the one change you cannot otherwise make.

Consumer groups and rebalancing

N caps how many consumers in a group can work, because each partition is read by one member of the group at a time. Which member reads which partition is decided by the group itself, and it is decided again whenever a member joins, leaves, or is declared dead. That redistribution is a rebalance, and it costs three things:

A pause. Partitions that change owner are not consumed until the new owner has them.
Repeats. A partition moves with the offset its old owner last committed. Everything the old owner processed after that commit is processed again by the new owner, so a rebalance is a source of duplicates like a crash.
Cold state. A stateful consumer must rebuild or fetch the state of each partition it receives before it can continue.

How much of the group pays depends on the protocol:

Protocol	What happens on a change	Who pauses
Eager (the original, "stop the world")	every member gives up all its partitions, the group re-forms, and every partition is assigned again	the whole group, for every change
Cooperative (incremental, Kafka's `CooperativeStickyAssignor`)	only the partitions that must move are revoked, over two short rounds; the rest keep running	only the members that lose or gain a partition
Static membership (`group.instance.id`)	a member that restarts within the session timeout gets its partitions back with no rebalance at all	nobody, for a rolling restart

Two habits keep the repeats small. Commit the offsets of work already done when partitions are revoked (Kafka calls the consumer back in onPartitionsRevoked), so the new owner starts where the old one stopped. And keep the commit interval short enough that what a rebalance repeats is something the sink can absorb. A rebalance that is triggered by a slow member rather than a dead one feeds itself: see the rebalance storm in Connection pooling, keep-alive and heartbeats.

Changing N later

For a log, increasing the partition count breaks the key-to-partition mapping, so per-key order is broken across the change. For stateful consumers, the state must be redistributed. The clean path is usually a new topic with the new partition count, then dual-write (write to both topics) or reprocess, then cut over, not an in-place change. Budget for it. It is a project, not a configuration edit.

What to carry out of this article

Balance and locality are opposed, and you must pick. Everything else here is a technique for making the loss tolerable.

Your partition key is your ordering scope, your state locality, your parallelism ceiling and your skew profile. That is one decision with four consequences. It is the least reversible choice in your pipeline, so spend your time accordingly.

A DLQ is not a safety net. It is a trade of completeness and order for availability. If nobody monitors it, it is data loss with a dashboard.

Multiplexing relocates head-of-line blocking. It does not eliminate it. Always ask what the layer below guarantees.

Case, resolvedConstructed case · Ravi at Cartwheel

One partition hours behind, and time running backwards

With orders keyed by seller_id across 12 partitions, one partition fell hours behind on Black Friday while six extra consumers sat idle, and a few OrderShipped events were processed before their OrderPlaced after a DLQ replay. Where it started: Routing and distribution →

Ravi and Lena start with the dashboard. The per-partition lag shows one partition at 1.4 million records behind. They check the key histogram: FreshBox produces 40% of all orders. This is key skew, not partition skew. No single key dominates the other 11 partitions, and no hash function can split one key. That is also why the six extra consumers did nothing: a partition is read by one consumer at a time, so the consumers beyond the partition count had nothing to read.

The arithmetic. At the Black Friday peak of 2,000 orders per second, FreshBox sends 0.4 × 2,000 = 800 per second to one partition. A consumer handles 400 per second. The backlog therefore grows at 800 − 400 = 400 records per second, which is 1.44 million per hour. That matches the dashboard. The other eleven partitions each carry (2,000 − 800) / 11 ≈ 109 per second, far below capacity.

They make four decisions.

Change the key for order events. Order events are keyed by order_id, not seller_id. The order_id key has high cardinality, an even distribution, and matches the real ordering need: the events of one order must stay in order. Nobody needs the events of one seller in order.
Compute seller-level numbers with two-phase aggregation. The analytics job pre-aggregates per order_id-partition, then combines the small partial results by seller_id. The FreshBox hot key now receives a handful of partial counts per minute instead of 48,000 raw records. For FreshBox's dedicated seller dashboard, a separate pipeline isolates that one hot key.
Classify failures before sending anything to the DLQ. The out-of-order events came from an old rule: any failure was retried three times and then sent to the DLQ. During a short Inventory outage (transient and systemic), thousands of OrderPlaced events went to the DLQ, while the matching OrderShipped events, which did not need Inventory, kept flowing. When the DLQ was replayed an hour later, the old events arrived after the new ones. The new policy: transient failures retry with backoff and are never dead-lettered; malformed records go to the DLQ at once with full provenance; a DLQ rate above 2% of throughput pauses the consumer and pages Sana. Because order matters for these events, a stuck partition is paused, not drained. Consumers also use the order's version number, so a late OrderPlaced can never overwrite a newer OrderShipped state.
Re-choose N with a plan. They create a new topic orders.v2 with 24 partitions keyed by order_id. They dual-write for a week, move the consumers over, then retire the old topic. They call it a project, because it is one.

On the next sale day the max partition lag stays under two seconds. Lena adds the alert on max(partition_lag), and removes the old alert on total lag.