Two kinds of skew: different problems, different fixes

How do you tell key skew from partition skew, and which fix fits each?

Every keyed scheme has this problem. It gets its own section because it dominates operational reality, and because it is exactly Ravi's first problem.

Key skew. One key has far more records than the others: a celebrity account, a null key, a default tenant, an UNKNOWN bucket, a bot. Hashing does not help, because hash("null") is a single value. All null-keyed records land on one partition, however good the hash is. At Cartwheel, FreshBox is this: one seller that produces 40% of all orders.

Partition skew. The keys are spread unevenly across partitions even though each key has a similar volume. With N partitions and K keys, the expected number of keys per partition is K/N, but with random variance. If K is not much larger than N, chance alone gives real imbalance. (Think of 14 keys over 12 partitions: some partitions get 0 keys, some get 3.) This is fixed by increasing N or by a better assignment, not by touching the keys.

The diagnostic: if one key dominates, it is key skew. If no single key dominates but one partition is overloaded, it is partition skew. They look identical on a partition-lag dashboard and need opposite fixes.

Why scaling cannot fix key skew

When one key is bigger than one partition's capacity, no amount of scaling helps. A key is indivisible by definition: it is the unit that must be kept together (co-located) for correctness, for example all events of one seller, or all updates of one account. You can add a thousand nodes, and the hot key still lands on one of them. Real keys are Zipfian (a few are extremely popular, and the top key alone can be several percent of the volume), so this is the normal case, not an edge case.

Scaling is powerless against a single key that exceeds single-node capacity. It is a hard constraint, not a tuning problem. Every escape below works by weakening either the co-location requirement or the correctness requirement. And the skew is in the world, not in the system: some tenants are larger, some products are popular, null is a value. A partitioning scheme is a bet on a distribution that you do not control, and the distribution will change without any change to your code. Skew can only be adapted to.

Fixes for key skew

Salting (key splitting). Use key + "#" + random(0, M) as the key. A hot key is now spread across M partitions. The cost: you have broken per-key order and per-key state locality for that key. Any aggregation over it now needs a second stage to combine the M partial results. This works only for associative and commutative aggregations (sum, count, max, min), meaning that grouping and ordering of the pieces does not change the result: (a + b) + c = a + (b + c) and a + b = b + a. It does not work for anything that depends on order.

import random

def route_key(seller_id: str, hot_sellers: set, salt_buckets: int = 4) -> str:
    if seller_id in hot_sellers:                       # only split the known hot keys
        return f"{seller_id}#{random.randrange(salt_buckets)}"
    return seller_id

Two-phase aggregation. This is the general form of salting. First pre-aggregate locally using the salted key. Then re-shuffle the (much smaller) partial results by the true key and combine them. This is how Spark handles skewed group-by queries. It is also the right mental model: the skew is in the raw volume, not in the aggregated result, so aggregate before you shuffle.

The hot partition went from 800 records per second to almost nothing.

Isolate the hot key. Route it to a dedicated partition, a dedicated consumer, or a separate pipeline. This is bulkheading (separating resources so one tenant's trouble cannot spread) applied to routing. It is often the most practical answer for a small, known set of hot keys. When the set is not known, detect the top keys with a streaming sketch such as Space-Saving or Misra-Gries (they find frequent items using little memory), handle them on the separate path, and combine the results afterwards. That code path runs only under skew, so it is rarely exercised, which is a reliability concern of its own.

Cache at the reader. Only for read-hot keys. It brings back the cache cliff: a cache with a high hit rate hides the key's cost until the cache is lost.

Skew join and broadcast. When you join a large skewed side with a small side, send a full copy of the small side to every worker (a broadcast) and skip the shuffle completely. Spark's adaptive query execution (AQE) does this automatically now. It still matters to understand why, because it fails when "small" turns out not to be small. Runtime splitting is the only escape that does not need you to know the skew in advance, which is why engines converged on it: skew changes every day, and a static fix is a fix for yesterday's distribution.

Fix the key. This is frequently the real answer. null keys and UNKNOWN defaults are data-quality problems that only look like performance problems.

Detection

Monitor per-partition lag and throughput. Never rely on the aggregate. The total lag looks healthy while one partition is hours behind, and by the time the total moves you have been broken for a long time. The metric that matters is max(partition_lag), and the alert should fire on the maximum, not on the sum or the mean.