Queue versus log: where progress state lives

Why is queue versus log the most consequential transport choice, and what is the log's real tax?

Queue versus log: the load-bearing distinction

This is the single most consequential choice in the field, and most of the other patterns are downstream of it. The difference is not durability, not throughput, not the vendor. It is where the progress state lives.

The queue

The broker tracks the state of every individual message. A message moves through states:

• available: waiting to be delivered;
• in-flight: delivered to a consumer but not yet acknowledged, with a visibility timer running (if the timer ends with no ack, the broker makes the message available again);
• deleted: acknowledged, gone forever;
• redelivered: a nack or a timeout sent it back to available;
• dead-lettered: moved to a separate dead-letter queue (DLQ) after N failed attempts.

Consumption is destructive and claiming. One consumer gets each message. When it acknowledges, the message is gone.

The log

The broker tracks nothing per message. It stores records in order and deletes them only by a retention policy (by time, by size, or by compaction, explained below). Each consumer remembers its own position: an offset, which is one integer per partition ("I have read up to record 48,213").

Consumption is non-destructive and positional. Reading removes nothing. Many independent consumers can read the same records at different positions.

What follows, mechanically

(O(x) means "grows in proportion to x".)

The trade in one line:

This is two classic trade-offs at once: ordering versus parallelism, and mutability versus immutability. The log is the immutable choice: it only appends, and a cleanup process (retention) removes old data. It collects the benefits you would expect from immutability (reproducibility, many cheap concurrent readers) and pays the expected costs (space, and janitorial work: retention policy, compaction, offset management, and tiered storage, which moves old data to cheaper storage).

The head-of-line problem is the log's real tax

Everyone learns the parallelism ceiling. Fewer people understand this one: in a log, a single record your consumer cannot process stalls the whole partition for as long as the problem lasts. This is head-of-line blocking: the item at the front of the line blocks everything behind it.

The three general exits (stop, skip, or set the record aside) and what each pays are in Poison pills and skip-and-log. On a log, setting it aside means a retry topic or a DLQ, and that breaks the ordering for that key: you have converted the log back into a queue, locally (Dead-letter queues).

There is a log-specific option that people use too rarely: pause that one partition, keep consuming the others, and raise an alert. The blast radius is bounded, nothing is silently lost, and order is not broken. The cost is a control loop that you have to write yourself.

When to pick which

Choose a queue when: each unit of work is independent, order truly does not matter, items have variable and unpredictable cost, you need to scale consumers to the volume of work, and history has no value. Examples: a resized thumbnail, a sent email, a webhook delivery.

Choose a log when: the records are facts about the world and not tasks, more than one consumer will want them, order per entity matters, or you will need to reprocess. In data engineering you always will, because of a bug fix, a new column, a schema change or a new downstream table.

The data-engineering default is a log, and rightly so, because our records are facts and our consumers multiply. But use a queue without embarrassment when the thing really is a task queue. Running a work queue on Kafka partitions only because Kafka is what you have is a common and expensive mistake: tasks with variable cost on partitioned consumption give you idle consumers next to a stalled hot partition, and no per-task retry.

At Cartwheel this gives a clear split: "order placed" is a fact, so it goes on a log. "Resize this product photo" and "send this email" are tasks, so they go on a queue.

Compaction: the third mode

A compacted log is worth naming because it fits neither category. It keeps only the latest record per key, forever. It stops being a stream of events and becomes a durable, replayable snapshot of state with change notifications attached. Retention is now semantic (keep the latest version of each key), not temporal (keep seven days).

This is what lets a log act as the source of truth for a materialized table (for example Kafka Streams' KTable, or Debezium topics that feed a lakehouse; Debezium is a CDC tool, and a lakehouse is a data store built on table files in object storage). Hold these consequences in your head:

• Deletes must be explicit. You write a special record with the key and an empty value, called a tombstone.
• Tombstones themselves must eventually be cleaned up. After that, a slow consumer that was offline for a long time can resurrect a deleted row: it never saw the tombstone, and it sees an older record for the key somewhere else. There is a window of safety, and past it you get wrong data.
• You lose intermediate states. Anything that consumes deltas (changes) instead of full state cannot use a compacted topic.