Protocol framing

How does a receiver find message boundaries in a byte stream, and where does each framing make a bad record fail?

The problem: TCP has no messages

TCP is a byte stream. It has no concept of a message. If a program writes 100 bytes and then 200 bytes, the receiver may read 300 bytes at once, or 150 and then 150, or one byte 300 times. Any message boundary is something you invented, and the receiver must reconstruct it from the stream.

Framing is the method for doing this. A frame is one message together with whatever the receiver needs to find its end. There are three common designs. The choice also decides where failures show up, as we see at the end of this section.

Length-prefixed frames

[4-byte length][payload][4-byte length][payload] ...

Each message starts with a small header that says how many bytes follow.

Example: a 21-byte JSON event is sent as the 4 bytes 00 00 00 15 (21 in hexadecimal) followed by the 21 payload bytes. The receiver reads 4 bytes, learns "21", allocates a 21-byte buffer once and reads exactly 21 bytes.

Advantages:

• O(1) boundary detection. The receiver reads the header and knows exactly how many bytes to expect. No scanning.
• The payload is fully opaque. It can be arbitrary binary data, with no escaping and no reserved bytes. This is the property that matters most: nothing in your data can ever be mistaken for a delimiter.
• Skipping is cheap. You can jump past a message without parsing it.

Costs. The security cost is real:

• You must know the length before you write. So you must buffer the whole message first, or reserve space and fill in the length afterwards. This rules out streaming a message whose size is unknown until the end.
• The length is controlled by whoever sent it. A declared length of 4 GB means either an out-of-memory crash or a hang while waiting for bytes that never come. Always enforce a maximum frame size, and reject larger frames before you allocate memory. This is the most common vulnerability in hand-written protocols. It is a two-line fix that people skip.
• A corrupt length breaks the stream permanently. From that point you read payload bytes as if they were headers, and there is no way to find the next real boundary. The only recovery is to drop the connection.

MAX_FRAME = 1_000_000          # 1 MB: choose a limit that fits your records

def read_frame(sock):
    header = read_exactly(sock, 4)             # read_exactly loops until 4 bytes arrive
    length = int.from_bytes(header, "big")
    if length > MAX_FRAME:                     # check BEFORE allocating or reading
        raise ProtocolError("frame too large: closing connection")
    return read_exactly(sock, length)

Length-prefixing is the default for essentially all binary protocols: Kafka, gRPC over HTTP/2 frames, Avro object container file (OCF) blocks, Protobuf on the wire, Thrift.

Delimited frames

payload<delimiter>payload<delimiter>payload<delimiter> ...

The usual delimiter is a newline. This is how NDJSON/JSONL (one JSON object per line), CSV/TSV, text logs, parts of Redis's RESP protocol and SMTP work.

Advantages:

• Streamable without knowing the size. The writer writes as it goes.
• Human-readable and easy to debug with cat and grep. Any tool can split it.
• Resynchronizable after corruption. If a record is damaged, skip to the next delimiter and continue. This is the one real advantage over length-prefixing.

Costs:

• Every byte must be scanned for the delimiter. That is O(n) work with a branch per byte.
• The delimiter cannot appear inside the payload, so you need escaping or encoding. Escaping is where the bugs live: a newline inside a quoted CSV field, an embedded \n in a JSON string that was written unescaped, a Windows CRLF line ending where you expected LF. Every "our pipeline broke on one weird record" story in the newline-delimited world is a delimiter-collision story.
• There is ambiguity about trailing delimiters, empty records, and whether the delimiter ends each record or separates two records.

Why data engineering cares: splittability

A big data job reads one large file with many workers in parallel. Each worker gets a byte range of the file. This is called splitting.

• A newline-delimited file can be split at any byte offset. The worker scans forward to the next newline and starts there. Its neighbor does the same at the other end.
• A length-prefixed file cannot. At an arbitrary offset you cannot tell a length header from payload bytes.

That is why text formats survive in big-data work, although they are worse in nearly every other respect. Binary formats solve the problem with block framing plus sync markers. Avro OCF and Parquet write independently readable blocks, each starting with a recognizable marker. That gives binary framing and places where a reader may start.

The general pattern is worth remembering: frame at two levels. Put records inside blocks, and make the blocks findable independently.

Chunked frames

[chunk-size]\r\n[chunk-data]\r\n ... 0\r\n\r\n

This is length-prefixed at the level of each chunk, and delimited at the level of the whole message (a final chunk of size 0 marks the end). It lets you stream content of unknown total length while every piece describes itself.

• It enables streaming without knowing the total size. This is HTTP/1.1 chunked transfer encoding, and it is the reason you can stream a response while you are still generating it.
• It supports trailers: metadata sent after the body. This is how gRPC delivers its final status code after a stream ends.
• It costs per-chunk overhead. The choice of chunk size brings back the batching trade-off from Flow control, now at a lower level.

A security note from history: request smuggling attacks come almost entirely from proxies and origin servers that disagree about framing. HTTP lets a request say where its body ends in two ways (Content-Length, a fixed size, or Transfer-Encoding: chunked). If a proxy and a server read the same bytes with different rules, they disagree about where one request ends. An attacker hides a second request in that gap. Framing disagreement between two hops is a security boundary, not just a parsing detail.

Where does a bad record fail?

Framing determines where a malformed record fails:

• Length-prefixed, binary, with a schema. The writer must produce a valid frame. A bad record fails at write time, inside the producer, with the producer's stack trace, at a time when the author of the producer is present. This is early binding: the format is checked as soon as the data is created.
• Delimited, text, without a schema. Anything can be written. A bad record fails at read time, inside the consumer, possibly months later, possibly in another team's job, with a stack trace that says nothing about who wrote it. This is late binding: the format is checked only when the data is used.

This is the whole trade. It is the same as schema-on-write versus schema-on-read, one level up. The cost of late binding is not that failures happen. It is that they happen far from their cause, in someone else's system.

The practical rule: if you choose late binding for framing (and text formats are often the right call, for interoperability and splittability), still push validation as early as you can. Validate at ingestion, not at consumption. You do not get early binding's guarantees, but you recover most of its diagnostic value.