Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Text formats: CSV, JSON and XML

What do text formats cost the reader, and why are they still everywhere?

Text formats: CSV/TSV, JSON/NDJSON, XML

The constraint: no agreement was reached. So the file must be readable by someone who was told nothing. Therefore:

  1. The alphabet is human-legible (letters and digits).
  2. The structure is delimiter-based (commas, tabs, braces, newlines).
  3. The schema is , so the reader must infer it.

The consequence of 3: the same bytes can legitimately mean different things to two readers, and neither is wrong.

The forced consequences

  • Inference is a sampling problem. A column looks like integers until row 4,000,000 contains an empty string. A reader that sampled only the first 1,000 rows made the wrong choice. Example: a file whose column weight reads 12, 15, 9, ... for four million rows, and then "".
  • Parsing is character by character and full of branches. Text ingestion is often CPU-bound (the processor is the slow part). In many systems the rest of the pipeline is limited by input and output instead. (This is a typical observation. Memory bandwidth also limits scans, as in many analytics workloads, so treat "CPU-bound" as common, not universal.)
  • Delimiters collide with data. If a field contains a comma, the writer must add quoting and escaping. Now the parser is a small state machine. Record boundaries can no longer be found by scanning for a newline, because a quoted field can contain a newline.
id,note
1,"Fragile
handle with care"
2,ok

This is two records on four lines. A reader that starts in the middle of this file and looks for the next newline may start inside the quoted note. So CSV with embedded newlines in quoted fields is not reliably splittable. This surprises teams every year (it is Priya's cut-in-half rows).

NDJSON: framing is worth more than encoding

NDJSON (newline-delimited JSON) is a pure decision. Take JSON, which has no boundary discipline, and add one rule: one record, one line. The newline inside strings is written as \n, so a real newline never appears inside a record.

{"id": 1, "note": "Fragile\nhandle with care"}
{"id": 2, "note": "ok"}

You gain (any worker can jump to a byte , skip to the next newline, and begin) and streamability (you can process line by line while the file is still arriving). The price is only prettiness. It is the cheapest structural win in the whole section. The general lesson: framing is worth more than encoding.

A claim to correct

Text formats are not "for small data" or "legacy". They are the equilibrium (the stable outcome) for boundaries where the other party is unknown or unwilling to agree. Their persistence is an economic fact, not an engineering failure (Principle B: adoption compounds).

In one sentence: text is the format for when no agreement was possible. You pay for that with the and with inference risk,