Structured logging

Why are logs the costliest signal per insight, and what can only logs give you?

A log is a record of one event, usually text. An unstructured log line is a projection into a string that must be re-parsed by a regular expression at read time:

2026-03-02 02:14:09 ERROR order 8841 failed for user 77 after 3 retries: timeout

That is schema-on-read dressed up as an operations tool. Structured logging is early binding: you serialize fields when you write, pay bytes and CPU then, and get indexability and cheap queries later.

{"ts":"2026-03-02T02:14:09Z","level":"ERROR","event":"order_failed",
 "order_id":8841,"user_id":77,"retries":3,"reason":"timeout"}

Logs are the most expensive signal per unit of insight, because nothing aggregates. Volume is event rate × bytes per event, and both grow with the system. The escapes:

• level filtering (keep only WARN and above in normal times);
• sampling (keep one in N of the routine lines);
• raising verbosity dynamically only during an incident.

The last one is a precompute vs. compute-on-read trade, where the "read" is a rare event. You pay for detail only when you need it.

How log stores are built

Log stores converged on one design: index the labels, brute-force scan the body. The alternative, a full inverted index (a map from every word to the list of lines containing it) over free text, has a cost that grows with the number of distinct terms, which is unbounded (order numbers, IDs, timestamps). But the real query is nearly always "a narrow label filter, plus search for a rare word, in a small time window." Scanning a small slice of compressed text is cheap. Independent systems arrived at the same split. That agreement is the evidence. The reason is the read-optimized vs. write-optimized trade: a big index makes every write more expensive for queries that rarely need it.

The irreplaceable property

Logs are the only signal that keeps the specific record. No metric will ever tell you which row broke the job. (This is the third symptom of Nadia's problem.)