Three clocks: which time is on the record?

Which 'time' is on a record, who wrote it, and how far can we trust it?

The record's clock replaces the machine's clock

In earlier thinking about data systems, time is mostly a cost: latency to reduce, ordering to keep, and a clock to synchronize so that machines can agree which thing happened first. In this mechanism that clock becomes metadata. A different clock becomes the important one: the timestamp carried inside the record, which says when something happened in the world. The machine's own timeline is no longer the authority. It becomes one more column.

This change creates a correctness rule that earlier mechanisms did not have:

Replay a year of history in ten minutes, or process it live at one event per second, and the answer must be bit-for-bit identical. In earlier topics correctness was defined relative to the execution: what a reader sees when writes happen at the same time, or what survives a crash. Here the speed of execution is explicitly not part of the answer. Any place where speed leaks into the answer is a bug, not a performance property. This is exactly what happened to Kavya: the live run and the re-run differ because something in her pipeline depends on when it ran.

The new constraint is about information, not physics

The mapping from event time (when it happened) to processing time (when we see it) behaves like a random function with a delay that has no known upper bound and cannot be observed directly:

• It is not monotonic: events arrive out of order.
• It has no known upper limit: a phone that was offline for a week sends its events a week late.
• Silence is ambiguous. (This is the most important point.) A source that sends nothing might be slow, broken, disconnected, or simply idle. You cannot tell which. You cannot observe completeness. You can only wait longer and become more confident, but never fully certain, forever.

What stops mattering is physical latency. You are no longer trying to make the answer arrive sooner. You are choosing when to declare it.

So the governing question is not "what time is it?" but:

Who stamps the timestamp?

An event is an occurrence. A record is an artifact about it. Between the two there is a chain of custody: the device, the app SDK, an edge server, the message broker, the storage sink. Each handoff is a place where someone could stamp a time.

The first question about any timestamp is therefore: who assigned it, and can that party lie or be wrong? This must be answered before you build watermarks, because it decides whether event time is trustworthy enough to use at all.

Device clocks are not just imprecise. They are adversarial and arbitrarily wrong. Consumer phones have clocks that are wrong by hours, by years, and by "reset to the epoch after the battery died". Users change them by hand. This is exactly Kavya's 1970 and future-dated events.

The three clocks, and why there are exactly three

The three common clocks are simply the three trust boundaries that matter.

Event time

Event time is stamped nearest the occurrence, usually by a device you do not control. It carries the semantic value you want and the trust you cannot have. Nothing later in the pipeline can verify a claimed event time. You can only bound it: reject timestamps in the future, reject absurdly old ones. Those limits are guesses too. This is the clearest case of a question you should ask of every system: what does it assume that it cannot verify?

A small defensive check might look like this (the limits are examples you must choose and document):

from datetime import datetime, timedelta, timezone

def plausible(event_ts: datetime, ingest_ts: datetime) -> bool:
    too_far_ahead = ingest_ts + timedelta(minutes=5)   # device clock fast
    too_old       = ingest_ts - timedelta(days=30)     # absurdly old
    return too_old <= event_ts <= too_far_ahead

Processing time

Processing time is the clock of the machine that does the computation, at the moment it computes. It is always available. It means nothing about the world. It is different on every run. It is the only one of the three clocks that makes a pipeline non-reproducible.

Ingestion time

Ingestion time is stamped at your own trust boundary: when the message broker appended the record, or when the landing zone wrote it. It is the only clock that is at the same time trustworthy, monotone per partition, and stable when you replay. Its defect: it puts your operational health into the meaning of your data. A four-hour outage of your pipeline becomes, forever, a four-hour gap that looks like a statement about the world ("no parcels were delivered between 10:00 and 14:00").

A fourth ordering that is not a clock: log position

There is a fourth way to order records that is not a clock at all: the log position (offset, log sequence number, sequence number). It is assigned by the single writer of a partition. It is gap-free and increasing by construction, which is exactly what clocks cannot be. It gives you a total order without a measurement of time: you can say "before" and "after" but not "how long". Several correctness problems are solved by abandoning time and ordering by position.

This is the old distinction between logical clocks (counters that only capture "happened before") and wall clocks (they measure real time), now seen as a data-modeling choice instead of a coordination problem. It explains the usual division of labour:

(For example, a Kafka consumer commits an offset to remember how far it has read, and the offset, not a timestamp, is used to resume exactly where it stopped.)

The skew distribution

The most important number for the rest of this mechanism is the skew of each record:

Its median is roughly your pipeline latency. Its tail is the world misbehaving. Worked example: a median skew of 2 seconds with a p99.99 skew of 3 days is normal for a mobile fleet. 3 days = 3 × 86,400 = 259,200 seconds. The ratio between the worst typical case and the normal case is 259,200 / 2 = 129,600, about 10⁵. A ratio between 10⁴ and 10⁵ is the usual range. So every waiting decision here is a percentile decision, not an average decision, and it is sharper here than anywhere else, because the tail is so much longer than the middle.