Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Cron, calendars and data intervals

Why is a run identified by the interval it covers rather than the time it fired?

answers the question "when does something fire?" That is not the question the pipeline is asking. The pipeline asks: "which slice of data does this run own?"

The offset is forced

The derivation is short. Work that covers an interval cannot begin until the interval has closed. A run covering [T, T + Δ) (from T up to, but not including, T + Δ) must therefore fire at or after T + Δ. So the firing time and the covered interval are necessarily offset by at least one period.

Example. The daily run for 5 March covers 00:00 on 5 March up to 00:00 on 6 March. It cannot start before 00:00 on 6 March. Tomás schedules it at 02:00 to leave a margin for late data. So the "5 March" run starts on 6 March.

Once you accept this, you must decide which of the two is the run's identity. It has to be the interval, because the interval determines the output. Two runs of the same interval must produce the same output. Two runs at the same wall-clock time need not.

A naming bug that confused a decade

This is the source of the single most confusing piece of API design in the field. Airflow's historical execution_date was the start of the covered interval, not the time the task ran. That is why a daily job "for the 5th" runs on the morning of the 6th. The name caused years of confusion and was eventually replaced by explicit data_interval_start and data_interval_end. The confusion was a naming bug. The offset itself is not a bug. It is forced.

The interval key as an address

The general form: It makes runs

  • idempotent (same key, same output),
  • comparable (run N against run N−1),
  • backfillable (loop over the key space), and
  • reconcilable against the output (the is the key).

Everything later in this mechanism depends on this key existing.

Calendar rules are a correctness issue

Calendar mechanics are not a formatting detail. They affect correctness.

  • Months have unequal lengths, so a "monthly" interval is not a fixed Δ.
  • Daylight saving time makes local time skip and repeat an hour a year, so an hourly schedule in a local timezone skips an interval or produces a duplicate key. The answer is to schedule in UTC and convert to local time only for display. A wrong event timestamp misplaces one row; a wrong misidentifies a whole run, so the mistake costs more here.

Cron's hidden assumption

Cron also assumes that the previous run has finished. Cron is stateless and fires on time whatever happens. A run that takes longer than its period therefore causes overlap: two instances writing the same output. The fix is a concurrency bound (Teams and limits). Cron on its own is incomplete and always needs such a guard.

The run's identity is the interval it covers, not the clock time it fired; the offset between them is forced, and everything addressable downstream hangs off the interval key.