Dataset-aware scheduling

What does depending on a dataset name instead of a job buy, and where does it stop?

Dataset-aware (data-driven) scheduling

The two earlier mechanisms bind a consumer to a producer, to its schedule or to its notification. Dataset-aware scheduling binds it to an artifact name. Tasks declare their inputs and outputs (reads: sales.daily, writes: sales.daily), and the scheduler derives the arrows by matching names.

Rebinding

The gain is a rebinding, and it is the important part. A consumer no longer needs to know which job produces a table, how often it runs, or that it was rewritten last quarter. It needs to know only the table's name. Producer identity is separated from artifact identity. This is the same indirection as content-addressed storage (finding data by a hash of its content) or a service registry (finding a service by name). The second-order effect matters more than the first: cross-team dependencies stop requiring a conversation about schedules. They require only an agreement about a namespace.

A build system with a time axis

This is Make with time. (Make is the classic build tool: given a target and its declared dependencies, it works out what is out of date and rebuilds it.) The difference is instructive. Here the inputs are unbounded and constantly appended to, so "stale" cannot mean comparing file modification times. It must mean "a new version or partition exists that this consumer has not consumed." So the mechanism works only over stores that expose a version or partition-arrival signal. That is why it appeared together with modern table formats (formats such as Iceberg or Delta, which keep a snapshot history of a table) and not before them. Snapshot metadata is what makes this scheduling model possible.

Two limits

Granularity. If a consumer declares a dependency on a whole table while its real dependency is one partition, every producer write wakes every consumer, and most of them then do nothing. The declaration is coarser than the truth. This is granularity mismatch causing amplification, appearing as spurious wake-ups instead of write amplification.
No validation. The declaration is still unverifiable. A task that reads a table it did not declare gets no arrow and no error. The scheduler trusts the manifest. Only runtime lineage capture can check it, and that is a separate system.

Depend on a name, not on a job: it decouples consumers from producer identity, at the price of a declaration nobody validates and a granularity coarser than the real dependency.

Case, resolvedConstructed case · Tomás at Marigold Market

Every worker busy, and nothing running

The delivery partner uploaded its file at a different hour each day, and 300 sensors waiting for it filled every worker slot until nothing else could run. Where it started: Triggers →

The frozen workers. 300 sensors times four hours of waiting meant 1,200 slots wanted from a pool of 100: a deadlock waiting for load. Tomás does three things.

Moves the sensors to deferrable mode. They register with one shared poller and give their slots back. The poller checks the object store every 5 minutes. Now the waiting costs about 1 shared poller, not 1,200 slots. Mean detection delay is 2.5 minutes, instead of waiting until 11:00, the latest arrival time, to be safe (up to 8 hours late when the file comes at 03:00).
Asks the partner to write a _SUCCESS marker after the last file. This is a sentinel. He also adds an expected file count in the manifest, because a sentinel alone is lost if the partner's job crashes between the last file and the marker. A final timeout at 14:00 raises an alert instead of guessing.
For the tables of other internal teams, he declares dataset names instead of polling their output folders. The refund team's table is now refunds.daily, and a change of its folder layout no longer breaks him.

He knows the limit: if the partner forgets the marker, the pipeline waits until the timeout, and the timeout raises the alert.