Dynamic task generation
Why is fan-out a batch-size decision, and what else does it cost?
Dynamic task generation (fan-out over discovered inputs) makes the graph's shape a function of data rather than of code. It is useful for per-partition, per-file and per-tenant work where the count is not known when you write the code.
It is a batch-size decision
The sizing rule is the fixed-cost law without disguise. Launching a task has a fixed cost: dependency resolution, container or slot acquisition, log setup and state writes. Call it tens of seconds. Generating N tasks costs N times that, plus N rows in the orchestrator's metadata database and N entries in every UI.
The mechanism is correct when the work per unit is much larger than the launch cost, and wrong when it is not.
The correct shape is one task that loops, or a fan-out over batches of files, for example 100 batches of 100 files. This is exactly the batching derivation from the hardware level (do not do tiny operations; combine them), reappearing with the fixed cost moved from a disk seek to a container start.
Two further costs
- Comparability across runs dies. If run N has 400 tasks and run N+1 has 12,000, you cannot compare them, you cannot attribute a duration regression, and you cannot retry "the same task", because task identity came from a listing that has changed. Stable, content-derived task IDs (IDs computed from what the task processes, such as a hash of the file name) partly fix this, and most implementations do not use them.
- Parse cost scales with data. The scheduler must evaluate the generating expression to know the graph. If that expression lists an object-store prefix, the scheduler now does I/O proportional to the data's size on every parse cycle. A loop with a fixed rate meets a growing input, and the scheduler falls behind on every workflow, not only this one.
Fan-out is a batch-size decision: below roughly a minute of work per unit, you are paying orchestration overhead for nothing and the metadata store is the thing that breaks.