Task groups and reusable components
Why should grouping help the human but never become a scheduling unit?
Task groups, sub-DAGs and reusable components
Two different needs get mixed together here. Separating them explains the historical mess.
- Visual and naming: collapse forty tasks into one box and give their IDs a consistent prefix. This is pure presentation. It is free and uncontroversial.
- Reuse: parameterize a subgraph and create it many times. The tension: reuse wants the subgraph to be an opaque unit, while debugging wants every task flat and individually addressable, retryable and inspectable. One construct cannot give both. Attempts to do so produce an abstraction that hides exactly the task you need to see when it fails at 3 a.m.
A historical failure
The instructive failure is making the group a scheduling unit. A sub-DAG implemented as a parent task that waits for its child tasks occupies a slot while its children compete for slots. That is the sensor deadlock, found again. Airflow's SubDAGs were deprecated for exactly this reason. The replacement, TaskGroups, is deliberately only a naming and display construct with no scheduling meaning. Treat this as evidence for a general rule, not as product history:
The stable position: groups are namespaces; reuse happens at definition time by generating tasks from a shared function; the runtime sees a flat graph.
Group for the human, flatten for the scheduler — any grouping construct that holds a slot while its members queue for slots is a deadlock waiting for load.