How many files a pipeline writes

Worked examples, recomputed

Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.

• 50 × 24 × 120 per hour = 144,000 files per hour
• 50 writers × 100 partitions × 1 commit/min → 7.2 million files a day

Where it appears

Each mechanism states it in its own names. The general symbol each name stands for is shown with it.

• Cost Mechanics

  What is metered, and what is the smallest increment you are charged for?

  $$\text{tasks}\times \text{partitions}\times \text{commits}$$

  Read it in place →

• Table & Storage Maintenance

  Who pays for the gap between the order data arrives in and the order it is read in: the writer, every reader, or a background job, and what is the break-even?

  $$\text{files per interval}\approx \text{writers}\times \text{partitions touched}\times \text{commits}$$

  Read it in place →