Scenarios

Requests = files touched × queries. With 10 million files and 1,000 queries a day each touching 1%: 10⁸ GET requests a day, about $40 a day or $15,000 a year, against about $2,000 a year of storage.

Too many files touched per query: millions of 1–5 MB files. The fee scales with files × queries, not bytes, and it is a large share of cost even above the 100 KB break-even. Parallel reads hid the latency, so no performance metric flagged it.

Compact to about 128 MB files, which cuts requests about 25-fold.

Compaction is a permanent maintenance job, large files prune less finely, and manifests need their own compaction and snapshot expiry. Measure files per partition monthly.

Spot is 60–90% cheaper, but with 100 nodes at 5% per hour, the whole job survives an hour with probability 0.6%, and the first interruption comes after about 12 minutes.

The coordinator and shuffle state on on-demand machines, workers on spot, and a checkpoint every 5 minutes, so an interruption loses about 2.5 minutes of work on average rather than 3 hours. Expect about 40% of the on-demand cost: the real price of spot includes the lost work.

Spot capacity disappears when total demand peaks, so keep an on-demand fallback. Handling interruptions costs engineering time, and the job must be able to recompute lost pieces from lineage.

If the idle timeout is shorter than the gap between queries, every query pays for a billed cold start; if longer, the warehouse never suspends. Sparse traffic is always cold and never idle.

Elastic capacity wins only when the duty cycle × the elastic premium is below 1; here the duty cycle is near 1 and the premium 3, giving 3. A 30-second cold start at $4 an hour costs about $0.033; a 10-minute idle tail, about $0.67.

Replace it with a small always-on warehouse, or set the timeout above the typical gap if the premium is lower.

A monthly feedback loop cannot catch changes that multiply cost within hours.

A dry-run cost estimate with a cap (for example 1 TB unless approved), a minimum schedule interval (say 1 hour), and same-day anomaly detection (alert at 3 times the 7-day average). Under those rules, 288 runs × 400 GB ≈ 115 TB a day is stopped on day one.

Tags (team, pipeline, environment), per-team warehouses (accepting the loss of sharing, since attribution cannot be finer than isolation), and unit economics using two denominators that are biased in opposite directions.

What the cap's false positives cost, and that any alert still has some lag.

An unused committed unit costs the committed price (0.6 of list); a missing one costs only the discount (0.4). Commit at the forecast percentile equal to the discount (the 40th for 40% off), not the peak; unused commitment is a flat line nobody notices.

If list prices fall 20% a year, the real saving is about 32%, not 45%, and almost all of it is in the first year. A one-year deal at a smaller discount may be better.

Check how bytes are counted, compressed or not, since that changes what compression is worth.

Convertible commitments lower the cost of over-committing.

Build $30, a daily refresh at $1 for 30 days, and storage $5, total $65; saving $0.50 per query gives N* = 130 queries a month, so 1,000 clears it easily. But if the table changes hourly with $10 refreshes, refreshes alone cost $7,200 a month, and it never breaks even. Incremental refresh shrinks the refresh cost from the table's size to the change's size, about 100× for 100 GB of changes on a 10 TB table.

On a cluster you already pay for, the marginal cost is near zero; under pricing per byte scanned, it is the full rate.

The liability of keeping the view correct appears on no bill.

20 × 100 ms = 2 seconds per page, against about 20 ms within one region, plus egress charges per byte. Nothing at the call site shows that a region boundary is crossed.

Co-locate the service and the data, cache, or batch the calls; add a lint rule or a warning whenever a client's region differs from its database's.

Copying data wins when the total bytes of results over the planning horizon exceed the source's size; querying it remotely wins for rare, selective access, provided filters are pushed down to the source.

Build a table: for each meter, the quantity, billing increment, minimum, expected utilization and rate, including request counts that depend on file sizes. For example, 50 GB a day kept for a year is about 18 TB, about $420 a month at $0.023 per GB-month; writing it as 1 KB objects would mean 50 million PUT requests a day.

Price the labour: $4,000 of work saving $400 a month takes 10 months to pay back, before maintenance, so it is marginal; it is worth doing when spend is on a growth curve.

Change the workload first, then the meter, then the layout, then the architecture.