Skip to content
About
Main content

Cut the bill

The decision

The invoice for a data platform came in far above plan: warehouse compute, object-storage requests, data crossing zones. Every way to reduce it moves a cost somewhere else, and the escapes differ greatly in what they cost to use. Engineers usually reach first for the technical fix, which is the most expensive one. What is the cheapest change that cuts this bill?

You will end with the cheapest escape that reduces the spend, what it costs to use, and an alarm so the problem shows up before the next invoice.


Before you choose

1

Find where the spend concentrates

Rank queries and jobs by cost, not by . It is routine for the top 1% of queries to carry most of the scan spend, because bytes scanned span about six orders of magnitude while latency spans far fewer. Look there before anywhere else.

2

Compute the effective price, not the rate

What the vendor meters (bytes scanned, requests, provisioned hours), the billing (per second, per minute, minimum charges) and your each move the real price more than the list rate, and they multiply. An engine three times cheaper per hour costs more at 20% utilization than the old one at 80%.

3

State the read/write ratio

How often data is read compared with how often it is written decides file , materialization, compression and tiering at once. Measure it before changing any of them.


Decide

1

Can the workload itself shrink?

Fewer queries, longer schedule intervals, fewer columns, a dashboard refreshed hourly instead of every minute, SELECT * replaced by the columns used.

Why

It is the cheapest escape, because it changes nothing technical. But it is a negotiation with the people who generate the work, not an engineering task.

2

Is the usage steady and predictable enough to buy differently?

A baseline that runs every day, or scan-priced queries that would be cheaper on provisioned capacity.

3

Is the cost driven by how the data is laid out?

Millions of small files multiplying request fees, unpartitioned tables scanned in full, data that cannot be pruned.


Where you can land

Change the workload

Use it when the work itself can shrink.

What it costs: agreement from the people who run the queries and own the schedules; nothing technical.

Change the meter

Use it when usage is steady enough to commit to, or a different pricing model fits it better.

Commit or reserve capacity, or switch between scan-priced and provisioned pricing.

What it costs: you trade a cost risk for a forecasting risk. A commitment sells flexibility: if usage falls, you still pay. A multi-year one is also a bet that prices will not fall.

Change the layout

Use it when file sizes, partitioning or clustering drive the cost.

Compact small files (request fees scale with files × queries, not data volume), partition by the columns queries filter on, cluster so queries can skip data.

What it costs: a permanent maintenance obligation: and reclustering run, and are paid for, forever.

Change the architecture

Use it when nothing cheaper reaches the target.

Move regions, replace the engine, separate storage from compute.

What it costs: the most of any escape, and usually more than planned, because the switching cost was never budgeted.


Whichever you land on

1

Price the switching cost

Moving from a bad design to a good one costs compute, downtime risk and engineering time: rewriting 10 TB of files is a bill of its own. A design is only better if you can afford to get there.

2

Do not mistake a performance fix for a cost fix

Concurrency hides latency but touches nothing that is metered. A thousand parallel requests finish quickly and still cost a thousand request fees.

3

Price the safety you are buying, deliberately

Replication factor, snapshot depth and kept for reprocessing are money paid for the ability to recover. Decide each on purpose; they rarely appear in cost reviews.

4

Give cost an alarm

Nothing in the system raises an error when cost runs away; the bill is the only signal, and it arrives late and totalled. Tag spend to owners, and put caps (governors) on the queries and jobs that can run away.