Skip to content

Alert fatigue is arithmetic Amplification Break-even reuse count Cost per unit of work Cost-aware eviction: value density Costs of a hash shuffle Dictionary code width Endurance budget Erasure coding: overhead and repair Error-budget burn rate Event-time skew Forces per commit Freshness and ETA Freshness lag Headroom a reactive autoscaler needs How big keyed state gets How long a backfill takes How many files a pipeline writes How many runs an experiment needs How many time series a metric makes Junk dimension size Key groups Lease drift margin Load per node under replication Memory grant Memory held by batching Moving data through a rate limit Off-CPU time One commit per catalog round trip Periodic snapshot size Population stability index Reciprocal rank fusion Recovering a streaming job Retention tiers by resolution Roaring container break-even Routing between two price models Serverless cost is memory × time Sliding windows multiply work Sparse index granule The expiry invariant The miss-ratio curve The partition of RAM The real price of spot capacity The value of not reading Transactional sink timeout Usable buffer-pool frames Utilization Varint length Vector batch size What an instrument costs What keeping an old interface costs What polling costs When statistics refresh When to rewrite away deletes Write-back coalescing

The Data EngineeringBeta

Main content

Relationships
The partition of RAM

Relationships

Definition

The partition of RAM

B and P are substitutes, and c·g is not a substitute for anything.

R \geq B + c g + P + O

$B$: Buffer pool; the buffer pool
$c g$: Working memory; concurrency × per-query working memory
$P$: Page cache; the operating system page cache
$O$: Overhead; everything else the process and OS need

Worked examples, recomputed

Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.

96 + 80 + 40 + 16 = 232 GB of a 256 GB server

Where it appears

Each mechanism states it in its own names. The general symbol each name stands for is shown with it.

Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
$R \geq B + c \cdot g + P + O$
Read it in place →

The miss-ratio curve

Previous Page

The real price of spot capacity

Next Page

The Data Engineering

Learn

Mechanisms
Symptoms
Routes

Practice

Labs
Playbooks
Interview

Reference

Trade-offs
Relationships
Glossary

The project

About
Privacy

Text CC BY-NC 4.0 · Lab code MIT

Written by Aayush Sharma