Generality vs. specialization
Where you meet it: general engines vs. purpose-built structures.
What it is
Choosing between physical and logical replication is the generality-versus-specialization trade-off (a general format works everywhere but costs more; a specialized one is fast but narrow), and again early versus late binding.
From Data Movement › CDC, replication, files
"Specialized engines always beat general ones for these workloads" is currently true and eroding, not permanent. It is an economic claim about where engineering effort has been spent. The gap narrows each time a general columnar engine absorbs an encoding or an index type. So "should this be a separate system?" is a cost question with a moving answer. The cost is only moved: a separate system relocates the cost from query speed into operations, and a general engine relocates it the other way.
From Specialized Processing Models › Earning its own system
Where it appears
In 10 mechanisms. Each says what the mechanism is about, then how this trade-off shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
This is the generality-versus-specialization trade-off (a general format works everywhere but costs more; a specialized one is fast but narrow), and again early versus late binding.
Read it inChange feeds, replication, bulk transfer and the shapes
- Storage: Physical Organization
What should sit next to what, given that I must decide at write time and find out at read time?
So Z-order is insurance, not optimization.
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
The principle is worth naming: local density adaptivity.
Read it inBitmap indexes and Roaring bitmapsWhat lookup structures forbidHeaps and priority queues
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
The specialization win is real, and the limit on it is a hardware resource that nobody budgets.
- Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
Generality wins on the write path even when it loses on the read path, because indexes are paid for per write and used per query.
Read it inComposite indexes: everything follows from lexicographic orderEquality and total orderWhen is an index possible at all?
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
This is generality versus specialization at its sharpest here.
Read it inRange, interval and as-of joins
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
The payoff is whatever the general-purpose allocator was charging for the freedom you just gave up.
Read it inLifetime is phase-structured
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
> Placement is the layer where the machine stops being a number and becomes a location.
Read it inThe machine becomes a location
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
One line: triples are maximal normalization.
Read it inProperty graph versus RDF, and what graph forbidsWhen does a workload earn its own system?
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Principle: internal and external validity trade against each other, and the exchange rate is what you pinned.
Read it inThreats to validity and scale honestyWhy standard benchmarks exist, and what each stresses
Related
Made on the same pages most often.