Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Where you meet it: schema-on-write vs. read, compile vs. interpret, ID- vs. name-based column resolution, contracts vs. inference.
What it is
Where the schema lives is a choice of early versus late binding. Binding means fixing the meaning of something. Early binding fixes meaning in advance (before reading), which moves cost off the read path and onto the agreement. Late binding fixes meaning at read time and does the reverse. Early binding is the spine of this whole section.
From Interfaces & Formats › Where the schema lives
Every mechanism in this article sits on one dial: early versus late binding, meaning whether a decision is made from the query text (before seeing data) or from the data (after). Static versus dynamic pruning is one position on it. Early versus late materialization is another. Metadata is space bought to save time; statistics written at commit time are precomputation; and on engines that bill by bytes scanned, every byte not read is money.
From Query Planning & Optimization › Late materialization, sampling
A sketch has one more limit, and it has operational teeth: you cannot recover the data from the sketch. Sketching is an early decision about the question. A rollup table of HLL sketches keyed by day and three dimensions has committed you to those dimensions. A fourth dimension means reprocessing the raw data, if you still have it. Sketch-based pre-aggregation trades query cost for the permanent loss of unasked questions, and the bill arrives months later, when someone asks a reasonable question that the pipeline structurally cannot answer.
From Core Data Structures › What probabilistic structures forbid
Where it appears
In 22 mechanisms. Each says what the mechanism is about, then how this trade-off shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
Early versus late binding: when do you find out that a record is bad?
Read it inWhat changed: from one machine to twoProtocol framing
- Storage: Physical Organization
What should sit next to what, given that I must decide at write time and find out at read time?
If you declare it in the schema, you place the bet earlier and with less information: you must predict at design time which columns are read together.
Read it inLayout models: row, column and PAXOpaque values, documents and where order comes fromPartitions, buckets, row groups and zone maps
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
Sketch-based pre-aggregation trades query cost for the permanent loss of unasked questions, and the bill arrives months later, when someone asks a reasonable question that the pipeline structurally cannot answer.
Read it inWhat probabilistic structures forbid
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
Schema-on-write does not remove read-time failures.
Read it inColumnar files, Arrow and schema resolutionSerialization: where the description lives
- Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
The lesson: choosing a primary key looks like a logical modelling decision and is actually a physical layout decision.
Read it inPrimary versus secondary: identity is not location
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
It is early versus late binding (deciding the schema before or after seeing data) appearing inside a single operator.
Read it inSet operators, pivots and recursion
- Query Planning & Optimization
How does a system choose a plan for a question when the facts that decide the best plan are unknown until the plan has run?
An optimization that depends on a relationship between two columns needs that relationship to be declared somewhere the optimizer can read it.
Read it inMetadata-only aggregates and partition pruningAdaptive query executionCompile or interpret, materialize or pipelinePrepared statements and plan cachingA pipeline of one-way stepsParsing, the logical plan and the physical planThe reframe: a layer limited by what the system can knowThe three kinds of optimization
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
It cannot be preempted, so it must be rationed before the work starts and reformulated when the ration runs out.
Read it inMemory cannot be time-slicedColumnar memory pooling (Arrow-style buffers)What it costs to take a resource backScan resistance by declaration
- Caching
What happens when we deliberately keep a second copy of data that no protocol forces to be correct?
A query plan is a declaration of the future access sequence.
Read it inEviction has a proven limitMaterialized views: a cache whose admission was declared
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
Seen from the other side, a lease is a cache entry whose value is authority and whose freshness bound is a TTL.
Read it inLocks, leases and fences: why mutual exclusion is unavailable
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
> When you cannot measure a number, you can sometimes price it so that the holder volunteers it.
Read it inBackfill schedulingFIFO, priority and multi-level feedback
- Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
A multi-year commitment is a bet against price deflation, and the bet is usually not stated.
- Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
Storing local time is therefore early binding (fixing a value early) of something defined by an outside authority that will change it.
Read it inTime zones and calendarsThree clocks: which time is on the record?
- Streaming
What does it cost to remember, when the input never ends?
Correct enrichment requires the dimension to arrive as a stream with its own versions and its own watermark.
Read it inStream-table joins and enrichmentSavepoints and state schema evolutionWindows are retention rules in time
- Change Data Capture & Incremental Processing
What does it cost to know what changed, and is that cheaper than not needing to know?
In one line: if arrival order and partition order agree, incrementality is a watermark comparison and costs almost nothing.
Read it inWatermark-based incremental batchReplay and dual-write avoidance
- Data Modeling
What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?
Early versus late binding (decide structure at load time or at read time).
Read it inA layer with no oracleAnchor modelingData VaultEvent sourcing and activity schema
- Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
A pinned required core, plus a late-bound payload.
Read it inSchema-on-write versus schema-on-readVariant and union columnsDefaults, nullability and the three-state problemVersioned schemas and migrationPartition spec evolutionIdentity by name and by ID
- Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
The trajectory of this whole field over the last decade is converting advisory metadata into load-bearing metadata, because load-bearing metadata is the only kind that stays true.
Read it inTechnical catalog versus business glossaryLoad-bearing versus advisoryAsset identity: what is the row keyed on?Statistics stores
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
But the bucket boundaries are fixed when you emit (this is early binding: you decide the structure before you know the question).
Read it inMetrics: counters, gauges and histograms
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
The deeper constraint is informational: normalization destroys distinctions at write time, and no read-time cleverness recovers them.
Read it inAnalyzers and tokenizersEmbeddings, and why exact indexes fail
- Interfaces & Formats
What can two parties agree on when they cannot talk to each other?
In one sentence: zero-copy is not a speed trick.
Read it inColumnar files and in-memory formatsWhere does the schema live?Connector frameworks, ELT and ETLWhat interfaces forbid: a treaty, a conversation, a bookmark
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Fix the decision rule while you are still ignorant of the outcome.
Read it inPredictions, hypotheses and variables
Related
Made on the same pages most often.