Connector frameworks, ELT and ETL
Why is a connector an offset store, and what does the order of transform and load decide?
Connector frameworks
The N-by-M argument from database drivers returns here with a twist. Connectors split a pipeline into source, transport and sink, so each system is integrated once, not once per partner. But the thing being factored out is new, and it is the one we just derived: offset management.
What a connector framework gives you is not "connectivity". It is:
- a place to put the bookmark,
- a rule for when to move it,
- a promise to restart from it.
Everything else is convenience. That is why connector frameworks feel heavy for a one-time copy and necessary for a fleet: you are not paying for data movement. You are paying to have the position bookkeeping done the same way a hundred times.
The leak: what is a position?
The abstraction leaks in one predictable place. Sources differ in what a position even is:
| Position kind | Resumable? | Monotonic (only increases)? | Unique? |
|---|---|---|---|
| Log offset | Yes | Yes | Yes |
| Timestamp high-watermark (largest timestamp seen) | Only if writes arrive in order | Not if writes arrive out of order | No (many rows can share a millisecond) |
| Page cursor | Depends on the provider | Depends | Depends |
| Primary-key ceiling (largest ID seen) | Yes if IDs only grow | Often | Yes |
| Vendor opaque token | Only by the vendor | Unknown | Unknown |
A framework that treats these as interchangeable is lying about resumability. The rule to keep:
Total means that any two positions can be compared (one is before the other). A timestamp position on a source that allows out-of-order writes or same-millisecond writes cannot be advanced safely, and no framework hides that. This is the point that independence is assumed everywhere and true almost nowhere: the assumption that records arrive in position order is false in most real sources.
Commit order
There is a classic question: do you move the bookmark before or after the sink confirms the write? Move it before, and a crash loses data. Move it after, and a crash causes duplicates, so the sink must be idempotent. For interface design, the lesson is this: the source cannot tell you which is safe, so the connector must know, per source, and that knowledge is not portable.
In one sentence: a connector is an offset store with data movement attached; the data movement is the easy half.
ELT versus ETL as mechanism orderings
ETL means Extract, Transform, Load: change the data before it reaches the destination. ELT means Extract, Load, Transform: land the raw data first, then change it inside the destination. Without the vocabulary this is a question about where the transformation runs, and three consequences follow from the answer.
| Transform before load (ETL) | Transform after load (ELT) | |
|---|---|---|
| What the destination receives | Only conforming data | Raw data as the source sent it |
| When errors appear | Early | Later, when transforms run |
| If the transform has a bug | The raw form was discarded or held elsewhere: you must re-extract from the source | The raw form is kept: recompute inside the destination |
| New consumers with different needs | Cannot go back to the original | Can use the original |
| Source schema changes (drift) | Stopped at the transform step (a firewall) | Flow through to every consumer |
This settles two choices together: precompute versus compute-on-read and early versus late binding. The decisive property is not efficiency. It is reprocessability: the ability to fix a mistake without going back to the source.
Why the industry moved: economics
The usual story is muddled, so state it exactly. Transform-before-load was mandatory when destination storage was expensive and destination compute was fixed and scarce: you could not afford to store what you would not use, and you could not borrow the warehouse's CPU for cleaning. When storage became cheap and compute became elastic (you can rent more on demand), both constraints disappeared at the same time.
This is the conserved-cost idea at work: the transform cost did not vanish. It moved to where compute is cheaper to buy. And it is a contingent fact, not a principle. The order flips back wherever the destination's compute is still scarce, which is why ETL persists where each query is expensive.
The cost people underweight
When you land raw data, you have accepted the source's schema, including its drift, as a permanent input to your system. The early transform was also a firewall. Removing it moves the pain of schema change downstream to every consumer.
In one sentence: transform late and you can fix mistakes; transform early and you cannot, but you never inherit the source's mess.