Choose a distributed commit
One business operation writes to several systems, say an orders database, an inventory database and a payment API. Do you make it all-or-nothing, so every system commits or none does? Or do you let each system commit on its own, and handle failures another way?
You will end with one design, with what it costs you and what to check before you build it.
Decide
Questions 1–4 are for when the writes must be atomic. Questions 5–6 are for designs without an atomic commit.
Must these writes succeed or fail together?
For example: is it acceptable for a short time that the order exists but the stock is not yet reserved?
Why
Most "distributed transaction" requirements are met by steps that each commit locally, with a way to repair failures. An atomic commit across systems costs lock time and availability, so ask before you design.
Can every system 'prepare': promise to commit, then wait for the decision?
Databases that support XA or two-phase commit can. An external API usually cannot: a payment provider will not hold your charge open until you decide.
YesGo to 3
NoAtomic commit is not available here. Go to 5: you will design without it, and give up atomicity
Why
An atomic commit needs every participant to vote "yes, and I will commit if told", holding its locks until the decision arrives. A system that cannot do that cannot take part. This is a capability check, not a preference.
Can readers find and finish an incomplete commit themselves?
True when you build on a store with atomic writes to a single row and you control how every read works, as Percolator did on Bigtable and TiDB does.
Why
If the commit decision is written into a row that every reader must consult anyway, a reader that meets an unfinished transaction can settle it, so no coordinator has to survive.
Are each transaction's reads and writes known before it starts, with no user interaction mid-way?
Why
Two-phase commit exists to find out whether any participant wants to abort for a local reason. If all participants run the same transactions in one agreed order, every outcome is determined in advance, and there is nothing to vote on. That needs the access sets up front.
Is it one database write that must reliably trigger one message?
For example: mark the order paid, and publish 'order paid' to a message bus.
Why
Writing to a database and publishing a message are two systems, with no single write that covers both. The outbox moves the message inside the database's own transaction, which makes this case simple.
Is repeating each step's effect harmless?
Setting status = 'shipped' twice is harmless; subtracting 10 from a balance twice is not.
Why
If every effect can be repeated safely, retrying until each step succeeds is enough, and nothing ever needs undoing. If not, each step needs a compensating step that undoes it when a later step fails.
Where you can land
Two-phase commit
Use it when the writes must be atomic, every system can prepare, and neither simpler design fits.
A coordinator asks every participant to prepare, then tells all of them to commit or abort.
Before you build it
Lock time
Rows stay locked for local work + 2 network round trips + 2 forced log writes. Across regions, at 70 ms per round trip, that is about 150 ms. A hot row locked for 5 ms serves 200 transactions a second in turn; locked for 150 ms, it serves 6.7. If your hot rows cannot absorb it: repartition so the transaction stays on one machine.
Why
The round trips are not what hurts; the lock time is. Longer locks mean more transactions waiting at once, and conflict grows with the square of that number.
Availability
Every participant must be up, so their availabilities multiply: ten systems at 99.9% each are up together about 99% of the time. If that is too low: replicate each participant; each one's availability improves, but the multiplication stays.
Stuck transactions
If the coordinator dies after a participant voted yes, that participant can neither commit nor abort, and its rows stay locked until someone resolves them. Before launch: write down how an operator finds in-doubt transactions, decides each one, and reconciles the systems.
Why
This outcome cannot be designed away. Real deployments let an operator force a decision, knowingly risking atomicity, because rows locked indefinitely are worse than a rare inconsistency fixed by hand.
Ordering by real time (only if you need it)
Clock disagreement between machines must be paid somewhere. With a tight, guaranteed bound (Google's TrueTime), each commit waits it out, a few milliseconds. With a looser one (hybrid logical clocks), reads that meet an uncertain value restart. Choose by your read/write ratio.
Decision stored in the data (Percolator style)
Use it when your store has atomic single-row writes and you control every read path.
One row is the primary. The transaction commits the moment one atomic write lands on that row; any reader that meets a leftover lock follows it to the primary and finishes or rolls back the transaction itself.
What it costs: a timestamp service that every transaction goes through; read latency spikes when readers clean up abandoned transactions; extra writes for locks and cleanup; and it gives snapshot isolation, not serializability. It was built for throughput workloads where seconds of latency are acceptable.
Deterministic execution (Calvin style)
Use it when every transaction's reads and writes are known before it starts, and none is interactive.
A replicated sequencer agrees the order of all transactions first; every partition then runs its share in that order. No vote, no prepare, no stuck transactions.
What it costs: access sets declared up front (guessing them with a cheap pre-read brings back optimistic validation); the sequencer bounds throughput and adds batching delay; interactive transactions are impossible.
Transactional outbox
Use it when one database change must reliably trigger one message.
Write the message as a row in an outbox table, in the same local transaction as the change. A relay publishes the rows and keeps retrying until the broker acknowledges.
Before you build it
Consumers handle duplicates
The relay delivers at least once, so duplicates are guaranteed. A consumer that cannot handle them is a bug waiting to happen, not a risk.
The outbox table
Rows are inserted and deleted at a high rate, which bloats a database table. Mature setups read the outbox through change data capture instead of polling it. Publishing the database's log directly removes the table, but ties every consumer to your table layout.
Ordering
If messages for one key must stay in order, the relay must be single-threaded per key, or partition-aware.
Idempotent pipeline
Use it when each step can be repeated safely, so retrying until it succeeds is enough.
Before you build it
Make it naturally idempotent first
Express the effect so a repeat does nothing: set a value instead of adding to it. That costs nothing and has no failure mode.
If you must deduplicate
Store each processed event ID and skip repeats. The store needs a retention window, and a duplicate that arrives after its ID was evicted is applied silently. Choose the window knowing nothing will tell you if it is too short.
Saga
Use it when several steps across services must all happen, and each can be undone by a compensating step.
Each step commits on its own; if a later step fails, the earlier ones are undone by their compensations.
What it costs: atomicity is gone and intermediate states are visible: another service can see a seat as sold before your saga releases it. The usual mitigation is a "pending" status that others agree to respect.
Before you build it
Every compensation meets four requirements
It can be retried safely; it copes with arriving before the step it undoes; it eventually succeeds (in practice, the last resort is a queue for a human); and it exists at all. You cannot un-send an email or un-ship a package.
Put the irreversible step last
If the step that cannot be undone comes last, it decides whether everything before it counts, which gives you a commit point. Where that is impossible, defer the irreversible effect: authorize a card payment, and settle it only at the end.