Catalog APIs and webhooks

Why must a catalog become an arbiter, and why must a webhook receiver tolerate the same message twice?

Catalog APIs

What a catalog is, and what it can claim, is metadata's subject. The new thing at a boundary is the shift from a catalog as a database that engines read to a catalog as a service that engines call, and why it happened.

The forcing constraint is multi-engine access. If several independent engines can write the same table, the atomic commit of the metadata must be enforced in one central place, because no engine can trust the others to have done it right. A shared database makes every engine implement the same transaction logic correctly, an assumption nobody can check. A service makes the commit an operation the catalog performs itself, so correctness has to be right in one place. (This is the same argument that produces a coordination service in distributed systems, here arriving at the interface level.)

The second consequence. Once the catalog is a service, it sits in the authorization path. It can hand out scoped, short-lived credentials per table, instead of letting each engine hold blanket access to storage. The interface becomes the enforcement point because it is the only place where both the identity and the object are known at the same time. This is a general pattern: the narrow waist becomes the policy point (the narrow waist is the single thin interface that everything passes through). It is why interfaces collect governance duties they were not designed for.

In one sentence: the moment more than one engine can write, the catalog must stop being a place and become an arbiter.

Webhooks and callbacks

A webhook is a push: when something happens, the producer calls a URL that the consumer supplies. It resolves push versus pull at the boundary and inverts the client/server relationship: the consumer must now run a server, and the producer becomes a client with all of a client's problems.

Work through what that forces:

1. Duplicates are unavoidable. If the producer does not get a response (or the response is lost), it cannot know whether delivery worked, so it retries. There is no setting that removes this. The receiver must be idempotent.
2. Retries amplify load. Retrying against a slow receiver creates retry storms. So the producer needs backoff (waiting longer between tries), retry budgets and a dead-letter path (a place where messages go after too many failures).
3. Order cannot be guaranteed, because retries reorder messages. The receiver must tolerate out-of-order arrival or use sequence numbers.
4. The receiver's availability becomes part of the producer's pipeline. This is coupling in a direction nobody planned.

A minimal idempotent receiver:

def handle_webhook(event):
    # event["id"] is a unique key chosen by the producer.
    if already_processed(event["id"]):       # lookup in a durable store
        return 200                           # safe to acknowledge again
    apply_business_change(event)             # e.g. mark invoice paid
    mark_processed(event["id"])              # ideally in the SAME transaction
    return 200

Compare with polling:

Push's failure modes are all consequences of missing information about the consumer's progress. That is exactly what the next article makes durable. A webhook is the boundary case where the conversation has no memory, so everything must be re-derivable from the message itself.

In one sentence: push moves latency down and state up. The receiver must be able to be told the same thing twice.