Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Canaries and synthetic transactions

How does manufactured traffic observe absence, and what can it never prove?

Passive covers only the traffic that exists. Rare paths and idle periods are unobserved, and an absence produces nothing to observe. Synthetic traffic manufactures coverage. A is a test record you inject on purpose: for example, a fake order written at the source every 5 minutes and asserted at the sink ("it must appear in orders_clean within 10 minutes").

What a canary measures:

  • end-to-end liveness (does the whole path work at all?);
  • true end-to-end latency, including hops that emit nothing when idle.

Costs and limits:

  • Contamination. The canary must be distinguishable and excluded downstream. Otherwise fake orders end up in revenue. That filter logic leaks into every consumer's query.
  • Coverage only along the path the canary takes. A canary pinned to one key tells you nothing about a hot elsewhere.
  • Boundary condition: , never correctness at volume or under .
# Every 5 minutes: write a marked fake order, then check that it arrives.
canary_id = f"canary-{uuid4()}"
source.write({"order_id": canary_id, "is_canary": True, "amount": 0})
if not wait_until(lambda: sink.exists(canary_id), timeout_s=600):
    page("canary order did not arrive at the sink within 10 minutes")