How to Build Webhooks That Survive Retries and Replay

"Call us when something happens" is the whole pitch of a webhook, and it hides four separate operational problems: the same event can arrive twice, two events can arrive out of the order they happened, a delivery can vanish during an outage, and your endpoint can go down for an hour while the sender keeps trying. Every one of those is normal, not exceptional. If your handler assumes single delivery and correct order, it will eventually double-charge a customer or apply an inventory update backwards.

We run this problem in both directions on Retailer OS, which pulls order and inventory events from Google, Meta, Amazon and eBay and pushes its own updates back out to those same marketplaces. The marketplaces retry aggressively when they don't get a fast 200, and they don't guarantee order across parallel workers on their end. Our own outbound webhooks to POS terminals and third-party integrations have to make the same promises we demand from the platforms we consume.

The fix isn't a smarter webhook library. It's designing the four failure modes in from day one: retries, ordering, idempotency, and replay. Skip any one of them and the other three won't save you.

Delivery is a best effort, not a promise

Almost every webhook provider you'll integrate with, and almost every webhook system worth building yourself, gives you at-least-once delivery. That means the sender will retry until it gets an acknowledgment, and it means duplicates are a certainty, not an edge case. Exactly-once delivery across a network boundary is not something you can buy off the shelf; systems that claim it are doing deduplication on your behalf under the hood, which is exactly what you should be doing yourself.

Design for duplicates from the first line of handler code, not after the first incident report.

Idempotency keys are the only real fix for duplicates

Every event that matters should carry a stable identifier, and your handler's first job is to check whether that identifier has already been processed before it does anything else. A unique constraint at the database level, not an in-memory cache, is what makes this hold up under concurrent retries.

create table webhook_events (
  event_id text primary key,
  source text not null,
  payload jsonb not null,
  status text not null default 'received',
  received_at timestamptz not null default now(),
  processed_at timestamptz
);

-- handler, pseudocode
async function handle(event) {
  const inserted = await db.insert('webhook_events')
    .values({ event_id: event.id, source: event.source, payload: event })
    .onConflict('event_id').ignore();
  if (!inserted) return respond(200); // already seen, ack and stop
  await applyBusinessLogic(event);
  await db.update('webhook_events').set({ status: 'processed' }).where({ event_id: event.id });
  return respond(200);
}

The insert-and-check-conflict pattern beats a "select then insert" check because two retries arriving milliseconds apart will both pass the select. Let the database's unique constraint be the source of truth, not application logic racing itself.

Don't trust the order envelopes arrive in

Retries reorder events. A sender's first attempt at event 5 can time out, get retried, and land after event 6, which was delivered cleanly on the first try. If your handler blindly applies whatever arrives, you'll overwrite a newer state with an older one.

The fix is a sequence number scoped to the resource, not the global event stream. Attach it at the source if you're the sender: an order's version, an inventory row's updated_sequence. On the receiving side, compare the incoming sequence to the last-applied sequence for that specific resource before writing:

• If incoming sequence is less than or equal to what's stored, log it and skip — it's stale, not an error.
• If incoming sequence is exactly one more than what's stored, apply it and advance the counter.
• If incoming sequence skips ahead, either hold it in a short buffer waiting for the gap to fill, or accept the gap and reconcile later depending on how much the gap matters.

That last branch is a real business decision, not a technical detail. For inventory counts feeding a storefront, we accept the gap and let a periodic reconciliation job catch drift. For anything touching money, we hold and wait, because a wrong number for thirty seconds is cheaper than a wrong charge.

Retry policy: yours and theirs

You're on both sides of this. As a receiver, your only job is to acknowledge fast and process idempotently — return 200 as soon as the event is durably recorded, then process asynchronously. A handler that does business logic synchronously before acknowledging will time out under load and trigger the sender's retry storm, which then compounds the duplicate problem you already have.

As a sender, you need an explicit policy, not "retry until it works":

1. Retry on 5xx and network errors, never on 4xx — a 400 means the payload is wrong and retrying won't fix it.
2. Use exponential backoff with jitter: something like 1s, 5s, 30s, 5m, 30m, capped at five to eight attempts.
3. After the final attempt fails, write the event to a dead-letter table instead of dropping it silently.
4. Alert when the dead-letter table grows, not when an individual delivery fails — one failure is noise, a growing backlog is an incident.

This is the same shape of problem we cover in rate limiting a free tier without losing real users: the policy has to distinguish a customer's transient blip from a systemic failure, because the wrong response to each looks identical from the outside.

Replay has to be a first-class feature, not a support ticket

Every dropped or failed webhook eventually becomes a support conversation: "the order never showed up in our system." If your only record of the event was the sender's attempt log, you're at their mercy for how far back they'll look and whether they'll resend at all.

Keep your own immutable log of every event you receive, raw payload included, before you touch it. That table is cheap to write and it does two jobs: it backs the idempotency check above, and it gives you a replay source you control. When a handler bug ships and corrupts three hours of processed events, you fix the bug and replay those three hours from your own log rather than begging an upstream provider to resend.

This is the same discipline we use for migrating a production database off a platform we decommissioned: every cutover was gated by a per-tenant reconciliation report comparing source and destination state, because "the migration ran" and "the migration is correct" are different claims. A webhook replay needs the same reconciliation step — replaying isn't done until you've confirmed the resulting state matches what should have happened the first time.

That raw event log doubles as more than a replay source. It's frequently the exact evidence someone asks for during a billing dispute or a compliance review, which is the same ground we cover in usage metering that still holds up when a customer disputes it and audit logs that answer the question an auditor will actually ask.

What support and on-call actually need

An engineer with database access can always manually replay an event. That doesn't scale past the first incident. Support needs a console view that shows event status per tenant, per event type, with a resend button that re-runs the same idempotent handler path — not a special-case script that skips the checks the real handler enforces. If the resend button doesn't go through the same idempotency and sequence logic as production traffic, it will eventually cause the exact duplicate it was built to fix.

If you're running webhook receivers on Cloudflare Workers or another edge runtime, the durability requirement doesn't go away just because the compute is stateless — you still need a durable store behind the edge function before you acknowledge. We cover where that boundary sits in edge-first architecture and the limits of a worker.

Where full rigor isn't worth it

Not every event deserves the full treatment. A webhook that triggers a marketing email or a Slack notification doesn't need sequence tracking, a dead-letter queue, and a replay console — a duplicate email is annoying, not costly. Reserve the heavier machinery for events that move money, change inventory truth, or alter access. Applying database-backed idempotency and sequence enforcement to every low-stakes notification adds latency and code you'll maintain forever for a risk that doesn't justify it. Decide per event type, not per integration.

If you're scoping this as part of a larger platform build, it's worth putting time into the event model before writing the first handler. We help teams work through exactly this kind of design decision during scoping and delivery, and it's the sort of tradeoff we walk through directly in early conversations — reach out if you're weighing it for your own platform.