Webhooks that drop: delivery, duplicates and ordering

Webhook delivery is at-least-once and unordered. You will receive the same event twice, you will receive events out of sequence, and you will miss some entirely when your endpoint is briefly down. Design for all three: acknowledge fast and process asynchronously, deduplicate on the sender’s event id, and use timestamps rather than arrival order to decide what is current.

Most webhook endpoints are written assuming exactly-once, in-order delivery. Neither property is offered by anyone.

Key takeaways

Acknowledge within a second or two, then process. Slow endpoints cause the retries that cause duplicates.
Deduplicate on the sender’s event id, stored durably, not on payload contents.
Arrival order is not event order. Use the event timestamp to decide what is newer.
A brief outage means permanently lost events unless you can reconcile from the source.
Verify signatures. An unauthenticated webhook endpoint is an open write path into your systems.

Who this applies to

You are receiving webhooks - from a payment provider, a CRM, a helpdesk, a store platform - into n8n, Make, or your own service, and correctness matters.

What senders actually guarantee

Read the fine print of any major webhook provider and the pattern is consistent:

At-least-once delivery. They retry on failure, so you may receive duplicates.
No ordering guarantee. Events dispatched in one order can arrive in another, particularly when retries are involved.
A retry window, then abandonment. Typically some hours of exponential backoff, after which the event is gone.
A short timeout on your endpoint. Frequently a few seconds. Exceed it and they treat the delivery as failed and retry.

That last point causes the most self-inflicted damage. A workflow that acknowledges the webhook only after doing thirty seconds of processing has already caused a retry, and now the same event is being processed twice concurrently.

Defence one: acknowledge fast, process later

The single most important change.

Your endpoint should validate the signature, write the event to durable storage, return 200, and stop. Processing happens afterwards, from that store.

In n8n that means the webhook-triggered workflow does as little as possible - persist and respond - and a second workflow processes the queue. Where your platform supports separate webhook processes, that separation is exactly this pattern at the infrastructure level; see running n8n in production.

This decouples the sender’s timeout from your processing time, which turns a whole class of duplicate-and-timeout problems into a non-issue.

Defence two: deduplicate on the event id

Every serious webhook provider includes a unique event identifier in the payload or a header. Store it, check it, skip repeats.

Three things to get right.

Store the id durably, in a table with a unique constraint, not in memory. A restart must not forget what you have seen.

Let the database enforce it. Insert the id and let a unique-constraint violation tell you it is a duplicate. Checking first and then inserting is a race, and duplicate deliveries frequently arrive concurrently, which is exactly when the race loses.

Keep them long enough. Retry windows run to hours or a day. A dedup table that clears after ten minutes does not cover the case it exists for. Retain for at least the sender’s full retry window, and prune on a schedule.

Do not deduplicate on payload contents. Two genuinely different events can carry identical payloads - the same customer updating the same field twice - and hashing the body will silently drop the second.

Defence three: order by timestamp, not arrival

If two updates to the same record arrive out of order, processing them in arrival order writes the older value last.

The fix is to carry the event’s own timestamp or version and refuse to apply anything older than what you have already applied. Store the last-applied timestamp per record and compare before writing.

This makes out-of-order delivery harmless rather than something you have to prevent, which is the only workable stance since you cannot prevent it.

For events that are not idempotent in this way - a sequence that must be applied in order - buffer briefly and sort, or reconstruct from the source rather than from the event stream.

Sent evt 1evt 2 evt 3evt 4

Received evt 1evt 3 evt 2evt 2 EVT 4: NEVER ARRIVES

OUT OF ORDERDUPLICATE Dedupe on event idOrder by timestamp CATCHES THE DUPLICATECATCHES THE REORDER Scheduled reconciliation THE ONLY THING THAT FINDS EVT 4

Two of the three failures are handled inside the handler. The third is invisible from there - nothing in your logs records an event that was never delivered, which is why an event-driven system still needs a scheduled pass over the source.

The events you never receive

Deduplication and ordering handle what arrives. The harder problem is what does not.

If your endpoint is down for twenty minutes, the sender retries for its window and then gives up. Those events are gone, and nothing in your system knows they existed. This is the webhook equivalent of silent failure - see why automations fail silently.

Two defences.

Reconcile periodically. Run a scheduled job that queries the source system for records changed since your last successful sync and processes anything you do not have. Daily is enough for most cases. This is the only mechanism that recovers genuinely lost events, and it is the one most systems lack.

Monitor the gap. Alert if no webhook has arrived from a given source in longer than normal. Many providers also expose delivery logs or a failed-delivery view; check it, because it tells you about failures your side never saw.

The reconciliation job is unglamorous and it is what turns “we think we have everything” into something you can state.

Verify the signature

Most providers sign their webhooks with a shared secret. Verify it.

An endpoint that accepts unauthenticated POSTs is a write path into your systems that anyone who learns the URL can use. The URL will leak - into logs, into screenshots, into a support ticket.

Verify the signature before doing anything else, and reject with a 401 rather than a 200. Also check the timestamp in the signature, where the provider includes one, to reject replayed old requests.

What to log

For every delivery: the event id, the type, the arrival time, the sender’s own timestamp, whether it was a duplicate, and the processing outcome.

That set answers the questions you will actually have. “Did we receive event X” is the most common question during an incident, and without an id-level log the answer is inference.

What we build by default

Acknowledge-and-persist endpoints, a dedup table with a unique constraint retained past the sender’s retry window, timestamp-based ordering on record updates, signature verification, and a scheduled reconciliation job against every webhook source.

The reconciliation job is the one clients most often see as unnecessary, because webhooks appear to be working and the job finds nothing on most days. The argument for it is what it costs to not have: when a deployment or an outage causes twenty minutes of rejected deliveries, the difference between a system with reconciliation and one without is the difference between self-healing overnight and discovering a gap in the data three weeks later with no way to reconstruct it.

The pattern we insist on that occasionally reads as pedantic is letting the database enforce deduplication rather than checking first. Check-then-insert works in testing, because duplicates in testing arrive sequentially. In production they arrive concurrently, which is precisely when the check passes twice.

Where we would relax our defaults: for a low-stakes internal webhook where duplicate processing is harmless and idempotent, the dedup table is arguably more machinery than the problem needs. The signature verification is not negotiable regardless of stakes.

When this is more than you need

Internal webhooks between your own systems, where you control both sides and can guarantee delivery semantics yourself.

Genuinely idempotent processing where re-applying an event has no effect. Deduplication is then an optimisation rather than a correctness requirement - though it still saves work.

Very low volume with a human in the loop. If someone checks the output daily, they are the reconciliation.

Frequently asked questions

How fast must we acknowledge?

Under the sender’s timeout, which is often a few seconds. Aim for well under a second by doing nothing but validating and persisting.

How long should we retain event ids?

At least the sender’s full retry window, which is commonly several hours to a day. Retaining longer is cheap and makes incident investigation easier.

What if the provider does not send an event id?

Construct one from stable fields - resource id plus event type plus the source timestamp - and document that you are doing so. It is weaker than a real id and better than nothing.

Do we need reconciliation if delivery seems reliable?

Yes. Delivery being reliable most of the time is exactly the condition under which a rare gap goes unnoticed.

Should the webhook workflow do the processing?

No. Acknowledge and persist, then process from the store. That separation removes the timeout-induced duplicate problem entirely.

Next step

If your webhook endpoints process synchronously and there is no reconciliation job, those are the two changes that most improve reliability. The business process automation engagement builds intake with deduplication, ordering and reconciliation as standard.