Reliable webhook event handling requires durable event records, idempotent actions, and explicit rules for applying state changes. A duplicate delivery shouldn’t create another business action, and a delayed event shouldn’t overwrite newer state.
For AI agents, those protections matter because another execution can trigger another model call, tool invocation, or customer-facing write. I treat safe recovery as an application responsibility, outside the model’s reasoning loop.
Start by separating event delivery, workflow execution, and the business action each workflow authorizes.
Key Takeaways
- Store authenticated events durably before acknowledging delivery, then process them asynchronously.
- Use separate identities for incoming events and downstream business actions.
- Apply updates using authoritative versions or state rules, rather than arrival time.
- Reconcile uncertain external writes before retrying, and resume agents from durable checkpoints.
Why Duplicate and Out-of-Order Events Happen
Delivery Success Can Be Ambiguous
A receiver can save an event while its acknowledgment disappears during a network failure. The sender may retry even though the first delivery reached your system.
Stripe’s webhook endpoint guidance acknowledges duplicate event deliveries and recommends recording event IDs.
That creates two separate questions: has this event arrived before, and has its intended action already happened? Recording an event answers only the first. An agent that resumes later still needs a durable action record.
Arrival Order Doesn’t Define Business State
Concurrent requests, retries, and uneven processing times can change the order your application observes.
I wouldn’t use receipt timestamps to decide which subscription state is authoritative. Those timestamps describe your receiver, not necessarily the underlying business sequence.
The same distinction applies inside the workflow. Two workers can receive events in order, then finish in reverse order because one agent performs more retrieval or tool calls.
Make Webhook Event Handling Idempotent
Deduplicate Events With Database Constraints
Create an inbox table with a unique constraint on the provider, tenant, and stable event ID. Store the event type, affected resource, receipt time, payload reference, and processing status.
Don’t implement deduplication as an unprotected lookup followed by an insert. Two workers can both find nothing and proceed. Let the database enforce uniqueness.
A duplicate receipt should reuse the existing processing record. It shouldn’t launch another agent merely because the first execution remains unfinished.
For visual automation implementations, our n8n AI automation workflows cover the broader need for stable identifiers and controlled recovery.
Identify Each Intended Action Separately
Event deduplication doesn’t prevent different events from requesting the same business action.
Assign a durable operation_id to each authorized write. Scope it to the tenant and action type, then bind it to the exact approved payload. Reuse that identity across retries and worker restarts.
For local database changes, commit the state update and completion record in one transaction. For external writes, persist intent before dispatch and store the provider receipt afterward.
An inbox record must distinguish accepted, running, completed, failed, and unknown outcomes. Treating every existing record as completed can silently discard unfinished work.
An event ID identifies an incoming notification. An operation ID identifies the business action that must survive retries without changing identity.
Accept Events Before Running the Agent
Authenticate and Persist the Delivery
Verify the signature before allowing the event to influence workflow state. GitHub documents verification using a webhook secret and the X-Hub-Signature-256 header.
Use the provider’s required signing procedure, preserve the original request bytes where required, and enforce the destination tenant’s configuration.
Then persist the accepted event and acknowledge it promptly. Don’t keep the webhook request open while an LLM researches an issue or waits for approval.
Bridge persistence and queue publication with a transactional outbox or a worker that polls pending inbox records. Otherwise, a crash between those steps can strand an acknowledged event.
Give Workers Durable Ownership
Workers need controlled concurrency, recoverable claims, and bounded execution time. A crashed worker’s claim must eventually become recoverable without creating an unprotected second write.
Queue deduplication is useful but has defined boundaries. Amazon SQS FIFO documentation describes a five-minute deduplication interval. That interval belongs to SQS FIFO, not your application’s business ledger.
I keep action identity durable beyond individual queue attempts. A delayed replay still needs protection after a queue’s deduplication window ends.
Apply State Changes in the Right Order
Ordering belongs at the resource boundary. Serialize changes for one subscription, ticket, or account when those changes depend on each other. Unrelated resources can proceed concurrently.
If the source supplies a meaningful, monotonic resource version, apply updates conditionally. An older version mustn’t overwrite a newer one. A timestamp alone is weaker, particularly when events share timestamps or originate across different clocks.
Different event conditions need different responses:
| Event Condition | Processing Rule |
|---|---|
| Previously accepted event ID | Reuse the existing processing record. |
| Older resource version | Preserve history without replacing newer state. |
| Missing required predecessor | Wait or reconcile with the authoritative system. |
| Independent resource update | Process concurrently within capacity limits. |
The important distinction is whether the event describes current state or a historical action.
Fetching the latest source record can repair a stale subscription projection. It can’t substitute for every historical transaction when each transaction requires separate accounting.
Preserve those action events and their identities, even when a newer resource snapshot exists.
For agents, keep this logic deterministic. The model shouldn’t decide whether a stale cancellation overrides a newer reactivation. Encode allowed transitions and source-of-truth rules in application code.
This separation fits the backend controls in our AI agent architecture guide: planning can remain flexible while state transitions remain enforceable.
Recover Unknown Actions Without Replaying Writes
Reconcile Ambiguous Results
An external API can commit a write before the response reaches your worker. A post-dispatch timeout therefore means unknown outcome, not confirmed failure.
Record that state and reconcile using the original operation identity, provider receipt, or authoritative resource record. Don’t generate another key to escape the timeout.
I recommend bounded exponential backoff with jitter for retryable failures, provided the action has durable identity and idempotency protection. Malformed arguments, denied permissions, and policy failures need correction or escalation.
A retry budget should cover the logical operation across workers and agent resumes. Resetting it whenever execution restarts defeats the limit.
Resume From Confirmed Checkpoints
Persist what the agent needs to continue: workflow version, approved payload, operation state, provider receipt, and last confirmed checkpoint.
After reconciliation, give the agent the confirmed result. Avoid replaying its entire reasoning chain, which can produce another decision to act.
Chat history and vector retrieval can preserve useful context. They shouldn’t determine whether a payment, email, or ticket creation happened.
Revalidate permissions and approval validity before a paused action resumes. Restoring internal workflow state also doesn’t reverse an external payment or unsend an email. Those effects need their own recovery rules.
Test Failure Paths and Track Recovery
A successful single-worker retry test proves little about concurrent delivery. I would make these failure cases part of release testing:
- Deliver the same event concurrently and assert that only one authorized action occurs.
- Deliver updates in reverse order and confirm that newer resource state survives.
- Crash after inbox persistence but before queue publication, then confirm recovery.
- Interrupt execution after an external write but before receipt storage, then require reconciliation.
- Replay an old event after deduplication-cache expiry and confirm that durable records still protect the action.
For each test, inspect the inbox record, action ledger, provider result, and final agent-visible outcome.
Monitor duplicate receipts, stale updates, queue age, unknown-operation age, reconciliation time, and exhausted retries. Attach tenant, event, operation, and trace IDs so an alert identifies recoverable work.
Our Make.com AI automation review discusses validation and execution logging in workflow automation. Whatever platform you choose, failed events need an assigned owner and a controlled replay path.
Future Supporting Article Ideas
- Designing tenant-scoped idempotency keys for AI agent tool calls.
- Testing approval gates against stale payloads and resumed workflows.
- Reconstructing duplicate-action incidents with agent audit logs and provider receipts.
Frequently Asked Questions
Can a Queue Prevent Duplicate Agent Actions?
A queue can reduce duplicate processing within its documented behavior. End-to-end protection still requires durable action records and destination-side idempotency where available.
An external write can succeed before the worker records completion. Queue delivery controls don’t resolve that uncertain outcome by themselves.
Should Every Event for a Tenant Run Sequentially?
No. Tenant-wide serialization can create unnecessary bottlenecks. Order work at the narrowest boundary that preserves correctness, usually the affected resource or workflow.
Independent tickets or accounts can proceed concurrently under shared capacity limits.
How Long Should Deduplication Records Remain?
Retention should cover the replay period your application supports, including delayed recovery and manual reprocessing.
A short-lived cache is insufficient if older events can return. Keep durable business-action identities wherever repeating an old operation would create an unacceptable duplicate effect.
Keep Recovery Outside the Model
Reliable webhook event handling depends on durable identities, enforceable state rules, and a recovery path for uncertain outcomes. The agent should receive confirmed execution state rather than infer it from conversation history.
I would start with one standard: every accepted event must be traceable to a completed action, a pending decision, or an owned exception.
That makes duplicate delivery manageable without turning recovery into another opportunity to act twice.
