A secure webhook receiver needs four separate gates: preserve the exact request bytes and verify the sender's message authentication code before parsing; enforce a signed timestamp window and deduplicate a delivery or event identifier; make the business effect idempotent; and only then hand work to a durable processor. A matching HMAC alone does not prove freshness, exactly-once execution, authorization, or valid business meaning.

What does webhook verification actually prove?

A webhook is an external system calling a public endpoint. The receiver must answer four different questions:

  1. Integrity and possession: Was the message altered, and did the sender possess the agreed secret?
  2. Freshness: Is this an old valid request being replayed?
  3. Idempotency: Will a provider retry or attacker replay repeat the business effect?
  4. Authorization and semantics: May this signed event perform this action on this resource?

HMAC primarily addresses the first question. RFC 2104 defines keyed message authentication based on a cryptographic hash, but HMAC does not automatically include time, event identity, or business permissions. A perfectly valid signed request can be captured and sent again unless the protocol and receiver add replay controls.

Treat signature success as one state, not as “processed.” A useful lifecycle is RECEIVED, AUTHENTICATED, FRESH, DEDUPED, ACCEPTED, and APPLIED. Each transition has a distinct failure reason, retry policy, and metric.

Why must the receiver verify the raw body?

Providers usually sign the bytes they transmitted. If a framework parses JSON and then serializes it again, whitespace, key order, Unicode escaping, numbers, or line endings can change. The reconstructed text is semantically equivalent JSON but a different byte sequence, so the MAC differs. Middleware can also decompress, transcode, or consume the stream before application code sees it.

The GitHub webhook validation guide instructs receivers to calculate an HMAC-SHA-256 with the webhook secret and compare it with X-Hub-Signature-256. The safe ordering is:

receive bounded raw bytes
-> parse signature metadata
-> compute MAC over the exact required signing input
-> constant-time compare
-> check time and delivery identity
-> parse JSON
-> validate schema and authorization
-> enqueue or apply idempotently

This FastAPI-style fragment is an engineering illustration, not a complete implementation of any provider's header grammar:

import hashlib
import hmac

async def verify_webhook(request, secret: bytes):
    raw = await request.body()
    if len(raw) > MAX_WEBHOOK_BYTES:
        raise PayloadTooLarge()

    supplied = parse_signature(request.headers["X-Signature"])
    expected = hmac.new(secret, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(supplied, expected):
        raise InvalidSignature()
    return raw

Python's hmac documentation recommends compare_digest() for verification instead of ordinary equality to reduce exposure to timing attacks. Before comparing, strictly validate the algorithm, encoding, prefix, and length. Never let an untrusted request select none, a weak digest, or an arbitrary algorithm implementation.

The receiver also needs a body-size limit and read timeout. Otherwise, an attacker can consume memory or workers with an oversized or slow body before cryptographic verification starts. Resource boundaries are the first authentication precondition.

How does a signed timestamp limit replay?

When the provider includes a timestamp in the signing input, the receiver can verify both integrity and age. A common conceptual form is:

signed_payload = timestamp + "." + raw_body

Parse the timestamp, reject malformed or unreasonable values, recompute the signature over the timestamp and exact body, compare it in constant time, and then record the event identity. Merely checking an unsigned timestamp is useless because an attacker can replace it while replaying the old signed body.

There is no universal best tolerance. A window that is too short rejects legitimate queue delays, cross-region latency, or clock skew. A window that is too long expands the attacker's opportunity. Set it from the provider's documented delivery behavior and your NTP health, and alert on local clock problems separately from expired requests.

A time window still permits replay inside that window. Persist a stable provider delivery ID or event ID under a uniqueness constraint. If the provider offers no reliable ID, a receiver may derive a fingerprint from verified fields and the signature, but then it must define canonicalization, collision scope, retention, and whether the same semantic event can be legitimately resent with different metadata.

How is delivery deduplication different from business idempotency?

Delivery deduplication answers “have I seen this envelope?” Business idempotency answers “has the intended action already completed?” They are related but cannot safely collapse into one boolean.

State Meaning Retry behavior
RECEIVED Verified envelope is durably stored A worker may continue
PROCESSING One worker holds a lease Another may take over after expiry
APPLIED Business effect is confirmed A duplicate returns success without acting
FAILED_RETRYABLE A transient dependency failed Back off with the same operation ID
FAILED_FINAL Permanent schema, permission, or business error Alert or review; do not loop forever

Do not mark a delivery APPLIED before its effect. A crash then creates a permanent skip. Do not wait until after processing to insert the delivery for the first time; concurrent copies can both act. A common design inserts an inbox row under a unique key, gives one worker a lease through compare-and-set, and still passes a separate business idempotency key to the actual effect.

A payment delivery ID is not necessarily an order version. A provider can redeliver the same semantic event under new delivery metadata, and several event types can refer to the same order. The business update should validate provider event ID, object identity, event type, and a monotonic domain version or state transition.

This resembles the revision boundaries in the atomic FastAPI, MongoDB, and Redis publishing guide: request accepted, Mongo state committed, cache invalidated, and page visible are different facts. For a webhook, envelope deduplicated, domain committed, downstream accepted, and reconciliation complete are also different facts.

Should the endpoint return 2xx before processing finishes?

For slow work or external dependencies, synchronously enforce body limits, signature, freshness, durable deduplication, and basic schema checks. Return 2xx promptly only after the durable inbox commit succeeds. A worker then performs effects asynchronously.

Returning success before persistence can lose the event if the process exits. Waiting for every business effect can exceed the provider's timeout and trigger redelivery, increasing concurrency and duplicates. Use each provider's documented timeout and retry behavior rather than assuming all webhook systems behave alike.

The FastAPI background-task selection guide helps place this boundary. A critical webhook should not depend only on in-process BackgroundTasks: process termination loses the retry, and multiple application instances lack a shared lease. Persist an inbox or outbox first, then let a durable queue or workflow consume it.

A practical response policy is:

  • invalid signature, expired request, or oversized body: return 4xx and never enqueue;
  • known delivery already APPLIED: return a provider-compatible 2xx;
  • known delivery currently processing: usually return 2xx to avoid a retry storm;
  • unavailable durable inbox: return 5xx so the provider can retry;
  • permanent domain error after asynchronous acceptance: move to dead letter and human recovery.

The exact status codes must follow provider guidance. Do not expose internal exception details, expected signatures, secrets, or raw sensitive payloads in an error body.

How can secrets rotate without an outage?

Use a short dual-secret window. First deploy the receiver with the new and current secrets. Then switch the provider. Observe until old-key traffic reaches zero, and finally remove the old key. Each key can have an internal key ID, but logs must never contain the secret, complete signature, or sensitive raw body.

Limit how many keys can verify. Do not scan every historical secret after a failure; that expands both CPU cost and the compromise window. If the provider includes a version or key ID, validate it against a strict allowlist.

The Stripe signature troubleshooting guide emphasizes three inputs: the endpoint secret, the raw request body, and the actual signature header. Secrets from Dashboard endpoints, local CLI forwarding, tests, and production are not interchangeable. Apply that separation broadly: scope a secret by environment, endpoint, provider, and purpose rather than using one global webhook key.

Load secrets from a controlled secret manager, keep them in process memory only as needed, and audit reads and rotations. Safe operational logs include provider, endpoint ID, algorithm version, key ID, a non-reversible delivery ID digest, validation outcome, and failure class.

How should multiple signatures and algorithm upgrades work?

A provider may send several signatures during rotation. The parser must preserve repeated or comma-separated fields according to the official grammar; a generic header map must not silently overwrite one value. The acceptance rule should be “at least one allowed-algorithm signature matches a currently trusted key,” not “some value looks like a hash.”

Version algorithms in configuration. To upgrade, deploy receiver support for the new version while still accepting the old version, switch the sender, observe, and then remove the old algorithm. Never dynamically import an algorithm named by an untrusted header.

For a general protocol that signs the method, authority, path, and selected fields across intermediaries, review RFC 9421 HTTP Message Signatures. Do not invent an incompatible HTTP signature scheme for a provider that specifies a body HMAC; implement its official protocol exactly.

What can proxies and frameworks break?

  • Request decompression: the protocol may sign compressed or decompressed bytes; follow the specification.
  • Character conversion: decoding bytes to text and encoding again changes the signed input.
  • Header merging: repeated signatures can be overwritten or reordered.
  • Path rewriting: if the target URI is signed, an internal route is not the original external target.
  • JSON middleware: automatic parsing can discard the raw stream.
  • Exception logging: a default debug page may expose headers or payloads.
  • Proxy timeout: the edge may close or retry while the backend has not persisted the inbox.

Before release, send fixed test vectors through the real public CDN, WAF, reverse proxy, and application path. Direct handler unit tests are necessary but insufficient. Include a valid signature, a one-byte body change, wrong key, old timestamp, duplicate delivery, multiple signatures, oversized body, and slow upload.

Keep a provider-specific conformance fixture under version control without real secrets. It should include the exact bytes, safe test key, signature metadata, and expected accept or reject result. This catches accidental middleware changes during framework upgrades.

What should observability record?

For each request, emit structured and sanitized fields: provider, endpoint, irreversible digest of the delivery ID, body size, signature version, key ID, verification result, rejection class, inbox revision, processing state, and duration. Do not treat a signature header as ordinary debug data; retaining it can aid analysis of the verifier and increases secret-adjacent exposure.

Track signature failure rate, expiry rate, duplicate rate, inbox commit failure, processing lag, retry depth, dead-letter count, and application outcome by event type. A sudden signature-failure increase can mean a rotation mistake or an attack. A duplicate spike often means downstream work is slow or the receiver is missing its response deadline.

Use the FastAPI OpenTelemetry production guide to connect inbound delivery, durable inbox, worker, and downstream calls. Treat external trace headers as untrusted input: they must not override trusted sampling, baggage, tenant identity, or authorization state.

Alerts should lead to a bounded action. Authentication failures trigger key and provider checks, not payload replay. Inbox failures protect availability and ask the provider to retry. Processing failures operate on the persisted event with the same operation ID. Dead-letter recovery requires an operator-visible audit trail.

Production verification checklist

  • [ ] The receiver reads exact raw bytes before parsing or transformation.
  • [ ] Body size, header size, connection time, and read time are bounded.
  • [ ] Algorithm, signature format, encoding, and key IDs use strict allowlists.
  • [ ] Comparison uses compare_digest or an equivalent constant-time primitive.
  • [ ] Any timestamp is included in the signed input and checked against a justified window.
  • [ ] Delivery IDs are persisted under a unique constraint with a lease or CAS.
  • [ ] Business effects use a separate idempotency key or version transition.
  • [ ] The endpoint returns 2xx only after a durable inbox accepts the event.
  • [ ] Retry, dead-letter, manual recovery, and reconciliation paths have been exercised.
  • [ ] Secret rotation has a bounded overlap and isolates environments and endpoints.
  • [ ] Test vectors pass through the real external proxy chain.
  • [ ] Logs exclude secrets, complete signatures, and sensitive raw bodies.

Common failure modes

  • Verifying reserialized JSON instead of the original bytes.
  • Comparing signatures with ordinary equality.
  • Checking a timestamp that was not covered by the signature.
  • Using a time window without delivery deduplication.
  • Deduplicating the envelope while leaving the business action non-idempotent.
  • Returning 2xx before durable persistence.
  • Using an in-process task for a critical event with no retry store.
  • Reusing the same secret across development, tests, staging, and production.
  • Logging full headers and payloads during an incident.
  • Trusting an inbound trace or tenant header as authorization context.

FAQ

Why sign a webhook when HTTPS already encrypts it?

TLS protects the transport and authenticates the server the sender connects to. The receiving application still needs to verify the provider's webhook authentication protocol. Anyone who can reach the public endpoint can send an HTTPS request; a signature provides message-level proof of secret possession and integrity.

Can the handler execute immediately after a valid signature?

No. It must still verify freshness, delivery identity, schema, target resource, domain state, and authorization. A signature proves consistency with a secret; it does not authorize every possible business operation.

Is Redis enough for delivery deduplication?

Redis can be a fast first filter, but an irreversible or financial effect needs a durable record and domain idempotency. Cache eviction, expiry, failover, or data loss must not make a payment, email, or deletion execute again. Use caching as an optimization, not the only source of truth.

What if the worker fails permanently after the endpoint returned 2xx?

That is why durable inboxes, bounded retry, dead-letter state, and operator recovery exist. HTTP acceptance means the receiver reliably took responsibility; it does not mean the business effect completed. Continue tracking the event to APPLIED or an explicit final failure.

Webhook security is not finished when a route gains an HMAC comparison. Exact bytes, authentication, freshness, deduplication, domain idempotency, durable acceptance, and recoverable processing form one boundary. Collapsing any of them can turn a legitimate retry into a duplicate effect—or turn a valid signature into a replayable pass.