Module 06: Webhooks Security and Reliability

Why this matters

Receiving a webhook means exposing a public HTTP endpoint that performs actions based on what someone POSTs to it — mark an order paid, provision an account, credit a balance. That's a dangerous thing to leave lying around unauthenticated. The URL isn't secret (it travels through logs, browser history, the sender's dashboard), so anyone who learns it can POST a forged payment.succeeded and, if you trust the body, get free stuff. This is not theoretical: unverified webhook endpoints are a well-worn path to fraud.

The receiving side also inherits every reliability problem from the sender side, now pointed at you. The sender retries, so you will receive the same event more than once — process it naively and you ship the order twice. Senders expect a fast 2xx, so if you do the heavy processing inline and take too long, they time out and retry, amplifying load and duplicates. And you need to debug all of this against events that only originate from a third party's servers, which won't POST to localhost.

This module is the security-critical half of webhooks. You'll verify signatures with HMAC the way Stripe and GitHub actually do it, enforce a replay window, adopt the ack-fast-process-later pattern, make your receiver idempotent by deduping on event ID, and test the whole thing locally with a tunnel. Getting this right is the line between a webhook endpoint and a liability.

Concepts

Signature verification with HMAC

The core problem: how do you know a webhook POST actually came from the claimed sender and wasn't forged or tampered with, given that the URL is effectively public? The answer is a shared secret and an HMAC signature.

When a consumer registers an endpoint, the sender gives them a signing secret (a random string) — known only to the two parties. For every webhook, the sender computes HMAC-SHA256(secret, payload_bytes) and puts the resulting hex digest in a header (Stripe uses Stripe-Signature, GitHub uses X-Hub-Signature-256). You, the receiver, recompute the same HMAC over the exact raw bytes you received using your copy of the secret, and compare. If they match, the message came from someone who holds the secret (the real sender) and hasn't been altered in transit. An attacker who doesn't know the secret can't produce a valid signature for their forged body.

  SENDER (holds secret)                 RECEIVER (holds same secret)
  payload bytes ─┐                      raw bytes received ─┐
   secret ───────┼─► HMAC-SHA256 ─► sig  secret ────────────┼─► HMAC-SHA256 ─► expected
                 │        │                                 │
                 └── POST body + header ──────────────────► compare_digest(sig, expected)
                          (X-Signature)                        match  ─► trust, process
                                                               differ ─► 401, reject
  Attacker can POST the public URL but, lacking the secret, can't forge `sig`.

Two details that are easy to get catastrophically wrong:

1. Sign the raw bytes, not the parsed-and-reserialized JSON. If you await request.json() and then re-json.dumps() it to verify, any difference in key order, whitespace, or unicode escaping changes the bytes and breaks verification — or worse, tempts you to "normalize," opening a gap. Verify against await request.body() — the exact bytes.

2. Compare in constant time. Use hmac.compare_digest, not ==. A naive string comparison can leak, through timing, how many leading characters matched, enabling a byte-by-byte forgery of the signature. Constant-time comparison closes that side channel.

import hmac, hashlib

def verify(secret: bytes, raw_body: bytes, provided_sig: str) -> bool:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, provided_sig)   # constant-time

Timestamps and the replay attack

A valid signature proves authenticity, but not freshness. An attacker who captures one legitimately-signed request (from logs, a proxy, a leak) can replay it — POST the identical bytes and signature again later — and it verifies perfectly, because it's genuinely the sender's signed message. If that event credits an account, replaying it credits again.

The defense, used by Stripe: include a timestamp in the signed data and sign timestamp.payload together. The receiver checks that the timestamp is within a small tolerance of now (say, five minutes) and rejects anything older. A captured request is only replayable for that short window, and combined with idempotency (below) even a within-window replay is a no-op.

# Stripe-style: header carries t=<ts>,v1=<sig>; sender signs f"{t}.{body}"
def verify_signed_ts(secret, raw_body, header, tolerance=300):
    parts = dict(p.split("=", 1) for p in header.split(","))
    t, sig = parts["t"], parts["v1"]
    if abs(time.time() - int(t)) > tolerance:
        return False                       # too old (or clock-skewed) -> reject
    signed = f"{t}.".encode() + raw_body
    expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)

HTTPS is non-negotiable

The secret protects the payload's integrity/authenticity, but the request still travels the network. Serve your webhook endpoint over HTTPS so the payload (which often contains sensitive data) and headers aren't readable or modifiable in transit. Reject plain HTTP. This is table stakes and cheap; do it.

Acknowledge fast, process later

Senders treat a slow response as a failure and retry. So the receiver's job on the request is: verify the signature, persist the event, return 2xx — and do nothing slow. The actual work (updating your database, calling other services, sending mail) happens in a background task you enqueue. This is the same "don't do slow work in the request-response cycle" principle from module 00, now with a hard external deadline attached: if you provision the account inline and it takes 12 seconds, the sender times out at 10, marks the delivery failed, and retries — so now you're provisioning twice under load.

POST /webhook  ->  verify sig + freshness  ->  store raw event  ->  enqueue task  ->  return 200
                                                                          |
                                                          worker processes it later

The subtlety: return 2xx only once you've durably persisted the event (so you won't lose it if the worker or process dies), but before you process it. "Received and safely stored" is what a 2xx should mean to the sender — not "fully processed."

Idempotent receivers: dedupe by event ID

Because senders retry (and replays can slip inside the freshness window), you will receive the same event ID more than once. An idempotent receiver processes each event ID's effect exactly once. The mechanism: before processing, atomically record the event ID (a unique constraint / SET evt:{id} NX), and if it's already present, acknowledge with 2xx and skip — you've already handled it.

def already_processed(event_id) -> bool:
    # returns True if we've seen this id before (atomic check-and-set)
    return not redis.set(f"evt:{event_id}", "1", nx=True, ex=86400)

Note the interplay: you still return 2xx for a duplicate (so the sender stops retrying), you just don't re-do the work. Silently 500-ing on a duplicate would make the sender retry forever.

Retry logic on the sender side (and what the receiver owes it)

You met sender-side retries in module 05; as a receiver you must cooperate with them:

• Return 2xx for success and for already-processed duplicates.
• Return 5xx (or non-2xx) only when you genuinely couldn't accept the event and want a retry — e.g. your datastore is briefly down. The sender will back off and retry, which is what you want.
• Return 4xx for a bad/forged/unverifiable request you never want retried (bad signature → 401/400). A 4xx tells a well-behaved sender "don't bother retrying; this will never work."

Choosing the right status code is how the receiver steers the sender's retry behavior.

Logging and testing webhooks locally

Webhooks originate from a third party's servers, which can't reach localhost. To develop and debug, use a tunnel — ngrok, cloudflared, or similar — that gives you a public HTTPS URL forwarding to your local port. Register that URL with the sender, and real webhooks flow to your laptop. Log every received event (id, type, signature-valid, raw body) so you can diagnose failures and replay. Most webhook providers also offer a dashboard to inspect and manually resend deliveries, and a CLI (e.g. stripe listen) that forwards events to your local endpoint without a tunnel.

Command reference

A secure, reliable receiver in FastAPI — receiver.py:

import time, hmac, hashlib, os
from fastapi import FastAPI, Request, HTTPException
import redis

app = FastAPI()
SECRET = os.environ["WEBHOOK_SECRET"].encode()
r = redis.Redis()

def verify(raw_body: bytes, header: str, tolerance: int = 300) -> bool:
    try:
        parts = dict(p.split("=", 1) for p in header.split(","))
        t, sig = parts["t"], parts["v1"]
    except (ValueError, KeyError):
        return False
    if abs(time.time() - int(t)) > tolerance:      # freshness -> anti-replay
        return False
    signed = f"{t}.".encode() + raw_body           # sign timestamp + raw body
    expected = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)      # constant-time

@app.post("/webhook")
async def webhook(request: Request):
    raw = await request.body()                     # EXACT bytes, not reparsed
    sig_header = request.headers.get("X-Signature", "")
    if not verify(raw, sig_header):
        # forged/tampered/stale: 401, and 4xx tells the sender not to retry
        raise HTTPException(status_code=401, detail="invalid signature")

    event = json.loads(raw)
    event_id = event["id"]

    # Idempotency: record the id atomically; if seen, ack and skip.
    is_new = r.set(f"evt:{event_id}", "1", nx=True, ex=86400)
    if not is_new:
        return {"status": "duplicate ignored"}     # 2xx so sender stops retrying

    persist_raw_event(event_id, raw)               # durable BEFORE we ack
    process_event.delay(event_id)                  # slow work happens on a worker
    return {"status": "accepted"}                  # fast 2xx: received + stored

# --- the worker side (Celery) ---
@celery.task(autoretry_for=(TransientError,), max_retries=5, retry_backoff=True)
def process_event(event_id):
    event = load_raw_event(event_id)
    # the real work: update DB, call services, etc. Idempotent per event_id.
    handle(event)

Hands-on exercises

Continue in bg-queues. You'll run the receiver and a small "sender" script that signs requests. pip install redis requests. Set WEBHOOK_SECRET in both.

1. Verify a good signature, reject a bad one

Write a sender that signs f"{t}.{body}" with the shared secret and POSTs to /webhook with an X-Signature: t=...,v1=... header. Send one correctly signed and one with a tampered body (change a byte after signing). Expected: the valid one returns 202/accepted; the tampered one returns 401. You've proven authenticity + integrity.

2. Forge without the secret and fail

From an "attacker" script that doesn't know the secret, POST a payment.succeeded body with a made-up signature. Expected: 401. The public URL is not enough to forge — the secret is what gates action. Confirm no processing task was enqueued.

3. Raw-bytes vs. reparsed-JSON gotcha

Change the receiver to verify against json.dumps(await request.json()) instead of the raw body. Send a payload whose keys, if reserialized, come out in a different order or with different spacing. Expected: verification now fails for a legitimate request, because the bytes differ. Revert to await request.body(). Lesson: always sign/verify the exact received bytes.

4. Constant-time comparison

Replace hmac.compare_digest with ==. Everything still "works" functionally. Explain in writing why this is nonetheless a vulnerability (a timing side channel that leaks how many leading bytes matched, enabling incremental signature forgery). Restore compare_digest. Some bugs don't show up in a functional test — this is one.

5. Replay attack: diagnose and fix

Capture one valid, signed request (save its exact bytes + header). Replay it (POST the identical bytes + header) 10 minutes later. First, with no timestamp check: expected — it verifies and processes again (a replayed credit!). Now enable the freshness check (tolerance=300). Expected: the replay is rejected with 401 because its timestamp is stale. This is a "diagnose and fix" scenario: the symptom is "an event got processed twice from one real send"; the fix is signing+checking a timestamp.

6. Idempotent receiver survives duplicates

Keep the timestamp check but send the same event id twice within the window (a legitimate sender retry). Expected: the first is accepted and enqueues a task; the second returns duplicate ignored with 2xx and enqueues nothing. Confirm the effect happened exactly once. Note it must be 2xx, not 500, or the sender would keep retrying.

7. Ack fast, process later — and prove it

Make process_event sleep 8 seconds. Time the /webhook response. Expected: the endpoint returns in milliseconds (it only verified, stored, enqueued) while the 8-second work happens on the worker. Then move the work inline into the handler and re-time: now the response takes 8s and a sender with a 5s timeout would time out and retry. Restore the async version.

8. Steer sender retries with status codes

Simulate three cases and choose the right status: (a) valid, processable event → 2xx; (b) valid event but your datastore is momentarily down → 5xx (you want a retry); (c) invalid signature → 4xx (never retry). Have your sender script implement backoff-on-non-2xx and observe: it retries the 5xx, stops on the 4xx, and stops on the 2xx. Lesson: the status code is your control signal to the sender.

9. Test against a real tunnel

Start ngrok http 8000 (or cloudflared tunnel --url http://localhost:8000) to get a public HTTPS URL. Point your sender script (or a real provider's test webhook / stripe listen) at https://<tunnel>/webhook. Expected: real webhooks originating off your machine reach your local receiver over HTTPS. Confirm your logs show each event's id, type, and signature-valid flag. Lesson: this is how you develop against senders that can't see localhost.

Independent challenge

No code given. Build a hardened receiver for incoming payment.succeeded webhooks that credits a user's balance. Requirements: reject any request whose HMAC-SHA256 signature over the raw bytes doesn't verify (constant-time compare); reject stale requests outside a 5-minute freshness window (replay defense); serve only over HTTPS; acknowledge with a fast 2xx after durably storing the event but before crediting; credit the balance in a background task that is idempotent per event id so retries and within-window replays credit exactly once; and return status codes that make a well-behaved sender retry a transient storage outage but never retry a forged request. Prove correctness by (a) forging a request and getting 401, (b) replaying a valid request and confirming exactly one credit, and (c) sending the same event id five times and confirming one credit.

This combines the idempotency discipline from 02-retries-prioritization-and-rate-limiting-in-queues with the ack-fast pattern that traces back to 00-task-queues-fundamentals.

Common mistakes & troubleshooting

• Trusting the payload without verifying a signature. The URL is public; anyone can POST. Verify HMAC over the raw bytes with the shared secret before acting on anything.
• Verifying reparsed JSON instead of raw bytes. Reserializing changes the bytes and breaks (or weakens) verification. Verify await request.body() exactly as received.
• Comparing signatures with ==. Leaks match progress via timing. Use hmac.compare_digest for constant-time comparison.
• No freshness/timestamp check. A captured valid request can be replayed and re-processed. Sign and check a timestamp; reject stale requests.
• Processing inline and blowing the sender's timeout. Slow inline handling causes the sender to time out and retry, amplifying load and duplicates. Verify, store, enqueue, return 2xx; process on a worker.
• Non-idempotent processing. Sender retries and replays mean you'll see an event id more than once; without a dedupe guard you double-process. Record the id atomically and skip if seen.
• Returning 5xx for duplicates or forged requests. A 5xx makes the sender retry forever. Return 2xx for duplicates (already handled) and 4xx for forged/unverifiable requests (never retry).
• Testing only against localhost. Real senders can't reach your machine directly. Use a tunnel (ngrok/cloudflared) or the provider's CLI to receive real events locally over HTTPS.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Why isn't a hard-to-guess webhook URL enough security, and what actually authenticates a webhook?
2. Why must you compute the HMAC over the raw received bytes rather than over reparsed-and-reserialized JSON?
3. What is a replay attack against a webhook receiver, and what two mechanisms together defeat it?
4. Why use hmac.compare_digest instead of ==, given both return the right boolean?
5. Describe the ack-fast-process-later pattern and the one thing that must be true before you return 2xx.
6. A duplicate event arrives. What status code do you return and why — and what happens if you return 500 instead?
7. How do 2xx, 4xx, and 5xx responses each steer a well-behaved sender's retry behavior?

Further reading & sources

• Stripe: Verify webhook signatures - the timestamp+HMAC scheme and replay-window check this module models.
• GitHub: Validating webhook deliveries - X-Hub-Signature-256 HMAC verification with constant-time comparison.
• Python: hmac module - hmac.new, hexdigest, and compare_digest for constant-time comparison.
• OWASP: Webhook / SSRF and secure design cheat sheets - hardening a public endpoint that acts on untrusted input.
• ngrok documentation - exposing a local receiver over HTTPS to test against real senders.

Free PDF notes

Download these notes as a printable PDF, with every diagram and interview questions: SD-C-020: Webhooks. Each section in one file: Phase 2, APIs (PDF).

Next

07-websockets-and-server-sent-events — webhooks push events server-to-server. Next you'll push events server-to- browser in real time: when to use WebSockets vs. Server-Sent Events vs. polling, a real FastAPI WebSocket endpoint, the connection leak a careless implementation causes, and broadcasting to many connected clients. This module also carries the track's second cumulative review.