Module 12: Payment Processing in Practice

Why this matters

Module 06 taught webhook receiving generically: verify an HMAC signature, reject stale timestamps, dedupe by event id, ack fast and process later. Payment processing is where every one of those disciplines stops being theoretical hardening and becomes the only thing standing between your system and a real customer's real money moving incorrectly. A missed signature check means anyone can forge a charge.succeeded event and grant themselves a paid feature for free. A missing idempotency key on your outbound charge call means a network timeout — not a bug, just an ordinary retry — can charge a card twice. This module takes module 06's generic receiver discipline and applies it end to end to a real payment provider's flow (Stripe's actual webhook signature scheme, verified against real HMAC math, not a hypothetical), plus the one thing module 06 didn't cover: making the outbound charge-creation call itself safe to retry.

Concepts

Two directions, two different reliability problems

A payment integration has two independent flows, and conflating them is the most common design mistake:

• Outbound: your backend calls the payment provider to create a charge or payment intent. You are the caller here, and you control retries — this is where an idempotency key on the request itself is what prevents a retried network call from charging twice.
• Inbound: the provider calls you back via webhooks — charge.succeeded, charge.failed, charge.refunded — because payment confirmation is often asynchronous (bank transfers, 3-D Secure challenges, fraud review) and can't always be returned synchronously from the initial API call. This is exactly module 06's territory: verify the signature, dedupe by event id, ack fast.

  OUTBOUND (you retry)                  INBOUND (provider retries)
  your backend ──create charge──►       provider ──webhook──► your backend
       │         (idempotency key           │
       │          prevents double-charge     │ (module 06: HMAC verify,
       │          on YOUR retry)              │  dedupe by event id,
       └── you own this side                  │  ack fast)
                                               └── provider owns retry here

Idempotency keys on the outbound call: preventing your own retry from double-charging

A network call to create a charge can fail in a way that's genuinely ambiguous: did the request never reach the provider, or did it reach them and the charge succeeded but the response was lost? Blindly retrying "to be safe" risks a real double charge if the first attempt actually succeeded. The fix (module 01's idempotency-key pattern, applied to the highest-stakes case in this whole track) is to generate a stable key once per logical charge attempt and send it with every retry of that same attempt:

import httpx

def create_charge(amount_cents: int, idempotency_key: str):
    return httpx.post(
        "https://api.example-payments.com/v1/charges",
        json={"amount": amount_cents, "currency": "usd"},
        headers={"Idempotency-Key": idempotency_key},
    )

The provider stores the first response returned for a given idempotency key and replays that exact response for any repeat request with the same key — a retried call after a timeout gets back the original charge's result, not a second charge. The key must be generated once and reused across retries of that specific attempt (e.g. derived from your own order id), never regenerated per HTTP call — regenerating it defeats the entire mechanism.

Verifying a real payment provider's webhook signature

Module 06 taught HMAC signature verification generically. Real providers publish an exact scheme; Stripe's (representative of the pattern most providers use) is: the signed content is "{timestamp}.{raw_request_body}", HMAC-SHA256'd with your webhook signing secret, sent in a header shaped like t=<timestamp>,v1=<hex signature>:

import hmac, hashlib, time

def verify_webhook_signature(payload: str, sig_header: str, secret: str, tolerance_seconds: int = 300) -> bool:
    parts = dict(p.split("=", 1) for p in sig_header.split(","))
    timestamp = int(parts["t"])
    if abs(time.time() - timestamp) > tolerance_seconds:
        return False  # too old -- possible replay (module 06)
    signed_payload = f"{timestamp}.{payload}"
    computed_sig = hmac.new(secret.encode(), signed_payload.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(computed_sig, parts["v1"])

This is module 06's HMAC-and-timestamp discipline, verified here against the exact algorithm a real provider uses — the timestamp is folded into the signed content (not just carried alongside it), so tampering with the timestamp to bypass the replay-tolerance check also invalidates the signature. There's no way to slide the timestamp forward without recomputing a signature you don't have the secret to produce.

Out-of-order and duplicate webhook delivery

Providers explicitly do not guarantee webhook delivery order — a charge.failed event can arrive after a later charge.succeeded for the same charge if their infrastructure retries or reorders internally, and any event can be delivered more than once (module 06's dedupe-by-id lesson again, but now with a new wrinkle: dedup alone isn't enough if two different events for the same charge can race). The correct design treats the provider's charge object as the source of truth, not "whichever event I processed most recently": on receiving any event about a charge, use the event's own timestamp/data to decide whether it's newer than what you've already recorded, and only update your local state if it is.

# WRONG: blindly applies whatever event arrives, in arrival order
def handle_event_naive(event):
    charge_id = event["data"]["object"]["id"]
    db.update_charge_status(charge_id, event["type"])  # last-write-wins on ARRIVAL order

# RIGHT: compare against the event's own ordering info before applying
def handle_event_safe(event):
    charge_id = event["data"]["object"]["id"]
    event_created = event["created"]  # provider's own event timestamp
    current = db.get_charge(charge_id)
    if current and current.last_event_created >= event_created:
        return  # a newer (or same) event already recorded -- ignore this one
    db.update_charge_status(charge_id, event["type"], last_event_created=event_created)

Ledger correctness: append, never overwrite

Because money is involved, a payment system should keep an append-only ledger of every charge/refund/dispute event received — not just a mutable "current balance" column — so any state can be reconstructed and audited later (module 10's event-sourcing idea, applied specifically to money, where "what actually happened, in order" matters more than almost anywhere else). The "current status" your application reads is a derived view over that ledger, not the only copy of the truth.

Never trust a client-reported amount

The charge amount your backend tells the payment provider to charge must come from your own server-side price calculation, never from a value the client sends you — a client-controlled amount is a direct "let the user set their own price" vulnerability. Verify the cart/order total server-side immediately before creating the charge, using the same authoritative pricing logic the rest of your system trusts.

Command reference

This module uses a local mock payment API and webhook sender that reproduce a real payment provider's exact idempotency-key behavior and webhook signature scheme, so the exercises below exercise real HTTP behavior rather than a hypothetical.

Hands-on exercises

pip install fastapi uvicorn httpx.

1. Build the mock payment API with idempotency-key support

# mock_payment_api.py
from fastapi import FastAPI, Request, Header
import uuid

app = FastAPI()
idempotency_store = {}
charge_count = {"n": 0}

@app.post("/v1/charges")
async def create_charge(request: Request, idempotency_key: str = Header(None, alias="Idempotency-Key")):
    body = await request.json()
    if idempotency_key and idempotency_key in idempotency_store:
        return idempotency_store[idempotency_key]
    charge_count["n"] += 1
    result = {"id": f"ch_{uuid.uuid4().hex[:8]}", "amount": body["amount"], "status": "succeeded"}
    if idempotency_key:
        idempotency_store[idempotency_key] = result
    return result

@app.get("/debug/charge_count")
async def debug_count():
    return charge_count

python3 -m uvicorn mock_payment_api:app --port 8501

2. Prove a retried request with the same idempotency key doesn't double-charge

curl -s -X POST http://localhost:8501/v1/charges -H "Idempotency-Key: idem-abc" -H "Content-Type: application/json" -d '{"amount": 5000}'
curl -s -X POST http://localhost:8501/v1/charges -H "Idempotency-Key: idem-abc" -H "Content-Type: application/json" -d '{"amount": 5000}'
curl -s http://localhost:8501/debug/charge_count

Expected: both charge calls return the same charge id, and debug/charge_count shows {"n": 1} — exactly one real charge was created despite two HTTP calls, because the second request replayed the first's stored result instead of creating a new charge.

3. Prove a different idempotency key creates a genuinely new charge

curl -s -X POST http://localhost:8501/v1/charges -H "Idempotency-Key: idem-xyz" -H "Content-Type: application/json" -d '{"amount": 1000}'
curl -s http://localhost:8501/debug/charge_count

Expected: a new, different charge id, and the count increments to 2 — confirming the dedup is scoped to the key, not a blanket "ignore repeat calls" rule.

4. Build the webhook receiver and verify a real signature

# webhook_receiver.py
from fastapi import FastAPI, Request, HTTPException
import hmac, hashlib, time, json

app = FastAPI()
WEBHOOK_SECRET = "whsec_test_secret"
processed_events = set()

def verify_signature(payload: str, sig_header: str, secret: str, tolerance_seconds=300) -> bool:
    parts = dict(p.split("=", 1) for p in sig_header.split(","))
    timestamp = int(parts["t"])
    if abs(time.time() - timestamp) > tolerance_seconds:
        return False
    signed_payload = f"{timestamp}.{payload}"
    computed_sig = hmac.new(secret.encode(), signed_payload.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(computed_sig, parts["v1"])

@app.post("/webhooks/payment")
async def receive_webhook(request: Request):
    raw_body = (await request.body()).decode()
    sig_header = request.headers.get("X-Signature", "")
    if not verify_signature(raw_body, sig_header, WEBHOOK_SECRET):
        raise HTTPException(status_code=400, detail="invalid signature")
    event = json.loads(raw_body)
    if event["id"] in processed_events:
        return {"status": "duplicate_ignored"}
    processed_events.add(event["id"])
    return {"status": "processed", "type": event["type"]}

python3 -m uvicorn webhook_receiver:app --port 8502

Send a correctly-signed event:

import hmac, hashlib, time, json, httpx

WEBHOOK_SECRET = "whsec_test_secret"
def sign(payload, secret, ts):
    sig = hmac.new(secret.encode(), f"{ts}.{payload}".encode(), hashlib.sha256).hexdigest()
    return f"t={ts},v1={sig}"

payload = json.dumps({"id": "evt_1", "type": "charge.succeeded"})
ts = int(time.time())
header = sign(payload, WEBHOOK_SECRET, ts)
r = httpx.post("http://localhost:8502/webhooks/payment", content=payload, headers={"X-Signature": header})
print(r.json())

Expected: {"status": "processed", "type": "charge.succeeded"}.

5. Prove a redelivered event is ignored, and a forged one is rejected

Send the exact same signed payload from exercise 4 a second time:

Expected: {"status": "duplicate_ignored"} — the same event id was recognized and not reprocessed.

Now send a request with a signature that doesn't match (e.g. X-Signature: t=<ts>,v1=deadbeef):

Expected: HTTP 400 — an unsigned or incorrectly-signed request is rejected outright, exactly module 06's "verify before you trust anything in the body" rule, now exercised against a scheme matching a real provider's exact algorithm.

6. Simulate out-of-order delivery and confirm the safe handler wins

Using the "RIGHT" handler from the Concepts section, simulate a charge.succeeded event with created=200 arriving, followed by a stale charge.pending event with created=100 (an out-of-order redelivery) for the same charge. Expected: the naive last-write-wins handler would incorrectly downgrade the charge back to pending; the timestamp-comparing handler correctly ignores the stale event because its created value is older than what's already recorded. Write both handlers and run both scenarios to see the difference directly.

7. Diagnose and fix: a customer was charged twice

A team's checkout flow calls "create charge" directly from the request handler with no idempotency key. Under normal load it's fine; during a brief spike in provider latency, the client's HTTP request to your backend times out and the frontend automatically retries the checkout submission. Support tickets report a customer charged twice for one order.

8. Clean up

# Ctrl+C the running mock_payment_api and webhook_receiver uvicorn processes.

Independent challenge

No code given. Design the full payment flow for a subscription business: a monthly charge is attempted automatically for each active subscriber. Specify: (1) how the idempotency key for each month's charge attempt is derived so that automatic retries within the same billing cycle can't double-charge, but next month's charge is still a genuinely new attempt; (2) how your webhook handler treats a charge.succeeded arriving for a subscription that was cancelled by the customer after the charge was already initiated but before the webhook arrived — should the charge be refunded automatically, and what does the append-only ledger from the Concepts section let you reconstruct about what happened, in what order; (3) what happens if the payment provider's webhook delivery is degraded for several hours during your billing run — does your system still know which subscribers were actually charged.

Common mistakes & troubleshooting

• No idempotency key on the outbound charge call. As exercise 7 showed, an ordinary client retry after a timeout becomes a real double charge with no protection — this is the single highest-stakes gap this module addresses.
• Regenerating the idempotency key on every retry instead of once per attempt. A key that changes on each retry defeats the entire mechanism — the provider sees each retry as a brand-new request. The key must be stable across retries of the same logical attempt.
• Trusting a client-supplied charge amount. Always compute the amount server-side from your own authoritative pricing logic immediately before charging — never accept an amount from the request body as-is.
• Applying webhook events in arrival order instead of the event's own order. Providers don't guarantee delivery order; exercise 6 showed a stale, out-of-order redelivery can incorrectly overwrite newer state if you don't compare the event's own timestamp before applying it.
• Treating "current status" as the only source of truth. An append-only ledger (module 10's event-sourcing idea, applied to money) lets you reconstruct and audit what actually happened and in what order — a single mutable status column can't answer "wait, what happened here?" after the fact.
• Skipping module 06's signature verification because "it's just an internal endpoint." A payment webhook endpoint is exactly the highest-stakes place to apply that discipline rigorously, not a place to relax it.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What are the two independent directions in a payment integration, and which reliability mechanism belongs to which direction?
2. Why must an idempotency key be generated once per logical attempt and reused across retries, rather than generated fresh on every HTTP call?
3. In the Stripe-style signature scheme, why is the timestamp folded into the signed content instead of just being carried in a separate header field?
4. Why isn't deduplicating webhook events by their id alone always sufficient to keep your local state correct?
5. Why should a payment system's local record be an append-only ledger rather than a single mutable "current status" column?
6. In the double-charge scenario from exercise 7, why wouldn't perfect inbound webhook handling (module 06's discipline) have prevented the bug?

Further reading & sources

• Stripe: Idempotent requests - the exact idempotency-key mechanism this module's outbound examples are modeled on.
• Stripe: Webhook signatures - the precise t=...,v1=... signature scheme and manual verification algorithm this module's receiver reproduces exactly.
• Stripe: Best practices for webhooks - out-of-order delivery, retry behavior, and idempotent event handling from the payment provider's own side.
• PCI Security Standards Council: PCI DSS Quick Reference Guide - why you generally should never handle raw card numbers yourself and should rely on a provider's tokenization/hosted fields instead.
• Martin Fowler: Event Sourcing - the append-only-ledger reasoning behind this module's "never overwrite, always append" guidance, referenced from module 10.

Next

13-batch-etl-pipeline-orchestration — a different kind of "work that happens outside the request cycle": not a single job or a single webhook, but many dependent steps that must run on a schedule, in the right order, with the failure of one step correctly blocking the ones that depend on it.