Module 04: The Saga Pattern

Why this matters

The previous module left you with a verdict: two-phase commit gives you real cross-system atomicity, and you almost never want it, because it re-couples the availability of services you deliberately split apart and freezes them all behind a fragile coordinator. But the underlying problem didn't go away. You still have a checkout that must charge a card, decrement inventory, and create a shipment — three operations in three services with three databases — and you still can't let the charge succeed while the inventory decrement silently fails. If 2PC is off the table, what actually holds this together?

The saga pattern is the answer the whole industry converged on. Instead of one atomic commit across all three systems, you run a sequence of local transactions — each one committing independently and durably in its own database — and you make the whole sequence eventually consistent by attaching, to every step, a compensating transaction that semantically undoes it. If step 3 fails, you run the compensations for steps 2 and 1 in reverse, and the system converges back to a consistent state. You give up the instant, all-or-nothing atomicity of a single transaction; in exchange you get loose coupling, independent availability, no blocking coordinator, and a design that scales. This is not a niche technique — it is the default way multi-step business transactions are built across services, and almost every "workflow," "order pipeline," or "onboarding flow" you'll meet in a service-oriented system is a saga whether or not anyone called it that. This module is where the reliability primitives so far — consistency choices (CAP and consistency), idempotency (01), locking (02), and the 2PC trade-off (03) — combine into a pattern you'll actually ship.

Concepts

What a saga is: a sequence of local transactions plus compensations

A saga is a long-running transaction decomposed into a series of smaller, local transactions T₁, T₂, … Tₙ, each of which commits independently in a single service's database. Because each local transaction commits on its own, there is no global lock and no coordinator holding everyone hostage — but there is also no global rollback. A database can ROLLBACK an uncommitted transaction; it cannot un-commit T₁ once T₁ is durable. So a saga replaces rollback with compensation: for each forward transaction Tᵢ you define a compensating transaction Cᵢ that semantically undoes its effect. If the saga gets partway and a step fails, you execute the compensations for the already-completed steps, in reverse order, to walk the system back to consistency.

The word semantically is doing a lot of work. Cᵢ does not restore the exact prior bytes — it issues a new business action that negates the old one. The compensation for "charge the card" is not "pretend the charge never happened"; it's "issue a refund." The compensation for "reserve inventory" is "release the reservation." The compensation for "send the shipping label" might be "cancel the label" — or, if the package already went out, there is no compensation and you must design the saga so that irreversible steps come last. This is the central mental shift from 2PC: a saga does not promise the intermediate states never happened; it promises the system converges to a consistent end state, forward (all steps done) or backward (all done steps compensated).

  Forward:   T1 charge ──► T2 reserve ──► T3 confirm ──► CONFIRMED
             card         inventory      order          (all committed)
                              │
                              │ T3 fails (or T2 fails, etc.)
                              ▼
  Backward:  C1 refund ◄── C2 release ◄──┘   compensations run in REVERSE,
             card         inventory          only for steps that completed
                 │
                 ▼
             CANCELLED  (consistent end state — no lost money, no dangling hold)

The window of inconsistency, and why it's usually fine

A saga is, by construction, not atomic. Between T₁ committing and Tₙ committing (or between a failure and its compensations completing), the system is in a partially-updated state that a strict cross-system transaction would never expose: the payment is captured but the order isn't confirmed yet; the inventory is reserved but the shipment doesn't exist yet. This is precisely the "brief window of inconsistency" from module 03 — the thing 2PC exists to prevent and that you are now choosing to accept.

The per-operation consistency judgment (Consistency Models) is exactly what tells you whether that's acceptable, and it almost always is, because you model the window explicitly rather than pretending it doesn't exist. The order sits in a pending state — visible, well-defined, and self-correcting — until the saga completes and flips it to confirmed, or a compensation flips it to cancelled. The user sees "your order is being processed," not a half-broken screen. The business tolerates a few seconds where payment is captured but the shipment isn't yet booked, because the saga will converge and the states in between are named and handled. The senior instinct: never let an intermediate state be an accident; make it a first-class, queryable status that your UI and your operators understand.

Choreography: services react to events, no central brain

There are two ways to coordinate a saga, and choosing between them is the main design decision. The first is choreography: there is no central coordinator. Each service does its local transaction and then publishes an event; other services subscribe to those events and react by doing their own local transaction and publishing the next event. The saga advances as a chain reaction of events, and each service knows only "when I see event X, I do my work and emit event Y."

Concretely, for an order saga: the Order service creates a pending order and emits OrderCreated; the Payment service, subscribed to OrderCreated, charges the card and emits PaymentCaptured (or PaymentFailed); the Inventory service, subscribed to PaymentCaptured, reserves stock and emits InventoryReserved (or InventoryOutOfStock); the Order service, subscribed to InventoryReserved, flips the order to confirmed. Compensation is also event-driven: InventoryOutOfStock is consumed by the Payment service, which refunds and emits PaymentRefunded, which the Order service consumes to mark the order cancelled.

The appeal: services are maximally decoupled — no component knows the whole flow, you add a participant by subscribing it to the right event, and there's no central bottleneck. The cost: the business process is implicit, smeared across many services' subscriptions, so no single place tells you "here is what a checkout does." That makes the flow hard to see, hard to change, and prone to cyclic dependencies between services. Choreography shines for simple sagas with few steps; it becomes a maintenance nightmare as steps and branches multiply.

Orchestration: a coordinator tells each service what to do

The second style is orchestration: a single component — the saga orchestrator (or "saga coordinator," or in workflow engines a "process manager") — owns the flow. It knows the whole sequence and issues commands to each service in turn: "Payment, charge this card"; on success, "Inventory, reserve this stock"; on failure, it drives the compensations: "Payment, refund this charge." Each service just executes the command it's told and reports back; only the orchestrator knows the overall plan and the current step.

Do not confuse this with the 2PC coordinator. The 2PC coordinator holds locks open across all participants and blocks them until a global decision — the orchestrator holds no locks; each service commits its local transaction immediately and the orchestrator merely remembers "where are we, and what's next." An orchestrator crash doesn't freeze anyone's data; it just pauses the workflow, which resumes from its persisted state when the orchestrator recovers.

The appeal: the entire business process lives in one place, explicit and readable — you can see, test, and change the flow without archaeology across ten services; the compensation logic is centralized; and there are no hidden cyclic event dependencies. The cost: the orchestrator is a component you must build and operate (there's a risk of it becoming a bloated "god service" with all the business logic if you're not disciplined about keeping the work in the services and only the sequencing in the orchestrator). Rule of thumb: reach for choreography when the saga is short and simple and the services are naturally event-driven; reach for orchestration when the flow has several steps, branches, or compensations, or when you need to see and evolve the process — which, past a trivial size, is most of the time. Production systems increasingly use a workflow engine (Temporal, AWS Step Functions, Camunda) that is an orchestrator with durable state built in.

Compensation is hard: design for it deliberately

Compensating transactions look simple in a diagram and are the source of most real-world saga bugs. Four disciplines make them correct:

• Order irreversible steps last. Some actions can't be compensated: an email is sent, a package physically ships, a third-party API has no "undo." Structure the saga so these come as late as possible — ideally as the final step, after every reversible step has already succeeded — so you rarely need to unwind past them. When you must place an irreversible step early, wrap it in a reservation you can release (reserve inventory now, actually decrement on confirm) so the early action is itself reversible.
• Compensations must be idempotent (module 01). Sagas run on top of at-least-once messaging and retrying orchestrators, so every step and every compensation will sometimes run twice. "Refund the charge" must refund once even if invoked three times — key it on the charge id and dedupe. This is why module 01 came first: a saga without idempotent steps double-charges and double-refunds under the exact retries it depends on.
• Compensations can fail too — and must be retried, not abandoned. A refund can itself time out. A compensation that fails cannot simply give up (that leaves money captured for a cancelled order); it must be retried with backoff until it succeeds, and if it truly can't, escalated to a human / a dead-letter queue. Sagas are "forward recovery or backward recovery, but always to a consistent state" — you don't get to stop halfway.
• Beware semantic gaps the compensation can't close. Between reserving a seat and releasing it, someone saw "sold out." Between charging and refunding, the customer saw a charge on their statement. Compensation restores the system's invariants, not the observers' experience. Where that gap matters (a visible charge), prefer a design that avoids the forward action until you're sure — e.g. authorize the card (a hold) and only capture on full success, so a failure voids the hold and the customer never sees a real charge.

Command reference

A minimal orchestrator for the order/payment/inventory saga. The orchestrator persists its position so a crash resumes rather than restarts, drives forward on success and compensations on failure, and treats every call as idempotent:

from enum import Enum
from sqlalchemy import create_engine, text

db = create_engine("postgresql+psycopg://app@orchestrator-db:5432/sagas")

# CREATE TABLE saga_state (
#   saga_id     TEXT PRIMARY KEY,
#   order_id    TEXT NOT NULL,
#   step        TEXT NOT NULL,     -- current step / status
#   charge_id   TEXT,              -- filled once payment succeeds
#   created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
# );

class Step(str, Enum):
    STARTED            = "started"
    PAYMENT_DONE       = "payment_done"
    INVENTORY_DONE     = "inventory_done"
    CONFIRMED          = "confirmed"          # forward success (terminal)
    COMPENSATING       = "compensating"
    CANCELLED          = "cancelled"          # backward success (terminal)

def _set_step(saga_id: str, step: Step, **cols) -> None:
    sets = ", ".join(["step = :step"] + [f"{k} = :{k}" for k in cols])
    with db.begin() as c:
        c.execute(text(f"UPDATE saga_state SET {sets} WHERE saga_id = :sid"),
                  {"sid": saga_id, "step": step.value, **cols})

def run_order_saga(saga_id: str, order_id: str, card, items) -> str:
    """Drive the saga forward; on any failure, compensate backward.
    Every service call MUST be idempotent, keyed on saga_id, because this
    function can be retried from the top after an orchestrator crash."""
    with db.begin() as c:
        c.execute(text("INSERT INTO saga_state (saga_id, order_id, step) "
                       "VALUES (:sid, :oid, :s) ON CONFLICT (saga_id) DO NOTHING"),
                  {"sid": saga_id, "oid": order_id, "s": Step.STARTED.value})
    try:
        # T1: capture payment (idempotency key = saga_id -> provider dedupes retries)
        charge_id = payment_service.charge(idem_key=saga_id, card=card, order=order_id)
        _set_step(saga_id, Step.PAYMENT_DONE, charge_id=charge_id)

        # T2: reserve inventory (idempotent on saga_id)
        inventory_service.reserve(idem_key=saga_id, order=order_id, items=items)
        _set_step(saga_id, Step.INVENTORY_DONE)

        # T3 (last, hardest to undo): confirm the order
        order_service.confirm(order_id)
        _set_step(saga_id, Step.CONFIRMED)
        return "confirmed"
    except Exception as exc:
        _compensate(saga_id, order_id)
        return f"cancelled: {exc}"

def _compensate(saga_id: str, order_id: str) -> None:
    """Run compensations in REVERSE order for whatever forward steps completed.
    Each compensation is idempotent and retried until it succeeds."""
    _set_step(saga_id, Step.COMPENSATING)
    with db.begin() as c:
        st = c.execute(text("SELECT step, charge_id FROM saga_state "
                            "WHERE saga_id = :sid"), {"sid": saga_id}).one()
    # Undo inventory only if it was reserved.
    if st.step in (Step.INVENTORY_DONE.value,):
        inventory_service.release(idem_key=saga_id, order=order_id)
    # Undo payment only if it was captured.
    if st.charge_id:
        payment_service.refund(idem_key=saga_id, charge_id=st.charge_id)
    order_service.cancel(order_id)
    _set_step(saga_id, Step.CANCELLED)

The same first step done as choreography — the Payment service reacts to an event, does its local transaction, and publishes the next event (with a transactional outbox so the event can't be lost even if the process dies right after committing):

# Payment service: consumes OrderCreated, emits PaymentCaptured / PaymentFailed.
def on_order_created(event):
    saga_id, order_id = event["saga_id"], event["order_id"]
    try:
        with db.begin() as c:                       # one local transaction:
            # Idempotent claim so a redelivered OrderCreated doesn't double-charge.
            claimed = c.execute(
                text("INSERT INTO processed (saga_id) VALUES (:s) "
                     "ON CONFLICT DO NOTHING RETURNING saga_id"),
                {"s": saga_id}).first()
            if claimed is None:
                return                               # already handled this event
            charge_id = capture_card(order_id)       # the local effect
            # Outbox row commits in the SAME txn as the effect -> never lost.
            c.execute(text("INSERT INTO outbox (topic, payload) VALUES "
                           "('PaymentCaptured', :p)"),
                      {"p": json({"saga_id": saga_id, "order_id": order_id,
                                  "charge_id": charge_id})})
    except PaymentDeclined:
        with db.begin() as c:
            c.execute(text("INSERT INTO outbox (topic, payload) VALUES "
                           "('PaymentFailed', :p)"),
                      {"p": json({"saga_id": saga_id, "order_id": order_id})})
    # A separate relay process reads `outbox` and publishes to the broker,
    # at-least-once; downstream consumers are idempotent (module 01).

Hands-on exercises

You'll simulate three services. The lightest setup is one Postgres with three schemas (orders, payments, inventory) standing in for three databases, plus Redis or a simple in-process queue for events. docker run -d --name pg -p 5432:5432 -e POSTGRES_PASSWORD=pg postgres:16; pip install fastapi uvicorn "sqlalchemy>=2" psycopg[binary] redis. Create per-service tables: orders(id, status), payments(charge_id, order_id, amount, status), inventory(sku, reserved, order_id), plus saga_state from the command reference and a processed(saga_id) dedupe table.

1. Build the happy-path orchestrated saga

Implement run_order_saga with three stubbed services (payment_service.charge inserts a payments row and returns a charge_id; inventory_service.reserve inserts an inventory reservation; order_service.confirm sets the order to confirmed). Run it end to end.

Expected: the order ends confirmed, a payment row exists, inventory is reserved, and saga_state.step walks started → payment_done → inventory_done → confirmed. You've run a multi-service transaction with no 2PC and no global lock — each step committed independently.

2. Force a late failure and watch compensation run backward

Make inventory_service.reserve raise (simulate out-of-stock). Run the saga.

Expected: the saga catches the failure and _compensate runs — it refunds the already-captured payment and cancels the order, so the end state is cancelled with the payment refunded and no dangling reservation. Confirm the compensations ran in reverse order and only for the steps that had actually completed (no attempt to release inventory that was never reserved). This is backward recovery to a consistent state.

3. Observe the window of inconsistency

Add a sleep(3) between PAYMENT_DONE and the inventory step, and while it sleeps, query all three services' state and saga_state.

Expected: you can see the intermediate state — payment captured, order still pending, inventory not yet reserved. This is the window 2PC would have prevented. Note that the state is named (payment_done) and queryable, not a corrupt half-state — that's what makes the window acceptable. Write one sentence on what a user should see in your UI during this window.

4. Make every step idempotent and prove double-execution is safe

Give payment_service.charge an idempotency key (saga_id) and a processed-style dedupe so a second call with the same key returns the first charge_id instead of charging again. Call run_order_saga twice with the same saga_id (simulating an orchestrator crash-and-retry).

Expected: exactly one payment row, one reservation, one confirmed order — the second run finds every step already done and is a no-op that converges to the same state. Then remove the idempotency key from charge and rerun: two payment rows. This is why "sagas require idempotent steps" is not optional — the orchestrator will retry.

5. Crash the orchestrator mid-saga and resume from persisted state

After PAYMENT_DONE is written but before inventory, raise SystemExit. Restart and call the saga again with the same saga_id. Have run_order_saga read saga_state and skip steps already completed.

Expected: the resumed run does not re-charge (payment already done, and it's idempotent anyway), reserves inventory, and confirms. The persisted saga_state is what lets an orchestrator recover instead of restarting from zero or leaving a half-finished saga stranded. Contrast with 2PC: here the crash paused a workflow; it did not freeze anyone's rows behind held locks.

6. Build the same first two steps as choreography

Reimplement steps 1–2 as event-driven: OrderCreated → Payment consumes, emits PaymentCaptured → Inventory consumes, emits InventoryReserved → Order confirms. Use the transactional-outbox pattern from the command reference for at least one service.

Expected: the same end state, but now no component knows the whole flow — each only knows its own event-in/event-out. Then answer in writing: where does the "what does checkout do?" knowledge live now, and how would you add a Fraud-check step between payment and inventory? (In choreography you re-wire subscriptions; note how much harder the flow is to see than in the orchestrator.)

7. Design a compensation for an irreversible step

Add a step that sends a confirmation email (irreversible — you can't un-send it). Place it correctly in the saga and justify the placement. Then handle the case where inventory fails after a hypothetical early email.

Expected: the email must be the last step (after confirm), so a failure never needs to unwind past it; if forced earlier, its "compensation" is a follow-up "your order was cancelled" email, not a true undo. Articulate the rule: irreversible steps go last, and where they can't, their compensation is a new corrective action, not a rollback.

8. Reservation pattern to make an early irreversible step reversible

Change inventory from "decrement stock" (hard to undo — someone may have bought the freed unit) to "reserve stock" now and "commit the decrement" only on final confirm, with "release the reservation" as the compensation.

Expected: the inventory step is now cleanly reversible (release the hold) right up until final confirmation, so compensation is trivial and can't oversell. Note this is the same idea as authorize-then-capture for payments: turn an irreversible action into a reservation you can release.

9. Diagnose and fix: the saga that loses money on retries

Ops reports that during a broker incident, some cancelled orders left the customer charged while others were refunded twice. The saga is choreographed: Payment consumes OrderCreated and charges; on InventoryOutOfStock it refunds. The refund handler does provider.refund(charge_id) with no idempotency key, and the broker redelivered events during the incident. Explain both failure directions and give the fix.

Independent challenge

No code given. Recall 03-distributed-transactions-and-two-phase-commit — you rejected 2PC for a checkout there; now design the saga that replaces it. You're building "book a trip": the operation must reserve a flight seat, reserve a hotel room, and charge the customer once for the total, across three independent services with three databases. Any of the three can fail or be temporarily down. Write a design that specifies: (1) choreography or orchestration, and defend the choice for this flow's complexity; (2) the ordered list of forward steps and each step's compensating transaction, explicitly placing the irreversible payment where a failure rarely has to unwind past it (hint: authorize vs capture); (3) how you keep every step and compensation idempotent given at-least-once events; (4) what state the customer sees during the inconsistent window and what they see if the hotel reservation fails after the flight is already held; and (5) one sentence on why this is better than the 2PC you'd have written in module 03.

Common mistakes & troubleshooting

• Treating a compensation as a byte-level rollback. A saga cannot un-commit a local transaction; a compensation is a new business action (refund, release, cancel) that semantically negates the old one. If you find yourself wanting to "restore the previous row exactly," you're thinking in 2PC terms — redesign around a forward corrective action.
• Non-idempotent steps or compensations. Sagas run on at-least-once events and retrying orchestrators, so every step will sometimes run twice. A step or compensation without an idempotency key double-charges or double-refunds under the exact retries the pattern depends on. Key every step on (saga_id, step).
• Putting an irreversible step early. If an un-undoable action (send email, ship package, capture payment) runs before steps that can still fail, a later failure can't unwind past it. Order irreversible steps last, or wrap them in a reservation you can release (authorize-then-capture, reserve-then-commit).
• Letting a compensation fail silently. Compensations can time out too. A refund that fails and is abandoned leaves money captured for a cancelled order. Retry compensations with backoff until they succeed, and dead-letter/escalate the ones that truly can't — a saga must always reach a consistent end state, forward or backward.
• Unnamed intermediate states. If the "half-done" state is an accident rather than a first-class pending/compensating status, your UI and operators can't tell a mid-saga order from a corrupt one. Model every intermediate state explicitly and make it queryable.
• Reaching for choreography on a complex flow. Choreography's implicit, event-smeared flow is fine for two or three simple steps but becomes unreadable and cycle-prone as branches and compensations grow. Past that size, use an orchestrator (or a workflow engine) so the process lives in one place.
• Building a bloated orchestrator. The orchestrator should own sequencing and compensation, not the business work — keep the actual charging/reserving in the services. An orchestrator that accumulates all the domain logic becomes the distributed monolith you were trying to avoid.
• Confusing the orchestrator with a 2PC coordinator. The orchestrator holds no locks and blocks no one; each service commits locally and immediately, and an orchestrator crash pauses a workflow rather than freezing rows. If your "orchestrator" holds resources open across services, you've reinvented 2PC.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What is a saga, and how does it provide "all-or-nothing" without the atomic cross-system commit that 2PC uses?
2. Why is a compensating transaction not the same as a rollback? Give a concrete compensation for "charge the customer's card."
3. Contrast choreography and orchestration. Give one flow where each is the better choice and say why.
4. A saga has an unavoidable window of inconsistency. Why is that usually acceptable, and what must you do to make it acceptable?
5. Why must every saga step and every compensation be idempotent? What breaks if they aren't?
6. Why should irreversible steps be ordered last, and what technique makes an early "irreversible" step reversible?

Cumulative review

Closed book — cover the earlier modules and answer from memory. This ties modules 01–04, and the consistency lessons before them, into one chain, because they are one chain: each pattern exists to handle a failure the previous one exposed.

1. (SD 15 → 04) A saga leaves the system partially updated for a window. Name the consistency model that window represents, explain why CAP/PACELC says you're not getting something for nothing by accepting it, and give the per-operation reasoning that decides whether a particular saga's window is tolerable.
2. (01 → 04) Explain precisely why a saga is broken without idempotent steps. Reference at-least-once delivery and describe the exact mechanism (key + dedupe store) you'd use to make a saga's "charge payment" step safe to retry.
3. (02) A team guards a saga's "reserve the last unit of inventory" step with a single-Redis distributed lock and calls it correct. Using the efficiency-vs- correctness distinction, say what's still wrong and what actually makes the reservation correct under a process pause.
4. (03 → 04) In one paragraph, tell the story from 2PC to sagas: what problem 2PC solves, the specific failure mode that makes it unsuitable for a high-throughput checkout, and how a saga sidesteps that failure mode (and what it gives up to do so).
5. (01–04 synthesis) You're handed "process a refund request": validate the order, reverse the payment, restock the item, and email the customer. Sketch this as a saga — choreography or orchestration and why, the forward steps and their compensations, where the idempotency keys live, which step is irreversible and where it goes, and what consistency the customer experiences throughout.

Further reading & sources

• Saga pattern (microservices.io) - the reference definition of choreography vs orchestration sagas and compensating transactions.
• Sagas (Garcia-Molina & Salem, 1987) - the original paper that introduced sagas and semantic compensation.
• Saga distributed transactions pattern (Microsoft Azure) - a practical guide to orchestrated sagas, state, and compensation ordering.
• Compensating Transaction pattern (Microsoft Azure) - focused guidance on the "compensations are new business actions, and can fail too" discipline stressed here.
• Implementing the saga pattern with AWS Step Functions - a worked orchestrated saga on a durable workflow engine, the direction this module recommends past trivial size.

Next

05-cqrs — sagas coordinate writes across services, and a recurring theme has been that reads and writes want different things: the write path needs strong consistency and careful coordination, while many reads happily tolerate staleness and just want to be fast. The next module makes that split architectural. CQRS (Command Query Responsibility Segregation) separates the model you write through from the model(s) you read from — letting each be optimized, scaled, and made consistent independently — and, crucially, examines when that separation earns its considerable extra complexity and when it's an overkill you'll regret.