Module 08: Durable Workflow Orchestration

Why this matters

Module 04 built a real, correct saga orchestrator by hand: a saga_state table you write to after every step, a _compensate function you wrote yourself, idempotency keys you designed and applied one by one, and — in exercise 5 — a manual "read saga_state, skip completed steps" recovery path for when the orchestrator process itself crashes mid-saga. It works, and understanding it by hand is exactly what makes the next tool make sense: a durable execution engine (Temporal is this module's example; AWS Step Functions and Camunda solve the same problem for different platforms) gives you everything that hand-rolled machinery did — crash-resume, automatic retries, a durable record of every step — as a property of the platform, not code you wrote and must keep correct yourself. This module shows you exactly what that trade buys you, using the same order-charge-reserve-confirm flow from module 04, so you can compare the two approaches directly rather than taking the promise on faith.

Concepts

The problem restated: your hand-rolled orchestrator already does this — manually

Look back at module 04's run_order_saga: every step writes to saga_state before moving on, specifically so that a crash can be recovered from by re-reading that table and skipping finished steps. That pattern — persist progress, replay to resume — is exactly what a durable execution engine automates. The difference isn't a new concept; it's who implements the mechanism and how much of it you have to get right yourself, by hand, in every workflow you write.

Event history: how the engine remembers, without you writing to a table

A durable execution engine like Temporal doesn't ask you to write saga_state rows. Instead, the engine itself durably records an event history for every workflow run — "activity X started," "activity X completed with result Y," "timer fired," "signal received" — every time your workflow code does something meaningful. When a worker process restarts (crashes, redeploys, whatever), a fresh worker picking up that workflow doesn't start over: it replays the event history from the beginning, and your workflow code re-executes deterministically — but every already-recorded activity call returns its already-recorded result instantly instead of actually re-running the side effect. Only the point past what's in history actually does new work.

  Hand-rolled (module 04):              Durable execution engine:
  YOU write to saga_state               ENGINE writes event history
  after each step                       automatically, transparently

  YOU read saga_state on restart        ENGINE replays history on restart
  and skip completed steps              — your workflow code just runs
                                         again; completed activities
                                         return cached results, don't
                                         re-execute

Workflows and activities: the same split you already made in module 04

The vocabulary maps directly onto what you already built:

• A workflow function is your orchestration logic — the sequence of steps, exactly like run_order_saga. Rule: workflow code must be deterministic, because it gets replayed on every recovery. No direct network calls, no reading the real clock, no random numbers inside workflow code itself — anything with a real side effect or non-deterministic result belongs in an activity instead.
• An activity function is where actual side effects happen — calling the payment service, hitting a database, anything module 04 called "T1, T2, T3." Activities are not replayed — the engine records their result once and reuses it, which is exactly why workflow code can safely re-run from the top without re-charging a card.

from temporalio import workflow, activity
from datetime import timedelta

@activity.defn
async def charge_card(order_id: str) -> str:
    # Real side effect — calls out, may fail, may take time.
    return f"charge-{order_id}"

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        # Orchestration logic ONLY — no direct side effects here.
        charge_id = await workflow.execute_activity(
            charge_card, order_id, start_to_close_timeout=timedelta(seconds=10)
        )
        return charge_id

Retries are the engine's job now, not a try/except you wrote

Module 01 (idempotency) and module 04 both leaned on retries you had to reason about and often implement yourself. A durable execution engine retries a failed activity automatically, with a configurable backoff policy, and your workflow code doesn't contain a single retry loop:

from temporalio.common import RetryPolicy

result = await workflow.execute_activity(
    charge_card, order_id,
    start_to_close_timeout=timedelta(seconds=10),
    retry_policy=RetryPolicy(initial_interval=timedelta(milliseconds=100), maximum_attempts=5),
)

If charge_card raises, the engine automatically re-invokes it — with backoff, up to maximum_attempts — with no code in your workflow handling the retry. This is module 04's "the compensations must be retried, not abandoned" discipline, built into the platform instead of hand-written per step. (Idempotency still matters — module 01's lesson doesn't disappear — because the activity itself might partially succeed before failing; the engine retrying it doesn't make the underlying side effect idempotent for you.)

Signals: getting an external event into a running, possibly long-lived workflow

Module 04's saga ran start-to-finish in seconds. Real workflows are often much longer-lived — "wait for a human to approve this expense report," which could take days. A signal is how outside code delivers an event into an already-running workflow, and workflow.wait_condition is how the workflow waits for it without polling:

@workflow.defn
class ApprovalSaga:
    def __init__(self):
        self._approved = False

    @workflow.signal
    def approve(self):
        self._approved = True

    @workflow.run
    async def run(self, order_id: str) -> str:
        await workflow.wait_condition(lambda: self._approved)
        return f"{order_id}-approved"

The workflow can sit waiting for days — the engine doesn't hold a process or a thread open the whole time; it's durably parked in event history until a signal arrives, then resumes exactly where it left off. This is a capability module 04's hand-rolled saga_state table doesn't give you for free: a truly long-running, crash-proof "wait for something external" step.

What you give up: a platform dependency and a learning curve

This isn't a strictly-better replacement for everything module 04 taught — it's a different point on the same cost curve module 04 kept coming back to. You now depend on running (or paying for) the workflow engine itself; your workflow code must follow the determinism rule (no direct I/O, no datetime.now(), no raw randomness inside workflow functions — mistakes here are a real, non-obvious class of bug specific to this model); and you've added an infrastructure dependency for flows that might have been simple enough for module 04's choreography. Reach for a durable execution engine when: sagas are long-running (hours/days, not seconds), have many steps/branches, need human-in-the-loop signals, or you're maintaining enough hand-rolled orchestrators that the platform's cost is cheaper than N more saga_state tables. Stick with module 04's approach when: the saga is short, few steps, and adding a new piece of infrastructure isn't worth it yet — the same escalation judgment this whole track has taught from the start.

Command reference

This module uses Temporal, run locally with its own dev-mode server (no separate database or Docker Compose needed for local development).

Hands-on exercises

pip install temporalio. Start the local dev server once, in its own terminal, and leave it running for all exercises:

temporal server start-dev

1. Run the order saga as a durable workflow

import asyncio
from datetime import timedelta
from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker

@activity.defn
async def charge_card(order_id: str) -> str:
    return f"charge-{order_id}"

@activity.defn
async def reserve_inventory(order_id: str) -> None:
    pass

@activity.defn
async def confirm_order(order_id: str) -> None:
    pass

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        charge_id = await workflow.execute_activity(
            charge_card, order_id, start_to_close_timeout=timedelta(seconds=10)
        )
        await workflow.execute_activity(
            reserve_inventory, order_id, start_to_close_timeout=timedelta(seconds=10)
        )
        await workflow.execute_activity(
            confirm_order, order_id, start_to_close_timeout=timedelta(seconds=10)
        )
        return f"confirmed:{charge_id}"

async def main():
    client = await Client.connect("localhost:7233")
    async with Worker(
        client, task_queue="orders-tq",
        workflows=[OrderSaga], activities=[charge_card, reserve_inventory, confirm_order],
    ):
        result = await client.execute_workflow(
            OrderSaga.run, "order-1", id="saga-order-1", task_queue="orders-tq"
        )
        print("RESULT:", result)

if __name__ == "__main__":
    asyncio.run(main())

Expected: RESULT: confirmed:charge-order-1. Open http://localhost:8233 and find this workflow run — you can see its full event history (every activity call and result), something module 04's saga_state table only gave you a single current row of, not a full audit trail.

Gotcha to know before you hit it: Temporal's Python SDK re-imports your workflow-defining module inside a sandbox to guarantee determinism. If your script calls asyncio.run(main()) at module level with no guard, that call re-executes during the sandboxed re-import and crashes with RuntimeError: asyncio.run() cannot be called from a running event loop. Always guard your entrypoint with if __name__ == "__main__": — every example in this module already does.

2. Prove retries are automatic — no retry loop in your code

import asyncio
from datetime import timedelta
from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.common import RetryPolicy

@activity.defn
async def flaky_charge(order_id: str) -> str:
    info = activity.info()
    if info.attempt < 3:
        raise RuntimeError("simulated transient failure")
    return f"charge-{order_id}"

@workflow.defn
class FlakySaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        return await workflow.execute_activity(
            flaky_charge, order_id,
            start_to_close_timeout=timedelta(seconds=10),
            retry_policy=RetryPolicy(initial_interval=timedelta(milliseconds=100), maximum_attempts=5),
        )

async def main():
    client = await Client.connect("localhost:7233")
    async with Worker(client, task_queue="flaky-tq", workflows=[FlakySaga], activities=[flaky_charge]):
        result = await client.execute_workflow(FlakySaga.run, "order-2", id="saga-order-2", task_queue="flaky-tq")
        print("RESULT:", result)

if __name__ == "__main__":
    asyncio.run(main())

Expected: the activity raises on attempts 1 and 2 (visible in the worker's log output), succeeds on attempt 3, and the workflow still completes with RESULT: charge-order-2 — your workflow code contains no try/except, no backoff, no attempt counter. Compare this to module 04's payment_service.charge idempotency-key discipline: idempotency still matters here too (the retry policy will call flaky_charge again on failure, so if the activity's side effect can partially succeed before raising, it still needs to be safe to retry) — the engine automates the retry mechanics, not the idempotency of your side effect.

3. Kill the worker mid-workflow and prove it resumes without re-running completed work

This is the exercise that actually proves the "durable" claim rather than asserting it. Run one process that starts the workflow, a worker that runs just long enough to complete the first step, then a second, independent worker that resumes the same workflow:

import asyncio, os
from datetime import timedelta
from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker

SIDE_EFFECT_FILE = "side_effect.txt"

@activity.defn
async def step_one(order_id: str) -> str:
    with open(SIDE_EFFECT_FILE, "a") as f:
        f.write(f"step_one ran for {order_id}\n")
    return "step-one-done"

@activity.defn
async def step_two(order_id: str) -> str:
    with open(SIDE_EFFECT_FILE, "a") as f:
        f.write(f"step_two ran for {order_id}\n")
    return "step-two-done"

@workflow.defn
class CrashSaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        r1 = await workflow.execute_activity(step_one, order_id, start_to_close_timeout=timedelta(seconds=10))
        await asyncio.sleep(3)  # a window to "crash" the worker in
        r2 = await workflow.execute_activity(step_two, order_id, start_to_close_timeout=timedelta(seconds=10))
        return f"{r1}+{r2}"

async def main():
    client = await Client.connect("localhost:7233")

    await client.start_workflow(CrashSaga.run, "order-3", id="saga-order-3", task_queue="crash-tq")
    print("workflow started")

    # "Worker A": runs briefly, completes step_one, then stops (simulated crash).
    async with Worker(client, task_queue="crash-tq", workflows=[CrashSaga], activities=[step_one, step_two]):
        await asyncio.sleep(1.5)
    print("worker A stopped (simulated crash)")
    print("side effects so far:", open(SIDE_EFFECT_FILE).read())

    # "Worker B": a completely fresh worker, resuming the SAME workflow from history.
    async with Worker(client, task_queue="crash-tq", workflows=[CrashSaga], activities=[step_one, step_two]):
        handle = client.get_workflow_handle("saga-order-3")
        result = await handle.result()
        print("RESULT after resume:", result)
    print("side effects after resume:", open(SIDE_EFFECT_FILE).read())

if __name__ == "__main__":
    if os.path.exists(SIDE_EFFECT_FILE):
        os.remove(SIDE_EFFECT_FILE)
    asyncio.run(main())

Expected: after "worker A" stops, the side-effect file shows step_one ran exactly once, and step_two hasn't run yet. After "worker B" resumes and the workflow completes, the file shows step_one still only once, plus step_two once — the completed step_one activity's result came back from Temporal's event history instantly on replay; it was never re-executed by the new worker, even though the new worker is a totally separate process that never saw the first one run. This is module 04's manual "read saga_state, skip finished steps" recovery, except you didn't write any of the resume logic.

4. Build a signal-driven, human-in-the-loop approval step

import asyncio
from temporalio import workflow
from temporalio.client import Client
from temporalio.worker import Worker

@workflow.defn
class ApprovalSaga:
    def __init__(self):
        self._approved = False

    @workflow.signal
    def approve(self):
        self._approved = True

    @workflow.run
    async def run(self, order_id: str) -> str:
        await workflow.wait_condition(lambda: self._approved)
        return f"{order_id}-approved"

async def main():
    client = await Client.connect("localhost:7233")
    async with Worker(client, task_queue="approval-tq", workflows=[ApprovalSaga], activities=[]):
        handle = await client.start_workflow(
            ApprovalSaga.run, "order-4", id="saga-order-4", task_queue="approval-tq"
        )
        await asyncio.sleep(1)          # stand-in for "a human takes their time"
        await handle.signal(ApprovalSaga.approve)
        result = await handle.result()
        print("RESULT:", result)

if __name__ == "__main__":
    asyncio.run(main())

Expected: RESULT: order-4-approved. The workflow was durably parked in wait_condition the whole time — try increasing the sleep to 30 seconds and confirm it still works exactly the same way; a signal-waiting workflow doesn't cost a held-open connection or thread the way a hand-rolled "poll a status column every few seconds" approach would.

5. Diagnose and fix: a workflow that behaves differently after every deploy

A team's workflow function calls datetime.now() directly inside the workflow to compute a discount that only applies before a cutoff time, and reads a random number to pick between two shipping carriers. It works in testing, but in production, some workflows that get replayed after a worker restart mysteriously flip which carrier they chose, or apply a discount they shouldn't have.

6. Clean up

# Ctrl+C the `temporal server start-dev` terminal — an in-memory dev server,
# nothing persists once it stops.

Independent challenge

No code given. Redesign module 04's independent challenge — "book a trip" (reserve a flight, reserve a hotel, charge the customer once) — as a durable workflow instead of a hand-rolled orchestrator. Specify: (1) which parts become activities and which stay as pure workflow orchestration logic, and justify each choice against the determinism rule; (2) how you'd model "the flight reservation service is down for maintenance for the next 10 minutes" using retry policies instead of a hand-written retry loop; (3) how you'd add a genuinely long-lived step — "wait up to 24 hours for the customer to confirm the itinerary by clicking a link in an email" — using a signal, and what happens to that wait if the worker process restarts twice during those 24 hours; (4) one paragraph arguing, using this module's escalation judgment, whether "book a trip" actually justifies a durable execution engine over module 04's hand-rolled orchestrator, or whether it's simple enough that the extra infrastructure isn't worth it yet.

Common mistakes & troubleshooting

• Calling real I/O, the real clock, or real randomness directly inside workflow code. As the diagnose-and-fix exercise showed, workflow code is replayed to reconstruct state, and non-deterministic calls can return different answers on replay than they did the first time, silently corrupting decisions. Push anything non-deterministic into an activity.
• Forgetting the if __name__ == "__main__": guard on an entrypoint script. Temporal's Python SDK re-imports the workflow-defining module inside a sandbox; unguarded module-level code (like a bare asyncio.run(main())) re-executes during that re-import and crashes with a confusing RuntimeError, as exercise 1 noted.
• Assuming the engine's automatic retries remove the need for idempotency. The engine retries a failed activity call automatically, but if that activity's side effect can partially succeed before raising, a retry can still repeat real-world work. Module 01's idempotency-key discipline still applies inside activities.
• Reaching for a durable execution engine for a two-step, seconds-long saga. As the independent challenge's final question emphasizes, this is an escalation decision like every other in this track — a short, simple saga (module 04's choreography case) doesn't need a new infrastructure dependency; a long-running, branchy, human-in-the-loop workflow does.
• Confusing "activities aren't replayed" with "activities can't fail." Activities still fail, time out, and get retried per their policy — "not replayed" means a successful activity's result is reused on replay, not that failures are hidden or impossible.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What does a durable execution engine's "event history" replace from module 04's hand-rolled orchestrator, and what does replaying that history actually do on a worker restart?
2. What's the difference between a workflow function and an activity function, and why must workflow code be deterministic while activity code doesn't have that restriction?
3. If an activity has a RetryPolicy with maximum_attempts=5 and fails on the first two attempts, what code do you have to write to make the third attempt happen?
4. What is a signal, and what kind of requirement from module 04's saga design does it solve that a hand-rolled saga_state table doesn't solve well?
5. Give one concrete reason a short, two-step saga might not be worth moving to a durable execution engine.

Further reading & sources

• Temporal: Core application concepts - the official overview of workflows, activities, and event history this module is built on.
• Temporal Python SDK documentation - the SDK used throughout this module's exercises.
• Temporal: Workflow determinism constraints - the exact rules behind the "no real time/randomness/I-O directly in workflow code" discipline in this module.
• AWS Step Functions: How it works - the same durable-orchestration idea on a different, JSON-state-machine-based platform.
• microservices.io: Saga pattern - the pattern this module's engine automates the mechanics of; read alongside module 04 for the full picture.

Next

09-capstone-project — you now hold the whole toolkit: per-operation consistency (SD 15), idempotency (01), locking and fencing (02), the 2PC trade-off (03), sagas (04), CQRS (05), event sourcing (06), consensus/coordination (07), and durable workflow orchestration (08). The capstone stops teaching new patterns and asks you to combine them: design and partially build a distributed order-processing system that uses idempotency keys to make requests safe, a saga to coordinate the multi-step transaction across services, and event sourcing to record the order's state history.