Module 05: Service Configuration and Environments

Why this matters

You have one image (module 00), injected with config at run time (module 01), built by CI (module 02), rolled out safely (module 03). The thing that makes that one image run correctly as dev, staging, and prod is the config you feed it — and managing that config across environments, injecting secrets at deploy time without leaking them, and controlling what's on independently of what's deployed, is its own discipline with its own failure modes.

Get config management wrong and the symptoms are among the most painful in backend engineering: a staging run that accidentally points at the production database and corrupts real data; a secret printed into a log or an error response and now compromised; a config value that's set in dev but forgotten in prod so the service half-works in a way no test caught; a feature that can only be turned off by an emergency redeploy at 2am. Every one of these is a config-management failure, not a code bug, and the deploy pipeline is where you either prevent or cause them.

Track 08 (modules 02-03) taught config as an app concern: pydantic-settings, typed and validated config, SecretStr, config sources and layering. Track 09 taught secrets management as a security concern. This module is the deployment view that ties them together: how config differs across environments while the code stays identical, how secrets get injected at deploy time from a secret store (not committed, not in the image), and how feature flags let you separate deploying code from releasing behavior. The secret-store infrastructure — Azure Key Vault, Kubernetes Secrets/External Secrets, managed identity — is learn/03-kubernetes and learn/16-identity-deep-dive; the backend engineer's decisions are here.

Concepts

One codebase, many environments: what differs and what must not

Factor I/X: the same build artifact runs in dev, staging, and prod, and environments differ only by config, never by code. That principle is only useful if you're disciplined about what "config" is. Config is everything that varies between deploys: connection strings, hostnames, credentials, log levels, resource limits, feature toggles, external API endpoints. Code — including business logic, validation rules, and the set of features that exist — is the same everywhere.

                 ┌──────────────────────────┐
                 │  ONE image :git-<sha>     │  (no config, no secrets baked in)
                 └────────────┬─────────────┘
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
  ┌───────────┐         ┌───────────┐         ┌───────────┐
  │   DEV     │         │  STAGING  │         │   PROD    │   ← differ ONLY by
  ├───────────┤         ├───────────┤         ├───────────┤     injected config
  │ APP_ENV   │         │ APP_ENV   │         │ APP_ENV   │
  │ DB_URL    │         │ DB_URL    │         │ DB_URL    │
  │ secrets ◄─┼─store   │ secrets ◄─┼─store   │ secrets ◄─┼─store  (separate scopes)
  │ flags     │         │ flags     │         │ flags     │
  └───────────┘         └───────────┘         └───────────┘

Two properties this buys you, both from track 08's config track:

• Dev/prod parity (factor X): the environments should be as similar as possible — same backing service types (Postgres in dev, Postgres in prod, not SQLite-then-Postgres), same dependencies, small config surface. The more they diverge, the more "works in staging, breaks in prod" bugs you get. Config should express values that differ (URLs, sizes), not structural differences.
• The environment is named in config, not branched in code. An APP_ENV (dev/staging/prod) value tells the app which environment it's in, and the app reads its other config accordingly — but you should be deeply suspicious of if app_env == "prod": branches in business logic. Those are code paths that only run in prod, i.e. untested-until-prod code — the opposite of parity. Use config values to parameterize behavior, not if env branches to change it.

The validation angle matters at deploy time specifically: with pydantic-settings, a missing or malformed required config value should make the app fail to start (fail-fast, track 08 module 00) rather than boot half-configured. A deploy that would run with a broken config should crash on startup and fail the rollout's readiness check — loudly, immediately — not limp along serving errors.

The config spectrum: hardcoded → env vars → config service, and secrets apart

Not all config is the same, and the mechanism should match the kind:

• Non-secret, per-environment config (log level, feature flags, pool sizes, external URLs): env vars are the 12-factor default and are plenty for a modest config surface. When the surface grows to hundreds of settings, a config service / config file mounted per environment (a ConfigMap in Kubernetes, an App Configuration store) becomes reasonable — the pragmatic bend on factor III that track 08 module 10 flagged. Either way it's injected, not baked in.
• Secrets (DB passwords, API keys, signing keys, tokens) are config too, but they get stricter handling: never in the repo, never in the image, never in logs or error responses, and ideally sourced from a dedicated secret store rather than plain env vars. This is where track 09's discipline and SecretStr (track 08 module 03) come in — a SecretStr won't print its value when a config object is logged or an error is rendered, closing the most common leak path.

The litmus test remains track 08 module 10's: could you open-source this repo right now without leaking a credential? Secrets pass it by never being in the repo in the first place — they're injected at deploy time from the store.

Injecting secrets at deploy time

The safe path for a secret, end to end, keeps it out of everything durable and inspectable except the secret store:

1. The secret lives in a secret store (Azure Key Vault, Kubernetes Secret backed by a store, HashiCorp Vault) — not in Git, not in the CI config, not in the image.
2. At deploy/run time, the platform injects it into the container as an env var or a mounted file — the release step of build/release/run. The image (the build) never contained it.
3. The app reads it via pydantic-settings into a SecretStr, so it's typed and won't leak through logs or error rendering.
4. Rotation happens in the store; a redeploy (or a secret-refresh mechanism) picks up the new value — the code never changes.

The anti-patterns this rule set eliminates: secrets committed to Git (leaked forever in history — the reason for secret scanning in module 02), secrets baked into the image (anyone who pulls the image gets them), secrets in CI logs (echoed by a careless step), and secrets in a plaintext .env checked into the repo (why .env is in .gitignore and .dockerignore). The how — Key Vault, managed identity so the app authenticates to the store without a bootstrap secret, External Secrets Operator syncing into the cluster — is learn/16-identity-deep-dive and learn/03-kubernetes; the rule you own is store it, inject at release, type it as a secret, rotate in the store.

Feature flags: decoupling deploy from release

The deployment strategies in module 03 control how code reaches production. Feature flags control whether a feature is on, independently — and that separation is powerful. A feature flag is a config-driven runtime switch (if flags.new_checkout_enabled:) that lets you:

• Deploy code dark. Merge and deploy an unfinished or risky feature turned off, so it's in production but inert. This lets you deploy continuously (small, frequent, low-risk deploys) without waiting for a feature to be "done" — and it makes the risky expand/contract migrations of module 04 easier to sequence (the new read path can be flag-gated).
• Release without deploying. Turn a feature on by flipping a flag (a config change, not a code deploy), and turn it off the same way if it misbehaves — an instant kill switch that needs no rollback and no 2am redeploy.
• Gradual/targeted rollout. Enable a feature for 5% of users, or internal users only, or one tenant — a canary at the feature level rather than the deploy level.

The cost is real and worth naming: flags are config that lives in code paths, so every flag is a branch that must be tested in both states, and stale flags accumulate into an unreadable mess of dead conditionals. The discipline is to treat flags as temporary by default — a flag for a rollout should be removed once the feature is fully on and stable, exactly like the contract step removes the old column in module 04. Long-lived flags (true configuration, like per-tenant entitlements) are legitimate but should be recognized as permanent config, not leftover rollout scaffolding. Full flag platforms (LaunchDarkly, Unleng, targeting rules, experimentation) are a specialization; the backend engineer's core skill is knowing why to reach for a flag and to clean them up.

Config as part of the deploy, and validating it

Pulling it together at the pipeline level: config is a release input, and treating it with the same rigor as code prevents a whole class of incidents.

• Config is versioned and reviewed (per environment), just like code — a ConfigMap/values file in Git (minus secrets), changed by PR, so a prod config change is auditable and reversible. This is the GitOps idea; the tooling is learn/10-cicd-and-gitops.
• Config is validated at startup, failing fast. The app defines its config schema (pydantic-settings); a deploy with a missing required var or a malformed URL crashes on startup and fails the rollout's readiness check, rather than running misconfigured. This turns "prod is subtly broken because someone forgot to set REDIS_URL" into "the deploy never went live."
• Environment isolation is enforced, not hoped for. Staging config must not be able to reach prod's database. The most reliable enforcement is that staging simply doesn't have prod's credentials (separate secret stores/scopes per environment) — so a copy-paste mistake can't point staging at prod, because the connection string isn't available to it.

The recurring theme: config errors should fail early and loud (at deploy time, on startup) rather than late and quiet (in production, subtly), and the way you get that is typed validation + environment-scoped secrets + reviewed, versioned config.

Command reference

Typed, validated config that fails fast and won't leak secrets (track 08 modules 02-03, restated for deployment):

from pydantic import SecretStr, PostgresDsn, field_validator
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    app_env: str = "dev"                    # names the environment; does NOT branch logic
    database_url: PostgresDsn               # required — missing → startup crash (fail fast)
    redis_url: str
    log_level: str = "INFO"
    stripe_key: SecretStr                   # secret: won't print in logs/errors
    new_checkout_enabled: bool = False      # feature flag, config-driven

    @field_validator("app_env")
    @classmethod
    def known_env(cls, v: str) -> str:
        if v not in {"dev", "staging", "prod"}:
            raise ValueError(f"unknown APP_ENV: {v}")   # bad config fails the deploy, loudly
        return v

settings = Settings()   # a missing/invalid required value raises here → the container won't start

A feature flag decoupling release from deploy:

@app.post("/checkout")
async def checkout(cart: Cart):
    if settings.new_checkout_enabled:       # flip via config — no redeploy to turn on/off
        return await new_checkout(cart)     # deployed dark until the flag is flipped
    return await legacy_checkout(cart)
# TODO(flag): remove new_checkout_enabled + legacy_checkout once new path is fully on & stable

Injecting a secret at deploy time (Kubernetes shape; the store/identity mechanics are learn/03 + learn/16) — note the image never contains the secret:

# Secret lives in the store; injected as env at release. Image (build) has none of this.
envFrom:
  - configMapRef: { name: app-config }     # non-secret, per-env config
  - secretRef:    { name: app-secrets }    # secrets, injected at deploy — never in Git/image

Hands-on exercises

Use the containerized service from earlier modules with Postgres/Redis in Docker.

1. One image, three environment configs

Run the same image three times with dev/staging/prod config sets (different APP_ENV, DATABASE_URL, LOG_LEVEL). Add an /info endpoint returning app_env and the log level. Confirm behavior differs by config alone, no rebuild — and that no config is in the image (docker history).

2. Fail fast on missing config

Remove a required config var (e.g. DATABASE_URL) and start the container. Confirm it crashes on startup with a clear error rather than booting broken. Add an app_env validator that rejects an unknown value and prove a typo'd APP_ENV fails the start. Connect this to the rollout readiness check from module 03.

3. Type a secret so it can't leak

Store an API key as a plain str in Settings, then log the settings object and watch the key appear in the log. Change it to SecretStr and confirm it now prints as **********. Trigger an error that renders config and confirm the secret doesn't leak there either. This is track 08 module 03 / track 09, at deploy scope.

4. Keep secrets out of the image and repo

Confirm .env is in both .gitignore and .dockerignore. Grep the repo and image for anything secret-shaped (module 02's secret scan). Inject the secret at run via --env-file/secretRef instead, and verify the running app has it while the image does not.

5. Deploy code dark behind a flag

Add a feature flag defaulting to false, wrapping a new endpoint/behavior. Deploy it (the code is in prod, inert). Flip the flag on via config without redeploying (restart with the env var flipped, or a live flag source) and confirm the feature activates. Flip it off again as a kill switch.

6. Enforce environment isolation

Give staging and prod separate secret sets so staging simply doesn't have prod's DATABASE_URL. Try to "accidentally" point staging at prod and confirm it can't — the credential isn't available to it. Write one sentence on why "don't have the secret" beats "remember not to use it."

7. Avoid the if env trap

Find or write an if app_env == "prod": branch that changes business logic. Refactor it so the differing behavior is driven by a config value instead (if settings.rate_limit_enabled: set per env). Explain how this restores dev/prod parity and testability.

8. Diagnose and fix

A service has three recurring incidents: (1) last week a staging test run wiped a production table; (2) a support engineer found the Stripe key printed in the error logs; (3) turning off a broken feature required an emergency redeploy that took 40 minutes. Its config looks like this:

DATABASE_URL = "postgresql://prod-db/app"          # (a) hardcoded prod URL, same for all envs
STRIPE_KEY = "sk_live_51H..."                       # (b) hardcoded live secret
class Settings(BaseSettings):
    stripe_key: str                                 # (c) plain str
    # no APP_ENV, no validation
def checkout():
    if NEW_FLOW:  # (d) module-level constant, only changeable by editing code + redeploy
        ...

Independent challenge

No code given. Take the service you've carried through this track and give it a production-grade configuration setup across dev/staging/prod, then prove each property. (1) One image, many environments: run the same module 00 image as all three environments differing only by injected config, with no config baked into the image and no if env branches in business logic (parameterize with config values instead). (2) Secrets, safely: source every secret from a secret store, inject it at deploy time, type it as SecretStr so it can't leak into logs or errors, and confirm the repo and image contain no credentials — building on track 08's module 03 (Config sources and secrets) and track 09's secrets management for the security discipline. (3) Fail fast on bad config: make a missing/invalid required value crash the container on startup so it fails the rollout readiness check rather than running misconfigured. (4) Decouple deploy from release: put a risky new behavior behind a feature flag, deploy it dark, and demonstrate flipping it on and off via config with no redeploy — then write the note that says when you'll remove the flag. Enforce environment isolation by giving each environment its own secret scope, and point to learn/16-identity-deep-dive and learn/03-kubernetes for the secret-store and managed-identity mechanics you'd use in production.

Common mistakes & troubleshooting

• Hardcoded per-env config (esp. a prod URL) used everywhere. Staging hits prod's DB and corrupts data. Per-env config from the environment; separate secret scopes so staging can't reach prod.
• Secrets in the repo or image. Leaked forever / shipped to anyone who pulls the image. Store in a secret store, inject at deploy, keep .env out of Git and the image.
• Secrets as plain str. Printed in logs and error responses. Use SecretStr.
• if app_env == "prod" branches in logic. Code paths that only run in prod = untested-until-prod, breaks parity. Parameterize behavior with config values.
• Config not validated. A missing var boots a half-working service. Fail fast on startup with typed pydantic-settings so a bad config fails the rollout.
• Feature toggles as code constants. Turning a feature off needs an emergency redeploy. Use config-driven feature flags — flip without deploying.
• Flags that never get removed. Accumulate into dead, untested conditionals. Treat rollout flags as temporary; remove once fully on (like module 04's contract step).
• Environment isolation by convention. "Remember not to" fails. Enforce by not granting staging prod's credentials.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What exactly is allowed to differ between environments and what must stay identical, and why are if app_env == "prod" branches in business logic a parity/testability problem?
2. Trace a secret's safe path from store to app. Name every place it must not appear and what SecretStr specifically prevents.
3. Why should a missing or malformed required config value crash the container on startup, and how does that interact with the module 03 rollout?
4. What do feature flags decouple, and give one concrete win for deploying code "dark" and one for "releasing without deploying."
5. What's the downside of feature flags, and what discipline keeps it from becoming a mess (and which module's pattern is it analogous to)?
6. "Staging must never touch prod's database." Why is enforcing this by not granting the credential stronger than a rule saying "don't use prod in staging"?

Further reading & sources

• The Twelve-Factor App: Config (factor III) - Why config lives in the environment and the same build runs unchanged across dev/staging/prod.
• The Twelve-Factor App: Dev/prod parity (factor X) - Keeping environments as similar as possible, the principle behind avoiding if env branches.
• Kubernetes: Secrets - How a platform injects secrets into a container at deploy time without baking them into the image.
• Kubernetes: ConfigMaps - Injecting non-secret per-environment config, the ConfigMap side of the config spectrum.
• Martin Fowler: Feature Toggles (aka Feature Flags) - The definitive taxonomy of flags and the discipline of removing them, decoupling deploy from release.
• LaunchDarkly: What are feature flags? - A practical overview of flag-driven dark launches, kill switches, and targeted rollouts.

Next

06-health-checks-readiness-and-scaling-signals — your service is well-configured and its secrets are safe. Now we make it manageable by the platform: liveness and readiness probes from the app's side (building on track 08's graceful shutdown), what the app must expose so a scheduler knows when it's ready for traffic and when to restart it, and the metrics an autoscaler needs to scale it.