Module 01: Transformation and Normalization

Why this matters

Validation (module 00) answered "is this data acceptable?" Transformation answers a different question: "the data is acceptable, but is it in the canonical form my business logic and database actually want?" These are distinct steps, and a huge class of real-world bugs comes from skipping the second one. Two users sign up as Ada@Example.com and ada@example.com. Both are valid emails. But if you store them verbatim, you now have two accounts for one human, and your "email must be unique" constraint didn't save you — because to a raw string comparison, those are different strings. The fix isn't more validation; it's normalization: lowercase the email before you ever compare or store it.

Transformation shows up everywhere at the API edge. A query parameter arrives as the string "25" and your code wants the integer 25. A user pastes " ada@example.com " with stray whitespace. A phone number comes in as "(555) 123-4567" but your SMS provider wants +15551234567. A date arrives as "07/24/2026" from a US form but your database column is a proper date. In every case the raw input is acceptable but not yet usable — it needs casting, trimming, reformatting, or reshaping into one consistent representation. If you don't do this centrally at the edge, every downstream function ends up re-implementing it slightly differently, and the inconsistencies become bugs.

There's a security dimension too. Some transformation is defense. Sanitizing user-submitted strings — and, more importantly, using parameterized queries so those strings can never be interpreted as code — is how you stop SQL injection. Beginners often think "sanitization" means "strip out bad characters." That helps for display concerns, but the real protection against injection is never mixing untrusted data into a command string in the first place. This module makes the distinction concrete so you don't build a false sense of safety.

The order matters, and it's the mirror of a common instinct. You normalize before you validate the normalized form, and you validate before you trust. Trim and lowercase the email, then check uniqueness against the canonical value. Cast the string to an int, then bounds-check the int. Getting this pipeline in the right order is most of the skill.

Concepts

Type casting from string-typed inputs

Path and query parameters arrive as strings, always — they came off a URL. GET /items?page=2&limit=50 gives you "2" and "50" as text. FastAPI casts them for you when you annotate the types, and rejects un-castable values with a 422:

@app.get("/items")
async def list_items(page: int = 1, limit: int = 20):
    # page and limit are real ints here; "abc" would have been rejected as 422
    return {"page": page, "limit": limit}

This is transformation and validation fused: FastAPI parses the string, and if it can't produce an int, that's a validation error. Same for bool ("true"/"1"/"yes" → True), float, UUID, date, and datetime — all castable directly from the incoming string.

Normalization: one canonical form

Normalization collapses many equivalent inputs into a single canonical representation so comparisons and storage behave. The classic trio at the API edge:

• Lowercasing case-insensitive identifiers: emails, usernames (if case-insensitive), tags. Ada@Example.COM → ada@example.com.
• Trimming leading/trailing whitespace users didn't mean to type. " ada " → "ada". Copy-paste and mobile keyboards add this constantly.
• Canonicalizing formats: collapsing internal whitespace, standardizing separators, adding a default country code to a bare phone number.

   many equivalent inputs                    one canonical form
   ┌─────────────────────┐
   │ "Ada@Example.COM"   │ ─┐
   │ "  ada@example.com "  │ ├─► trim → lower ─► "ada@example.com"
   │ "ADA@EXAMPLE.COM"     │ ─┘                   (compare & store this)
   └─────────────────────┘

In Pydantic v2 you do this with a validator (they both validate and transform — same tool) or, for the common trim/lower cases, with a reusable annotated type:

from pydantic import BaseModel, EmailStr, field_validator

class Signup(BaseModel):
    email: EmailStr
    username: str

    @field_validator("email")
    @classmethod
    def normalize_email(cls, v: str) -> str:
        return v.strip().lower()      # trim THEN lower — order matters

    @field_validator("username")
    @classmethod
    def trim_username(cls, v: str) -> str:
        return v.strip()

Because a validator returns the transformed value, body.email downstream is already canonical. You never sprinkle .lower() across the codebase.

EmailStr already normalizes case for the domain — but not the whole story

A subtle real-world point: email domains are case-insensitive, but the local part (before the @) technically isn't, per spec. In practice almost everyone treats the whole address case-insensitively for login, and lowercasing the whole thing is the pragmatic, widely-used choice. Just be deliberate: decide your normalization rule once and apply it everywhere the email is compared.

Date and phone reformatting

Users and upstream systems send dates in many formats. Accept what they send, convert to one internal type (datetime.date) as early as possible, and let the rest of your code work with the real type — never juggle date strings.

from datetime import datetime, date
from pydantic import BaseModel, field_validator

class Booking(BaseModel):
    day: date        # Pydantic parses ISO "2026-07-24" for free

    @field_validator("day", mode="before")
    @classmethod
    def parse_us_format(cls, v):
        # accept US "MM/DD/YYYY" too, before Pydantic's own date parsing
        if isinstance(v, str) and "/" in v:
            return datetime.strptime(v, "%m/%d/%Y").date()
        return v

mode="before" runs your function on the raw input before Pydantic's own type coercion — the hook for accepting alternative input formats. Phone normalization follows the same shape: strip non-digits, add a country code if missing, store one canonical +E.164 string.

import re

@field_validator("phone")
@classmethod
def to_e164(cls, v: str) -> str:
    digits = re.sub(r"\D", "", v)          # keep only digits
    if len(digits) == 10:                  # bare US number
        digits = "1" + digits
    if len(digits) != 11:
        raise ValueError("phone must be a valid US number")
    return "+" + digits                    # +15551234567

Notice this is transformation and validation together: it reshapes the input and rejects what can't be reshaped.

Sanitization vs. real injection defense

"Sanitization" gets used loosely. Two different jobs hide under the word:

1. Rendering safety (output encoding): when you later put user data into HTML, escape it so <script> can't execute (XSS). This is an output concern — you encode at the moment of rendering, per context.
2. Injection safety for databases: the real defense against SQL injection is parameterized queries — you never build a SQL string by concatenating user input. The database driver keeps data and code strictly separate.

# WRONG — string interpolation lets input become SQL. Classic injection hole.
cursor.execute(f"SELECT * FROM users WHERE email = '{email}'")
# email = "x'; DROP TABLE users; --" is now catastrophic.

# RIGHT — parameterized: the driver binds `email` as data, never as code.
cursor.execute("SELECT * FROM users WHERE email = %s", (email,))

Trimming and lowercasing an email does not make it injection-safe — a lowercase string can still carry a SQL payload. Parameterization is the protection; normalization is about correctness, not injection defense. Keep those goals separate in your head so you don't build a false sense of security. (You'll go deep on injection in track 09; here the point is which technique actually does the job.)

The pipeline order

At the edge, run: cast → normalize → validate → use. Cast the string to the target type, normalize it to canonical form, validate the canonical value against your rules, then hand clean data to business logic. In Pydantic, mode="before" validators do casting/parsing of raw input, default (mode="after") validators normalize and validate the typed value, and Field constraints add declarative bounds — they compose into exactly this pipeline.

  raw "  25 "     ┌───────┐   ┌───────────┐   ┌──────────┐   clean
  from URL/body ─►│ CAST  │──►│ NORMALIZE │──►│ VALIDATE │──► int 25 → use
                  │(before│   │ trim/lower│   │  bounds  │
                  │  → type)   └───────────┘   │  & rules │
                  └───────┘    (after)         └──────────┘
     coercion         ▲            ▲                ▲
     failure → 422 ───┘            │        rule fail → 422
                          Field constraints + @field_validator

Command reference

mode="before" vs. default (mode="after"). A before validator sees the raw incoming value (often a str) before Pydantic coerces it to the declared type — use it to accept alternative formats or pre-clean input. An after validator sees the value after coercion (already the declared type) — use it to normalize and enforce domain rules. Order in the pipeline: before → coercion → after.

Reusable transforms with Annotated. When ten fields all need trimming, don't copy a validator ten times. Attach one:

from typing import Annotated
from pydantic import BaseModel, BeforeValidator

def _trim(v):
    return v.strip() if isinstance(v, str) else v

TrimmedStr = Annotated[str, BeforeValidator(_trim)]

class Profile(BaseModel):
    first_name: TrimmedStr
    last_name: TrimmedStr
    bio: TrimmedStr

Now TrimmedStr is a self-documenting, reusable transformation. This is the DRY way to normalize consistently across a large schema.

Casting failures return 422, not 500. If ?page=abc can't become an int, FastAPI produces a 422 validation error automatically. You don't try/except around the cast — the framework handles the failure as a client error, which is correct: bad input is the client's fault, not a server crash.

Hands-on exercises

Continue in the api-layer project from module 00.

1. Casting query parameters

@app.get("/search")
async def search(q: str, page: int = 1, per_page: int = 20, exact: bool = False):
    return {"q": q, "page": page, "per_page": per_page, "exact": exact}

Try GET /search?q=cats&page=2&per_page=50&exact=true and then GET /search?q=cats&page=abc.

Expected: the first returns real typed values (page is the int 2, exact is the bool True); the second returns 422 because "abc" can't be cast to int. You wrote no parsing code.

2. Normalize an email (trim then lowercase)

from pydantic import BaseModel, EmailStr, field_validator

class Signup(BaseModel):
    email: EmailStr

    @field_validator("email")
    @classmethod
    def normalize_email(cls, v: str) -> str:
        return v.strip().lower()

@app.post("/normalize-signup")
async def normalize_signup(body: Signup):
    return {"stored_as": body.email}

Send {"email": " Ada@Example.COM "}. Expected: stored_as is ada@example.com. Change the order to .lower().strip() and confirm it still works here — then reason about why order can matter when the transform is, say, "strip then validate length": stripping first changes what length you measure.

3. Prove why normalization matters for uniqueness

Keep a tiny in-memory set of seen emails and reject duplicates:

SEEN_EMAILS: set[str] = set()

@app.post("/register")
async def register(body: Signup):     # Signup already normalizes
    if body.email in SEEN_EMAILS:
        from fastapi import HTTPException
        raise HTTPException(status_code=409, detail="email already registered")
    SEEN_EMAILS.add(body.email)
    return {"registered": body.email}

Register Ada@Example.com, then try ada@EXAMPLE.com. Expected: the second is rejected 409 — because both normalized to the same canonical string. Now temporarily remove the normalizing validator and repeat: both succeed, creating a duplicate. That's the bug normalization prevents.

4. Phone number to E.164

import re
from pydantic import BaseModel, field_validator

class Contact(BaseModel):
    phone: str

    @field_validator("phone")
    @classmethod
    def to_e164(cls, v: str) -> str:
        digits = re.sub(r"\D", "", v)
        if len(digits) == 10:
            digits = "1" + digits
        if len(digits) != 11:
            raise ValueError("phone must be a valid US number")
        return "+" + digits

@app.post("/contact")
async def add_contact(body: Contact):
    return {"phone": body.phone}

Try "(555) 123-4567", "555.123.4567", "15551234567", and "123". Expected: the first three all normalize to +15551234567; the last returns 422. One canonical output from many input formats.

5. Accept a second date format with mode="before"

from datetime import datetime, date
from pydantic import BaseModel, field_validator

class Booking(BaseModel):
    day: date

    @field_validator("day", mode="before")
    @classmethod
    def parse_us(cls, v):
        if isinstance(v, str) and "/" in v:
            return datetime.strptime(v, "%m/%d/%Y").date()
        return v

@app.post("/book")
async def book(body: Booking):
    return {"day": body.day.isoformat()}

Try {"day": "2026-07-24"} and {"day": "07/24/2026"}. Expected: both yield "2026-07-24". Now try {"day": "24/07/2026"} — it raises because %m/%d/%Y can't parse month 24. Note how the failure is a clean 422.

6. Build a reusable TrimmedStr

from typing import Annotated
from pydantic import BaseModel, BeforeValidator

TrimmedStr = Annotated[str, BeforeValidator(lambda v: v.strip() if isinstance(v, str) else v)]

class Profile(BaseModel):
    first_name: TrimmedStr
    last_name: TrimmedStr

@app.post("/profile")
async def profile(body: Profile):
    return {"first": body.first_name, "last": body.last_name}

Send {"first_name": " Ada ", "last_name": " Lovelace "}. Expected: trimmed values. Confirm you only wrote the transform once but applied it to two fields — that's the DRY win.

7. See the injection hole, then close it (safely, in memory)

You won't wire a real database yet, but you can see the difference. Given email = "x' OR '1'='1", look at what an interpolated query string becomes:

email = "x' OR '1'='1"
bad_sql = f"SELECT * FROM users WHERE email = '{email}'"
print(bad_sql)
# SELECT * FROM users WHERE email = 'x' OR '1'='1'  -> matches EVERY row

Now write the parameterized form as a string and confirm the payload stays data:

good_sql = "SELECT * FROM users WHERE email = %s"
params = (email,)   # the driver binds this as a value; it can never become SQL
print(good_sql, params)

Expected takeaway: normalizing (.lower(), trimming) the malicious string would not have neutralized bad_sql — only parameterization does. Write that sentence down; it's the whole point.

8. Diagnose and fix

This endpoint is supposed to store a normalized, trimmed, lowercased tag and reject empty tags — but " " (spaces only) sneaks through as an empty tag, and "URGENT" is stored with different casing than "urgent". Find and fix the ordering bug.

from pydantic import BaseModel, Field, field_validator

class Tag(BaseModel):
    name: str = Field(min_length=1)

    @field_validator("name")
    @classmethod
    def clean(cls, v: str) -> str:
        v = v.lower()
        return v.strip()

class Tag(BaseModel):
    name: str

    @field_validator("name", mode="before")
    @classmethod
    def clean(cls, v):
        if not isinstance(v, str):
            return v
        v = v.strip().lower()
        if not v:
            raise ValueError("tag cannot be empty")
        return v

Independent challenge

No code given. Extend the POST /register endpoint you built to accept a country field that arrives inconsistently — users type "usa", "USA", "United States", "us" — and normalize all of them to the ISO 3166 two-letter code "US" before storage, rejecting anything you can't map. Then make the endpoint reuse the same normalization you applied to email in this module so that two registrations differing only in country casing or email casing are correctly detected as the same user. Reach back to module 00's idea of the three validation layers and state, for each field (email, phone, country), which layer each of your checks belongs to.

Common mistakes & troubleshooting

• Storing un-normalized identifiers. Emails/usernames stored with mixed case or stray whitespace defeat uniqueness constraints and cause "duplicate" accounts. Normalize before comparing or storing.
• Wrong pipeline order. Validating length before trimming (or lowercasing before validating a case-sensitive rule) gives wrong results. Decide cast→normalize→validate and place each step at the right mode.
• Thinking normalization prevents injection. Trimming/lowercasing is about correctness, not security. Parameterized queries are the injection defense; a lowercase string can still carry a payload.
• Manually parsing query params. Don't int(request.query_params["page"]) and try/except it — annotate the type and let FastAPI cast and return 422 on failure.
• Juggling date strings. Convert to datetime.date/datetime at the edge and pass the real type around; formatting back to a string is a presentation concern for the response only.
• Copy-pasting the same transform across many fields. Use an Annotated[...] reusable type (TrimmedStr) so normalization is defined once and stays consistent.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Why do path and query parameters always start as strings, and what does FastAPI do when a value can't be cast to the annotated type?
2. Two users register as Bob@Site.com and bob@site.com. What breaks if you don't normalize, and what's the one-line fix?
3. What's the difference between a mode="before" and a default (mode="after") field validator, and when do you reach for each?
4. Does lowercasing and trimming a user-submitted string protect you from SQL injection? What actually does?
5. In what order should you cast, normalize, and validate — and give one concrete bug that appears if you validate length before trimming.
6. You need to trim whitespace on twelve string fields. What's the DRY way to do it in Pydantic without twelve copies of the same validator?

Further reading & sources

• Pydantic — Validators - covers mode="before" vs mode="after" and how a validator both transforms and validates in one step.
• Pydantic — Custom types with Annotated - the reusable Annotated[str, BeforeValidator(...)] pattern behind TrimmedStr.
• FastAPI — Query Parameters and String Validations - how query/path strings are cast to typed values and constrained.
• OWASP — SQL Injection Prevention Cheat Sheet - why parameterized queries, not string sanitization, are the real injection defense.
• OWASP — Cross Site Scripting Prevention Cheat Sheet - the output-encoding side of "sanitization" and why it is a rendering concern.
• Wikipedia — E.164 - the international phone-number format your normalization canonicalizes toward.

Next

02-complex-validation-logic — single-field rules only get you so far; next you'll validate relationships between fields, handle conditional requirements, aggregate errors, and avoid leaking sensitive information in your error messages.