Horizontal Scaling Explained: Stateless Services, AutoscalingLesson 07: Horizontal Scaling
Roadmap
You are in System Design Interview Prep, Track 01, Core Fundamentals: lesson 7 of 19 (add more servers instead of a bigger one). It builds on Lesson 06: Vertical Scaling. Next comes Lesson 08: Latency.
These are the written notes for the lesson's video. Prefer paper? The same notes are a free, printable PDF: download SD-A-007.pdf.
Concept
Horizontal scaling (scaling out) runs many copies behind a load balancer; it only works when the service is stateless, and it's much harder for databases because they hold state.
Explanation
Scale out, not up
One machine always hits a ceiling. Scaling out runs many copies of a service side by side, with a load balancer in front spreading requests across them. Need more capacity? Add a copy. Traffic drops? Remove one. There's no hard ceiling, and losing one copy doesn't take the service down.
The rule: be stateless
Imagine a server keeps a user's session in its own memory. The next request lands on a different server, the session isn't there, and the user is logged out. The fix is to move state out of the servers: sessions to a shared store like Redis (or a signed token), files to object storage, data to the database.
Figure 1. A horizontally scaled service: identical, stateless copies behind a load balancer, with state in shared stores.
A real example: autoscaling on Kubernetes
Elastic FastAPI service. Twenty FastAPI pods run behind a Kubernetes Service. A Horizontal Pod Autoscaler (HPA) watches average CPU, targeting, say, 60%. When traffic spikes and CPU rises, it adds pods; when traffic falls, it removes them (with a cooldown so it doesn't flap). This ability to grow and shrink with demand is elasticity.
What changes when you scale out
| Effect | |
|---|---|
| Availability ↑ | One copy dies, the others keep serving (if health checks remove it quickly) |
| Cost flexible | Pay for what you run; scale down at night |
| Complexity ↑ | Load balancing, health checks, service discovery, config and deploys across many machines |
| Debugging harder | A request may touch any copy; you need central logs and tracing |
Why databases are harder to scale out
Figure 2. Stateless web servers share nothing. Databases share everything.
Read load can scale with read replicas (copies that follow the primary). Write load needs sharding: splitting data across machines by a key. Then queries that join across shards, and transactions that touch several shards, become slow or complicated. That's why teams scale the database up first, and shard only when they must.
Key terms
| Term | Meaning |
|---|---|
| Horizontal scaling | more machines (scale out) |
| Stateless | no user state kept between requests |
| Elasticity | grow and shrink with demand |
| HPA | Kubernetes autoscaler for pods |
| Read replica | a copy for reads |
| Sharding | splitting data across machines by a key |
Interview questions
Each question is tagged by kind: Basic (definitions), Scenario (apply it to a system) and Tricky (the follow-ups interviewers use to go deeper). Answer out loud before you open the answer.
What must be true before a service can scale horizontally? (Basic)
What a strong answer covers
It must be stateless: sessions, files and data live in shared stores, so any instance can serve any request and instances can be added or removed freely.
Why is scaling a database horizontally harder? (Basic)
What a strong answer covers
State must be replicated (kept in sync) or partitioned (sharded) while staying consistent; cross-shard joins and transactions become hard. Web servers share nothing.
What is elasticity? (Basic)
What a strong answer covers
Adding capacity when load rises and removing it when load falls, automatically, so you pay for what you use.
Users get logged out randomly after you add servers. Why, and how do you fix it? (Scenario)
What a strong answer covers
Sessions are stored in each server's memory, so a request on another server finds none. Move sessions to Redis or use signed tokens; sticky sessions only hide the problem.
You scale the API from 10 to 50 pods and the database falls over. What happened? (Scenario)
What a strong answer covers
Each pod opens its own connection pool, so connections jumped 5× past the database's limit. Use a connection pooler (e.g. PgBouncer), smaller per-pod pools, and cache reads.
Which metric would you autoscale a queue worker on? (Scenario)
What a strong answer covers
Queue depth or message age, not CPU: they measure the backlog directly. Workers waiting on I/O can have low CPU while the queue grows.
Are sticky sessions a good way to scale a stateful app? (Tricky)
What a strong answer covers
Only as a short-term patch: load becomes uneven, sessions are lost when a server dies, and deploys get harder. Externalise the state instead.
Can you scale writes with read replicas? (Tricky)
What a strong answer covers
No. Replicas copy the primary's writes, so every write still goes through the primary. Write scaling needs sharding or a database built to partition writes.
Common mistakes & troubleshooting
- Scaling out a service that keeps sessions in memory.
- Relying on sticky sessions as the fix (uneven load, lost sessions on failure).
- Autoscaling on CPU when the real bottleneck is the database.
- Forgetting the database connection limit when adding 50 app copies.
Free PDF notes
Download these notes as a printable PDF, with every diagram: SD-A-007.pdf. All Phase 1 notes in one file: Phase 1, Core Fundamentals (PDF).
Next
Continue with Lesson 08: Latency.