Module 02: Production Serving — Health, Load Balancing and Keepalive

Why this matters

This is the module where gRPC's central design decision — long-lived, multiplexed HTTP/2 connections — stops being an advantage and becomes the thing that breaks your deployment. A REST service behind a load balancer distributes fine, because every request opens (or reuses from a shared pool) a short connection and the balancer gets a decision point per request. gRPC opens one connection and sends thousands of requests down it. A connection-level load balancer therefore makes exactly one decision, and then every subsequent RPC goes to the same backend, forever.

The symptom is unmistakable once you know it: you scale to ten replicas, and one of them is at 100% CPU while nine sit idle. This module covers that problem and the rest of the operational surface — health checking, graceful shutdown, keepalives, and message limits — that separates a service that runs from a service that stays up.

Concepts

Why L4 load balancing silently fails for gRPC

HTTP/1.1 + L4 LB                     gRPC (HTTP/2) + L4 LB
─────────────────                    ──────────────────────
req 1 ──▶ conn A ──▶ backend 1       conn A ──▶ backend 1
req 2 ──▶ conn B ──▶ backend 2         ├─ req 1 ──▶ backend 1
req 3 ──▶ conn C ──▶ backend 3         ├─ req 2 ──▶ backend 1
                                       ├─ req 3 ──▶ backend 1
balanced ✓                             └─ …all of them ──▶ backend 1
                                     
                                     backends 2 and 3: idle ✗

An L4 (TCP/transport) balancer — Azure Load Balancer, AWS NLB, a Service of type LoadBalancer in Kubernetes, kube-proxy for a ClusterIP — balances connections. gRPC multiplexes many concurrent RPCs as HTTP/2 streams inside one connection, so after the initial placement there are no further connections to balance. (See also: Load balancing concepts for L4 vs L7 balancing in general.)

There are three real fixes:

1. L7 load balancing. Put something that speaks HTTP/2 and balances per-stream in the path: Envoy, Linkerd, Istio, NGINX with grpc_pass, Azure Application Gateway, or a Kubernetes Gateway API implementation. This is the standard answer, and a service mesh (learn/13-service-mesh) gives it to you transparently.
2. Client-side load balancing. The client resolves all backend addresses and balances across them itself, opening one subchannel per backend. gRPC has this built in — no proxy required.
3. MAX_CONNECTION_AGE. Force connections to be recycled periodically so the L4 balancer gets fresh placement decisions. A mitigation, not a fix — it converts permanent imbalance into periodic rebalancing.

Client-side load balancing in Python

# round_robin across every address the resolver returns.
# Without this, gRPC's default is pick_first: it connects to the first
# working address and sends everything there — the same imbalance again.
channel = grpc.insecure_channel(
    "dns:///orders.default.svc.cluster.local:50051",
    options=[("grpc.service_config",
              '{"loadBalancingConfig": [{"round_robin": {}}]}')],
)

Two things are load-bearing here. The dns:/// scheme (three slashes) tells gRPC to use the DNS resolver and keep re-resolving, so new backends are picked up; a plain host:port uses the default resolver and one address. And in Kubernetes this requires a headless Service (clusterIP: None), because a normal ClusterIP resolves to a single virtual IP — there is nothing for the client to balance across.

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  clusterIP: None          # headless: DNS returns every pod IP
  selector:
    app: orders
  ports:
    - port: 50051
      targetPort: 50051

The tradeoff of client-side balancing: every client now needs to know about every backend, which doesn't scale to very large fleets and puts balancing policy in your application. L7 proxies exist because that tradeoff is often worse than an extra hop.

The standard health checking protocol

gRPC defines a standard health service — grpc.health.v1.Health — so that load balancers, Kubernetes probes and service meshes have a uniform way to ask "are you ready?". Don't invent your own.

from grpc_health.v1 import health, health_pb2, health_pb2_grpc

health_servicer = health.HealthServicer()
health_pb2_grpc.add_HealthServicer_to_server(health_servicer, server)

# empty string "" is the conventional key for overall server health
health_servicer.set("", health_pb2.HealthCheckResponse.SERVING)
# per-service status is also supported:
health_servicer.set("shop.v1.OrderService", health_pb2.HealthCheckResponse.SERVING)

Health should reflect readiness to serve, not merely "the process is alive". If your service can't reach its database, it should report NOT_SERVING so the balancer stops sending it traffic — that's the whole point of the signal.

Kubernetes can probe this natively (1.24+), with no extra binary:

readinessProbe:
  grpc:
    port: 50051
    service: ""          # matches the key you set above
  initialDelaySeconds: 5
livenessProbe:
  grpc:
    port: 50051
  periodSeconds: 10

Remember module 01's warning: the health service must be exempt from your auth interceptor, or probes get UNAUTHENTICATED and the platform concludes your healthy service is broken.

Server reflection

Reflection lets tools discover your services and message shapes at runtime, which is what makes grpcurl usable without passing .proto files around:

from grpc_reflection.v1alpha import reflection

SERVICE_NAMES = (
    shop_pb2.DESCRIPTOR.services_by_name["OrderService"].full_name,
    health_pb2.DESCRIPTOR.services_by_name["Health"].full_name,
    reflection.SERVICE_NAME,
)
reflection.enable_server_reflection(SERVICE_NAMES, server)

Enable it in development and internal environments. On a public-facing endpoint it's an information disclosure — it publishes your entire API surface — so gate it behind an environment flag.

Graceful shutdown

A server killed abruptly drops in-flight RPCs, which clients see as UNAVAILABLE — indistinguishable from a real failure, and a common cause of error spikes during every deploy. The correct sequence is: stop advertising readiness, stop accepting new RPCs, let in-flight ones finish, then exit.

import signal

def shutdown(signum, frame):
    # 1. fail readiness first, so the LB removes us before we stop serving
    health_servicer.set("", health_pb2.HealthCheckResponse.NOT_SERVING)
    time.sleep(5)                       # let the LB notice (>= one probe interval)
    # 2. refuse new RPCs, give in-flight ones a grace period
    server.stop(grace=30).wait()

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)

The sleep between failing readiness and stopping is the part everyone omits. Without it you stop serving before the balancer has observed the readiness change, so it keeps routing to a socket that's already closing.

Keepalive: the setting that causes the most confusing outages

Idle HTTP/2 connections get silently dropped by NAT gateways, cloud load balancers and firewalls — typically after 4-5 minutes. The client doesn't notice, because nothing tells it; it discovers the connection is dead only when its next RPC fails with UNAVAILABLE. Keepalive pings prevent this by keeping the connection provably alive.

# client
channel = grpc.insecure_channel(target, options=[
    ("grpc.keepalive_time_ms", 30000),            # ping every 30s
    ("grpc.keepalive_timeout_ms", 10000),         # ping must be answered in 10s
    ("grpc.keepalive_permit_without_calls", 1),   # ping even when idle
    ("grpc.http2.max_pings_without_data", 0),     # 0 = unlimited
])

# server — MUST be at least as permissive as clients, or it will punish them
server = grpc.server(executor, options=[
    ("grpc.keepalive_time_ms", 30000),
    ("grpc.keepalive_timeout_ms", 10000),
    ("grpc.keepalive_permit_without_calls", 1),
    ("grpc.http2.min_ping_interval_without_data_ms", 10000),  # tolerate pings this often
    ("grpc.http2.max_ping_strikes", 0),                       # 0 = don't GOAWAY on "too many" pings
])

The trap: servers default to treating frequent pings as abuse and responding with GOAWAY and ENHANCE_YOUR_CALM. If you tune the client to ping every 30s but leave the server's min_ping_interval_without_data_ms at its default (5 minutes), the server will kill the connections of a correctly-configured client. Client and server keepalive settings must be designed together.

Message size and concurrency limits

server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=16),
    options=[
        ("grpc.max_receive_message_length", 16 * 1024 * 1024),   # default is 4 MB
        ("grpc.max_send_message_length", 16 * 1024 * 1024),
        ("grpc.max_concurrent_streams", 1000),
        ("grpc.max_connection_age_ms", 300000),      # recycle for L4 rebalancing
        ("grpc.max_connection_age_grace_ms", 30000),
    ],
)

The 4 MB default receive limit is the single most common "it worked in testing" failure — the payload that exceeds it fails with RESOURCE_EXHAUSTED, and the limit is enforced independently on each side, so raising it on the server alone doesn't help a client receiving a large response. If you're hitting it, prefer server streaming (track 11, module 02) over raising the ceiling: chunking is the actual fix, a bigger limit just moves the wall.

Note also that Python's grpc.server uses a thread pool — max_workers caps how many RPCs execute concurrently. If handlers do blocking I/O and the pool is small, requests queue invisibly and latency climbs while CPU sits idle.

Command reference

Hands-on exercises

pip install grpcio grpcio-tools grpcio-health-checking grpcio-reflection

1. Reproduce the L4 imbalance

Run three server instances on ports 50051-50053, each printing its own port on every request. Put a plain TCP proxy in front of them (or simply have the client dial one address that resolves to all three). Send 300 RPCs over a single channel.

Expected: all 300 land on one backend. This is the failure the whole module is about — reproduce it before fixing it, so you recognize the shape.

2. Fix it with client-side round robin

Point the client at a target that resolves to all three addresses and add the round_robin service config. Send 300 RPCs again.

Expected: roughly 100 per backend. Then remove only the service_config option and re-run to confirm the default pick_first policy reproduces the imbalance — proving it's the policy, not the resolver, doing the work.

3. Add health checking and flip it

Register the health servicer, set "" to SERVING, and verify:

grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

Expected: {"status": "SERVING"}. Now add an admin RPC (or a signal handler) that sets NOT_SERVING, flip it, and re-run the check. Confirm the status changes while the process is still running and still answering other RPCs — readiness and liveness are genuinely different signals.

4. Enable reflection and browse your own API

grpcurl -plaintext localhost:50051 list
grpcurl -plaintext localhost:50051 describe shop.v1.OrderService
grpcurl -plaintext -d '{"id":"A-1"}' localhost:50051 shop.v1.OrderService/GetOrder

Expected: all three work with no .proto file passed. Then disable reflection and confirm list fails — that's the information you're choosing whether to expose.

5. Prove abrupt shutdown drops calls

Start a long-running RPC (a handler that sleeps 10s), then kill -9 the server mid-call. Observe the client error. Repeat with the graceful server.stop(grace=30).wait() handler and kill -TERM.

Expected: kill -9 gives the client UNAVAILABLE; the graceful path lets the in-flight call complete normally. This difference is your deploy-time error spike.

6. Trigger ENHANCE_YOUR_CALM on purpose

Configure the client to ping aggressively (keepalive_time_ms: 1000, permit_without_calls: 1) against a server left at defaults. Hold the channel open and idle.

Expected: the server sends GOAWAY with ENHANCE_YOUR_CALM and the client sees UNAVAILABLE — caused by your own keepalive configuration, not by any network problem. Then set the server's min_ping_interval_without_data_ms to 1000 and max_ping_strikes to 0 and confirm the connection survives.

7. Hit the 4 MB message limit

Return a response containing a 5 MB bytes field.

Expected: RESOURCE_EXHAUSTED naming the size and the limit. Fix it twice: once by raising grpc.max_receive_message_length on the client, and once by converting the RPC to server streaming and chunking at 256 KB. Note which fix you'd actually ship.

8. Diagnose and fix: the deploy that always spikes errors

A team runs 6 replicas behind a Kubernetes Service. Every rolling deploy produces a burst of UNAVAILABLE errors in clients for about 30 seconds, even though the new pods are healthy and the old pods are given a 30-second terminationGracePeriodSeconds. Their shutdown handler is server.stop(0) on SIGTERM, and their readiness probe is a TCP socket check on port 50051.

Independent challenge

No solution given. You operate a gRPC service with 20 replicas and ~500 client processes, each holding a long-lived channel. Traffic is uneven: two replicas run hot while others idle, and the imbalance persists for hours. You cannot deploy a service mesh (org constraint), and clients are owned by five different teams you'd need to coordinate with to change.

Design a remediation you can ship without client changes, then a second, better design assuming you can eventually change clients. For each, state what it costs: extra latency, extra infrastructure, connection churn, or coordination. Then decide what you'd measure to prove the imbalance is actually fixed rather than merely moved.

Common mistakes & troubleshooting

• Assuming an L4 load balancer distributes gRPC. It balances connections, and gRPC makes one. This is the default failure, not an edge case.
• Using pick_first by accident. It's gRPC's default policy; without an explicit round_robin service config, client-side balancing does nothing.
• A non-headless Kubernetes Service with client-side balancing. A ClusterIP resolves to one virtual IP, so there is nothing to balance across.
• Health that reports process liveness rather than readiness. If health stays SERVING while the database is unreachable, the balancer keeps feeding traffic to an instance that can only fail.
• Auth interceptor not exempting grpc.health.v1.Health. Probes fail, instances get pulled, and it looks like an infrastructure outage.
• server.stop(0) or kill -9 on deploy. Severs in-flight RPCs and produces an error spike every release.
• Failing readiness and stopping in the same instant. The balancer needs at least one probe interval to notice; without the wait, you stop serving before you stop receiving.
• Client keepalive tuned more aggressively than the server tolerates. The server responds GOAWAY/ENHANCE_YOUR_CALM and you get connection drops that look like network faults.
• Raising the message limit on one side only. It's enforced independently on send and receive, on both peers.
• A thread pool too small for blocking handlers. RPCs queue silently; latency rises while CPU looks fine.

Checkpoint quiz

Further reading & sources

• gRPC: Load balancing - the canonical explanation of why connection-level balancing fails for gRPC and what the alternatives cost.
• gRPC health checking protocol - the specification behind grpc.health.v1.Health.
• Kubernetes: Configure gRPC liveness probes - native gRPC probing, 1.24+.
• gRPC keepalive guide - every keepalive channel argument and the server-side ping policy that rejects them.
• gRPC service config - the JSON schema used for loadBalancingConfig and retry policy.

Next

03-observability-and-testing — a service that stays up still has to be debuggable: interceptor-based logging, metrics and tracing, plus how to test gRPC without mocking the framework out of existence.