Module 00: Protobuf Schema Design and Evolution

Why this matters

Track 11 taught you protobuf syntax well enough to define a service and generate stubs. That works perfectly right up until the first time you need to change a .proto that other teams have already deployed against. At that point protobuf stops being a serialization format and becomes a contract with compatibility rules — rules that are mostly invisible, because breaking them usually doesn't raise an error. It silently misinterprets bytes.

The failure mode is specific and nasty: a client encodes field number 3 as an int64, the server has redefined field 3 as a string, and instead of crashing, the decoder reads whatever those bytes happen to mean. A REST API that breaks gives you a 400. A protobuf schema that breaks gives you corrupted data and a green dashboard. This module is the discipline that prevents that.

Concepts

The wire format is field numbers, not field names

This is the single fact everything else follows from. When protobuf encodes a message, it writes pairs of (field_number, wire_type) followed by the value. Field names are never transmitted — they exist only in the .proto and the generated code.

message User { string email = 2; }

on the wire:  [field 2, wire type 2 (length-delimited)] [len] [bytes...]
                      ^ this is all the decoder sees

Two direct consequences, both of which surprise people:

• Renaming a field is wire-compatible. string email = 2; → string email_address = 2; changes nothing on the wire. Old and new binaries interoperate fine. (It does break generated-code call sites and the protobuf-JSON mapping, so it isn't free — just not a wire break.)
• Reusing a field number is catastrophic, even with a different name. The decoder has no idea the meaning changed.

Wire types: what "compatible type change" actually means

There are only a handful of wire types, and compatibility is governed by them, not by how similar two types look in the .proto:

Rules that follow from this table:

• Safe: int32 ↔ int64 ↔ uint32 ↔ uint64 ↔ bool ↔ enum — all varint. Caveat: values that don't fit the narrower type are truncated, and a negative int32 read as int64 behaves as expected only because negative varints are always encoded in 10 bytes.
• Not safe, despite looking similar: int32 ↔ sint32. Both are varint, but sint32 uses zigzag encoding, so the bits mean different numbers. -1 as sint32 decodes as 1 if read as int32. Silent corruption.
• Safe: string ↔ bytes, provided the bytes are always valid UTF-8.
• Safe: a single field ↔ a repeated field of the same type, in the narrow sense that the decoder tolerates it (for length-delimited/packed types the last value wins on the singular side). Treat this as a last-resort migration, not a routine change.
• Never safe: anything that crosses wire types — int32 → string, float → int32, message → scalar.

reserved: the mechanism that makes deletion safe

When you delete a field, its number becomes free — and someone will reuse it a year later, having never seen the old schema. reserved makes the compiler stop them:

message User {
  reserved 3, 7, 9 to 11;
  reserved "legacy_role", "internal_flag";

  string id    = 1;
  string email = 2;
  string name  = 4;
}

Reserve both the numbers and the names: the numbers protect the wire format, the names protect anyone relying on protobuf-JSON or reflection. Reserving on deletion is not optional politeness — it is the only thing standing between you and a future silent corruption bug.

proto3 presence: the optional you probably need

In proto3, a plain scalar field has implicit presence: it is indistinguishable from its default. A server receiving int32 retry_count = 5; with value 0 cannot tell "the client sent 0" from "the client didn't set it at all" — both decode as 0, and 0 is not even written to the wire.

That matters enormously for partial updates. Marking a field optional (re-introduced for proto3 in protobuf 3.15) gives it explicit presence and a HasField check:

message UpdateUserRequest {
  string id = 1;
  optional string name  = 2;   // absent  != ""
  optional int32  age   = 3;   // absent  != 0
}

# server side
if request.HasField("name"):        # only valid because the field is `optional`
    user.name = request.name        # "" is now a legitimate value to set
if request.HasField("age"):
    user.age = request.age          # 0 is now a legitimate value to set

Without optional, "clear this user's name" is unexpressible — you cannot distinguish it from "don't touch the name." Adding optional to an existing field is wire-compatible (it does not change the encoding of a set field), so this is a safe retrofit.

Note that message-typed fields, oneof members, and optional fields all have explicit presence; repeated and map fields never do (empty and absent are the same thing for them, always).

FieldMask for partial updates at scale

optional per-field works fine for small messages. For a large resource, the convention is google.protobuf.FieldMask — the client states which paths it intends to modify:

import "google/protobuf/field_mask.proto";

message UpdateUserRequest {
  User user = 1;
  google.protobuf.FieldMask update_mask = 2;   // e.g. paths: ["name", "profile.bio"]
}

This is more explicit than presence checks, survives nested fields, and is what Google's own APIs use. The tradeoff is that you must implement mask application yourself — the runtime gives you the mask, not the merge.

oneof: the rules people get wrong

oneof gives you a tagged union — at most one member set at a time.

message Notification {
  string id = 1;
  oneof channel {
    EmailPayload email = 2;
    SmsPayload   sms   = 3;
    PushPayload  push  = 4;
  }
}

Evolution rules, which are narrower than they look:

• Safe: adding a new field to an existing oneof.
• Safe: moving a single existing field into a new oneof (it keeps its number, and a lone field in a oneof is wire-identical to a plain field).
• Not safe: moving multiple existing fields into one oneof. On the wire nothing stops an old client sending two of them; the new decoder keeps only the last, silently dropping data.
• Not safe: moving a field out of a oneof into another oneof, or splitting/merging oneofs.

Enums are open, and must have a zero value

proto3 requires the first enum value to be 0, and it should mean "unspecified" rather than a real state:

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;   // required zero; means "not set"
  ORDER_STATUS_PENDING     = 1;
  ORDER_STATUS_SHIPPED     = 2;
  ORDER_STATUS_DELIVERED   = 3;
}

Two reasons the UNSPECIFIED convention matters. First, because of implicit presence, an unset enum field decodes as 0 — so if 0 meant PENDING, every message that forgot to set status would claim to be pending. Second, proto3 enums are open: a value the receiver's schema doesn't know is preserved rather than rejected, so an old client receiving a newly-added ORDER_STATUS_CANCELLED = 4 sees the raw number 4, and any if status == SHIPPED ... else ... branch will quietly take the wrong path. Handle unknown enum values explicitly.

The prefixed naming (ORDER_STATUS_PENDING, not PENDING) isn't stylistic fussiness — enum value names share a C++-style scope with their enclosing namespace, so two enums in one package both declaring PENDING collide.

Package and directory layout: version in the path

The convention that makes breaking changes possible at all is versioning the package, so v1 and v2 can coexist during a migration:

proto/
  shop/
    v1/
      order.proto        package shop.v1;
      payment.proto      package shop.v1;
    v2/
      order.proto        package shop.v2;

syntax = "proto3";
package shop.v1;
option go_package = "github.com/you/shop/gen/shop/v1;shopv1";

A gRPC method's full path on the wire is /<package>.<Service>/<Method> — so shop.v1.OrderService/GetOrder and shop.v2.OrderService/GetOrder are genuinely different endpoints that can be served by the same process. That is the escape hatch for changes that cannot be made compatibly: don't break v1, publish v2 alongside it, migrate clients, then retire v1.

Well-known types: don't reinvent these

The design checklist

1. Field numbers 1-15 encode in one byte — spend them on your hottest, most-repeated fields, not on id out of habit.
2. Never reuse a number; always reserved on delete (numbers and names).
3. Use optional wherever "absent" and "zero" mean different things.
4. Every enum starts at 0 with an UNSPECIFIED member, prefixed.
5. Version the package (shop.v1), and mirror it in the directory path.
6. Wrap request/response per RPC (GetOrderRequest/GetOrderResponse) even when they'd be empty — you can add fields later without changing the method signature.
7. Prefer well-known types over hand-rolled timestamps and durations.

Command reference

protoc --decode_raw is the tool that makes the wire format concrete — it decodes bytes with no schema at all, showing you the field numbers and wire types that are genuinely present.

Hands-on exercises

python -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
pip install grpcio grpcio-tools protobuf
mkdir -p proto/shop/v1 gen

1. See that names don't exist on the wire

proto/shop/v1/order.proto:

syntax = "proto3";
package shop.v1;

message Order {
  string id = 1;
  int32 quantity = 2;
}

python -m grpc_tools.protoc -I proto --python_out=gen proto/shop/v1/order.proto
touch gen/__init__.py gen/shop/__init__.py gen/shop/v1/__init__.py

import sys; sys.path.insert(0, "gen")
from shop.v1 import order_pb2

o = order_pb2.Order(id="A-1", quantity=7)
raw = o.SerializeToString()
print(raw)          # b'\n\x03A-1\x10\x07'
open("payload.bin","wb").write(raw)

python -m grpc_tools.protoc --decode_raw < payload.bin

Expected output — note there are no field names anywhere:

1: "A-1"
2: 7

Now rename id to order_id in the .proto, regenerate, and decode the same payload.bin with the new schema. It still parses correctly into order_id. That is the "renaming is wire-compatible" rule, proven.

2. Prove that reusing a field number corrupts data

Serialize with the original schema (int32 quantity = 2), then change field 2 to string quantity and parse the old bytes with the new schema.

# with NEW schema (string quantity = 2), parse OLD bytes
from shop.v1 import order_pb2
o = order_pb2.Order()
o.ParseFromString(open("payload.bin","rb").read())
print(repr(o))
print(o.SerializeToString() == open("payload.bin","rb").read())

Expected — and this is worse than an error:

id: "A-1"
True

No exception, and quantity is simply gone. The old bytes encoded field 2 as a varint; the new schema expects field 2 to be length-delimited. Because the wire types don't match, the decoder can't treat it as field 2 at all, so it files it away as an unknown field — which is why it silently vanishes from the typed view yet still round-trips byte-for-byte on re-serialization.

Your server now sees an Order with no quantity and no indication anything was lost. No schema registry stopped you; in a distributed system these two versions are just two running processes.

3. Prove int32 ↔ sint32 is silent corruption

Define message T { int32 v = 1; }, serialize v = -1. Then redefine as sint32 v = 1;, regenerate, and parse those same bytes.

raw = A.T(v=-1).SerializeToString()
print(raw, len(raw))        # b'\x08\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01'  11
r = B.T(); r.ParseFromString(raw)
print(r.v)

Expected:

b'\x08\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01' 11
-2147483648

-1 became -2147483648, with no error at all. Both types are varints, so the wire types match perfectly and the decoder happily reads the value — it just applies zigzag decoding to bits that were never zigzag-encoded. (A negative int32 is always written as a full 10-byte varint, hence 11 bytes including the tag.)

Contrast with exercise 2: there a wire-type mismatch meant the field was quietly dropped; here the wire types agree, so you get a confident, wrong number instead. This is the most dangerous class of schema change.

4. Demonstrate implicit presence, then fix it with optional

message UpdateUserRequest {
  string id = 1;
  string name = 2;            // implicit presence
  optional string bio = 3;    // explicit presence
}

r = user_pb2.UpdateUserRequest(id="u1", name="", bio="")
print(r.SerializeToString())
print(r.HasField("bio"))
r.HasField("name")

Expected:

b'\n\x02u1\x1a\x00'
True
ValueError: Field p.UpdateUserRequest.name does not have presence.

Read those bytes carefully: \n\x02u1 is field 1 (id), \x1a\x00 is field 3 (bio) with length zero — name is absent from the wire entirely, even though you explicitly set it to "". The empty bio, by contrast, is transmitted, because optional gives it explicit presence.

Then write one sentence on why "clear the user's bio" is expressible but "clear the user's name" is not.

5. Reserve a deleted field and watch the compiler defend you

Delete quantity = 2 and add reserved 2; reserved "quantity";. Now try to add a new field reusing number 2:

message Order {
  reserved 2;
  reserved "quantity";
  string id = 1;
  string sku = 2;   // <-- deliberately reuse the reserved number
}

python -m grpc_tools.protoc -I proto --python_out=gen proto/shop/v1/order.proto

Expected — a hard compile error, with a helpful suggestion:

proto/shop/v1/order.proto:4:12: Field "sku" uses reserved number 2.
proto/shop/v1/order.proto:4:12: Suggested field numbers for shop.v1.Order: 3

Now prove the name half works too: drop reserved 2;, keep reserved "quantity";, and declare string quantity = 3;:

proto/shop/v1/order.proto:6:10: Field name "quantity" is reserved.

This is the whole point — the error you want, at build time, instead of corruption at runtime.

6. Add a field and prove forward compatibility

Serialize an Order with a new schema that has an extra field (string customer_id = 5;). Parse those bytes with the old generated code (regenerate into a separate directory first so you keep both). Confirm it parses fine and the unknown field is preserved on re-serialization:

old = old_pb2.Order(); old.ParseFromString(new_bytes)
assert old.SerializeToString() == new_bytes   # unknown fields round-trip intact

Expected: the assertion holds. proto3 has preserved unknown fields since 3.5, which is what makes "add a field" a safe, non-coordinated change.

7. Handle an unknown enum value

Define OrderStatus with values 0-3, serialize a message carrying status 4 (write the raw int — protobuf will accept it, enums are open), and parse it with a schema that only knows 0-3.

Expected: the value survives as 4. Now write the correct server-side branch: an explicit else that treats unknown status as "cannot handle," rather than an if/else that silently lumps it in with a known state.

8. Diagnose and fix: the compatible-looking change

Two teams share shop.v1.Order. The orders team needs quantity to support values above 2 billion, so they change int32 quantity = 2; to int64 quantity = 2;, reason that both are varints, deploy, and nothing breaks. Six months later they need price to be exact, so they change float price = 6; to double price = 6; by the same reasoning — and the payments service starts reading garbage prices.

Independent challenge

No solution given. You own shop.v1.Order, consumed by four services you don't control. You need to make all of these changes:

1. status is currently string (values "pending", "shipped") and should become a proper enum.
2. address is currently three flat string fields and should become a nested Address message.
3. discount_percent (an int32) must start supporting 0 as a deliberately set value, distinct from "not specified."
4. A new cancelled state must be added that old clients will never understand.

For each, decide: is it wire-compatible, achievable via an additive migration, or does it require shop.v2? Write the exact .proto diff for each change you can make compatibly, the migration sequence (what deploys in what order), and for anything requiring v2, what the coexistence period looks like. Then state which of the four is the most dangerous because it looks safe, and why.

Common mistakes & troubleshooting

• Reusing a field number after deleting a field. The corruption is silent. Always reserved the number and the name on deletion.
• Assuming similar types are compatible. int32→sint32 and float→double both look like widenings and are both wire breaks. Compatibility is decided by the wire-type table, not by intuition.
• Relying on default values to mean "unset." Without optional, 0, "" and false are indistinguishable from absent — which makes partial updates and "clear this field" impossible to express.
• Enums whose zero value is a real state. Every message that forgets to set the field silently claims that state. Reserve 0 for UNSPECIFIED.
• Treating proto3 enums as closed. Unknown values are preserved, not rejected — an if/else over known values will mis-handle them. Branch on unknown explicitly.
• No version in the package name. Without shop.v1, there is no way to ever ship a breaking change, because there's nowhere for v2 to live alongside it.
• Not enforcing any of this in CI. All of the above are exactly the kind of rule humans forget under deadline pressure — buf breaking against the main branch turns them into a failing build.

Checkpoint quiz

Further reading & sources

• Protocol Buffers: Proto3 language guide - the authoritative reference for syntax, presence, and enum semantics.
• Protocol Buffers: Encoding - the wire-type table and varint/zigzag details this module's compatibility rules derive from.
• Protocol Buffers: Field presence - the precise semantics of implicit vs explicit presence in proto3.
• Buf: Breaking change detection - the CI tooling that enforces these rules automatically.
• Google API Improvement Proposals (AIPs) - the versioning, FieldMask, and resource-naming conventions this module follows.

Next

01-security-tls-mtls-and-auth — with a schema that can survive change, module 01 makes the channel carrying it secure: TLS, mutual TLS, and per-call identity.