Four Years with Protobuf: A Backend Engineer's Perspective on APIs
In the world of distributed backend systems, especially within the demanding environment of banking technology, how our services communicate is not just a technical detail—it's fundamental to reliability, performance, and our ability to evolve. For the past four years, I've leaned heavily on Protocol Buffers (Protobuf) for service-to-service communication, and it has fundamentally improved the way I build and maintain systems.
Before adopting Protobuf, much of our internal API landscape was dominated by JSON over REST. It's the lingua franca of the web, it's human-readable, and it's easy to get started with. But in the complex, interconnected web of core banking, digital banking, and open banking services, we started hitting its limitations. We were spending too much time on deserialization errors, debating data formats, and dealing with the performance overhead of verbose text-based messages.
The Contract Is King
The first major win with Protobuf was its schema-first approach. With Protobuf, you define your data structures—your messages—in a simple .proto file. This file becomes the canonical, language-agnostic contract for your data.
// Example of a simplified transaction message
syntax = "proto3";
package banking.services;
message Transaction {
string transaction_id = 1;
string account_id = 2;
double amount = 3;
string currency_code = 4;
int64 timestamp_ms = 5;
}
This contract is not just documentation; it's executable. The Protobuf compiler (protoc) uses this file to generate native data classes in your language of choice—in my case, primarily Java. For a Java and Spring-based backend, this means I get immutable, typed objects with builders and efficient serialization/deserialization logic for free.
This immediately eliminates a whole class of problems. There's no more ambiguity about field names, data types, or whether a field is optional. The contract is enforced at compile time. In a domain like banking, where a transactionAmount being misinterpreted as a string instead of a decimal can have real consequences, this rigidity is a feature, not a bug. It provides the kind of safety that JSON, even with OpenAPI specifications, often struggles to enforce as strictly across different service boundaries.
Performance in an Event-Driven World
Many of the systems I build rely on event-driven architecture. A single customer action might trigger a cascade of events that are processed by dozens of downstream microservices. When you're publishing millions of these events per hour, message size and parsing speed matter.
This is where Protobuf's binary format shines. A JSON payload for a transaction might look like this:
{
"transaction_id": "txn_12345abcde",
"account_id": "acc_67890fghij",
"amount": 150.75,
"currency_code": "USD"
}
The equivalent Protobuf message is a compact binary representation. It doesn't waste bytes on field names like "transaction_id". Instead, it uses the numeric tags (like 1, 2, 3) from the .proto file. The result is a message that is significantly smaller and dramatically faster to parse. For our JVM-based services, the generated code is highly optimized, putting less pressure on the garbage collector and improving overall throughput. This efficiency translates directly to lower network bandwidth, reduced storage costs for event logs (like in Kafka), and faster end-to-end processing times.
Evolving Systems Gracefully
Perhaps the most critical feature for any long-lived enterprise system is the ability to evolve. In a distributed architecture, you can't update every service simultaneously. You will always have different versions of services running in production at the same time.
Protobuf is designed with this reality in mind. Its rules for backward and forward compatibility are simple and effective:
- Adding a new field: You can add a new field with a new tag number to a message. Older services that don't know about the new field will simply ignore it when they deserialize the message.
- Renaming a field: You can't. But you can mark the old field as
reservedto prevent its tag number from being reused and add the new field with a different name and tag. - Removing a field: You should mark the field and its tag as
reservedto ensure future developers don't accidentally reuse the tag for a new field with a different meaning, which would break compatibility.
This has been invaluable. We can add a new fraud_score to our Transaction message and deploy the new fraud-detection service. The existing downstream services that handle ledger updates or send notifications will continue to function perfectly, simply ignoring the new field until they are updated to use it. This capability is the bedrock of safe, incremental change in a complex system.
The Not-So-Shiny Parts
Of course, Protobuf isn't a silver bullet. After four years, I'm well aware of the trade-offs.
First, the binary format is not human-readable. You can't just curl an endpoint and inspect the payload in your terminal. Debugging requires tooling that can decode the binary messages, which adds a layer of friction compared to the immediate transparency of JSON.
Second, there's a build-time dependency. You have to integrate the protoc compiler into your build process. For a Java project using Maven or Gradle, this is a solved problem with plugins, but it's an extra step and a potential point of failure that doesn't exist with simpler formats.
Finally, while Protobuf is excellent for well-structured data, it's less flexible than JSON for unstructured or highly dynamic data. Its type system is more constrained. This is generally a good thing for service APIs, but it's something to be aware of.
For the kind of robust, high-performance, and evolvable backend services required in banking technology, these trade-offs are well worth it. The compile-time safety, performance gains, and built-in support for schema evolution have made Protobuf an indispensable tool in my toolbox.