Making Dynamic Protobuf Fast in Go

Most Go Protobuf services get to cheat: their schemas are known at build time. protoc-gen-go turns those schemas into concrete message types, accessor methods, and runtime metadata that the protobuf runtime can use efficiently.

FauxRPC does not get that luxury. It loads user-provided schemas at runtime so it can mock arbitrary gRPC, gRPC-Web, and Connect services without asking users to install protoc, Buf, or a Go toolchain first. Without generated Go types for each schema, it has to parse and inspect payloads dynamically. This is also the case for tools like buf, grpcurl, kreya, postman, and more.

That flexibility pushed the request path toward Go’s standard dynamicpb package. dynamicpb is flexible, but it pays for that flexibility with extra allocations, descriptor lookups, and reflection-heavy access paths.

Buf introduced hyperpb, a dynamic Protobuf parser that compiles descriptors into optimized parser bytecode at runtime. I wanted to see whether it could make FauxRPC’s read path faster without giving up runtime-loaded schemas.

The short version: yes, with caveats. hyperpb is read-only, so FauxRPC still uses dynamicpb to build mock responses. But for parsing incoming requests, the performance difference was large enough to be worth the split.

Reflection-Based dynamicpb

Go’s standard dynamicpb package takes a protoreflect.MessageDescriptor and constructs a dynamic message representation at runtime.

Since message layouts are only known at runtime, dynamicpb must route field access through descriptors and generic message representations rather than generated message types and runtime metadata. This introduces additional indirection and dynamic dispatch. The cost mostly shows up in two places:

1. It allocates a lot. Nested messages, repeated fields, map entries, and interface values turn into a pile of heap objects. This causes severe garbage collector (GC) pressure.
2. It chases pointers. Once the decoded message is spread across many small objects, the CPU spends more time bouncing around memory instead of reading predictable, contiguous data.

Bytecode Compilation with hyperpb

Buf’s hyperpb library takes a different approach. It compiles the message descriptor into dedicated, optimized table-driven parser bytecode at runtime.

The parser avoids Go’s reflection-heavy dynamic message construction in the unmarshalling hot path, while still exposing the parsed result through the standard protoreflect APIs. In many cases, it can parse dynamic payloads at speeds close to, or even faster than, generated Go protobuf messages.

Pre-Compiling at Runtime

Because hyperpb uses a custom VM under the hood, it requires a compilation phase before you can parse any payloads. Similar to compiling a regular expression with Go’s regexp.Compile, you must compile the schema definition at runtime. hyperpb.CompileFileDescriptorSet compiles a specific message type out of a FileDescriptorSet, while hyperpb.CompileMessageDescriptor compiles an already-resolved message descriptor:

// Done once at startup/initialization
hyperMsgType := hyperpb.CompileMessageDescriptor(messageDesc)

This compilation cost is paid once per message type, so the compiled types should be cached and reused.

Moving Allocation Out of the Hot Path

The real story is not just speed. It is allocation behavior.

hyperpb provides a reusable, pre-allocated memory arena pool through hyperpb.Shared. Pairing bytecode parsing with a reusable memory arena allows you to recycle memory buffers across multiple requests. This removes most of the per-message heap churn for read-only pipelines:

shared := new(hyperpb.Shared) // Instantiated once per goroutine/worker

for _, payload := range incoming {
    // Reuses the underlying pre-allocated memory arena
    msg := shared.NewMessage(mType)
    _ = proto.Unmarshal(payload, msg)

    route(msg)    // Note: Must be handled synchronously and complete before Free()
    shared.Free() // Recycles the arena back to the pool
}

Dynamic Reflection in Practice

To see how dynamicpb and hyperpb compare in code, we can use the classic ConnectRPC/Buf Eliza service schema.

A complete set of runnable examples is available in the dynamic-protobuf-in-go/go directory. It uses buf to compile the Protobuf definitions into a binary descriptor set, which is then loaded at runtime to perform dynamic serialization and reflection.

1. Compiling Protobuf Descriptors with Buf

Before using dynamic messages, we must compile the .proto schema into a FileDescriptorSet (a serialized binary image of the schemas). Using the Buf CLI, this is done with a single command:

buf build -o eliza.binpb

2. Loading Descriptors at Runtime

In Go, we read this descriptor set, unmarshal it into a descriptorpb.FileDescriptorSet, and load it into a protoregistry.Files registry:

	// Read compiled schema descriptors
	descriptorBytes, err := os.ReadFile("eliza.binpb")
	if err != nil {
		log.Fatalf("failed to read descriptor file (did you run 'buf build -o eliza.binpb'?): %v", err)
	}

	var fds descriptorpb.FileDescriptorSet
	if err := proto.Unmarshal(descriptorBytes, &fds); err != nil {
		log.Fatalf("failed to unmarshal file descriptor set: %v", err)
	}

	// Register files
	registry, err := protodesc.NewFiles(&fds)
	if err != nil {
		log.Fatalf("failed to create protodesc registry: %v", err)
	}
	

Once registered, we can look up the message descriptor by its full name and locate individual fields dynamically:

	// Retrieve message descriptor for Eliza's SayRequest
	sayRequestName := protoreflect.FullName("connectrpc.eliza.v1.SayRequest")
	desc, err := registry.FindDescriptorByName(sayRequestName)
	if err != nil {
		log.Fatalf("failed to find descriptor for %s: %v", sayRequestName, err)
	}

	sayRequestDesc, ok := desc.(protoreflect.MessageDescriptor)
	if !ok {
		log.Fatalf("descriptor for %s is not a message descriptor", sayRequestName)
	}

	sentenceField := sayRequestDesc.Fields().ByName("sentence")
	if sentenceField == nil {
		log.Fatalf("failed to find 'sentence' field in %s", sayRequestName)
	}
	

3. Dynamic Access with dynamicpb

Standard dynamicpb creates dynamic messages that support both reading and writing field values. The standard protoreflect interface is used for field access:

	// Create request message dynamically
	dynMsg := dynamicpb.NewMessage(sayRequestDesc)

	// Set field dynamically using the reflection interface
	dynMsg.ProtoReflect().Set(sentenceField, protoreflect.ValueOfString("Hello Eliza, how are you?"))

	// Marshal the message to binary wire format
	wireBytes, err := proto.Marshal(dynMsg)
	if err != nil {
		log.Fatalf("dynamicpb: failed to marshal message: %v", err)
	}
	fmt.Printf("Serialized bytes: %x\n", wireBytes)

	// Unmarshal wire format back into a new dynamicpb message
	dynMsg2 := dynamicpb.NewMessage(sayRequestDesc)
	if err := proto.Unmarshal(wireBytes, dynMsg2); err != nil {
		log.Fatalf("dynamicpb: failed to unmarshal message: %v", err)
	}

	// Get field dynamically
	val := dynMsg2.ProtoReflect().Get(sentenceField)
	fmt.Printf("Decoded message: %s\n", val.String())
	

4. High-Performance Read-Only Access with hyperpb

Because hyperpb is built for high-performance ingestion and routing, it only supports read-only access. Message descriptors must be compiled into optimized parser bytecode, and any attempt to write or mutate a message will panic:

	// Compile the descriptor into hyperpb optimized message type
	// Note: You should compile descriptors once at startup or pool them, not per request.
	hyperMsgType := hyperpb.CompileMessageDescriptor(sayRequestDesc)

	// Instantiate hyperpb message
	hyperMsg := hyperpb.NewMessage(hyperMsgType)

	// Unmarshal wire bytes into it (hyperpb parses without Go reflection overhead)
	if err := proto.Unmarshal(wireBytes, hyperMsg); err != nil {
		log.Fatalf("hyperpb: failed to unmarshal message: %v", err)
	}

	// Get field dynamically from hyperpb message using standard protoreflect API
	hyperVal := hyperMsg.ProtoReflect().Get(sentenceField)
	fmt.Printf("Decoded message: %s\n", hyperVal.String())
	

When running these examples (with go run .), we get the following output, verifying that both implementations decode the reflection values identically:

--- Step 1: dynamicpb (Standard Go Reflection) ---
Serialized bytes: 0a1948656c6c6f20456c697a612c20686f772061726520796f753f
Decoded message: Hello Eliza, how are you?

--- Step 2: hyperpb (Table-Driven Bytecode VM) ---
Decoded message: Hello Eliza, how are you?

--- Step 3: hyperpb + Shared (Memory Reuse Arena) ---
Decoded message: Hello Eliza, how are you?
Memory arena recycled.

By using the exact same standard protoreflect interface, hyperpb acts as a drop-in replacement for downstream read operations while executing faster and allocating much less in these benchmarks. So I wanted to check whether this held up in my own tiny benchmark goblin cave.

Performance Evaluation

I benchmarked three dynamic parsing strategies against statically generated Go Protobuf code to measure the difference in this setup:

The benchmarks evaluate performance across three payload scales:

• Small: A flat message with 4 primitive fields (ID, status, age, score).
• Medium: A nested event message containing an actor object, tags, and a metadata map.
• Large: An array repeating the Medium event 100 times.

The source code and setup for these benchmarks are available in the dynamic-protobuf-in-go/benchmarks directory.

All benchmarks were executed on an Apple M1 Pro (darwin/arm64) using Go 1.26. Descriptor compilation was excluded from timing and performed once during benchmark initialization. Measurements represent deserialization only (proto.Unmarshal) and were collected using go test -bench=. -benchmem.

Benchmark Results

Analysis: Speed and Memory Efficiency

The numbers get more interesting as the payloads get larger.

1. Execution Speedup

On a Large Payload, reflection-based dynamicpb takes 241,009 ns.

• Standard hyperpb executes in 24,664 ns (a 9.7x speedup).
• hyperpb + Shared executes in 17,967 ns (a 13.4x speedup).

Note: These benchmarks measure parsing and deserialization costs only. Work performed after unmarshalling (such as downstream data manipulation or field access) is excluded.

Interestingly, both hyperpb configurations outperform compile-time generated static Protobuf code (Concrete (proto) at 54,078 ns and reflection-free Concrete (vtproto) at 35,350 ns). That is the part that surprised me. It does not mean hyperpb’s parser engine is inherently faster than generated Go code. Rather, the combination of bytecode parsing and arena-backed allocation reduces object creation costs for large nested payloads. Standard generated Protobuf still allocates individual heap objects for the nested sub-messages in this benchmark. The memory arena in hyperpb + Shared allocates this memory contiguously.

In these benchmarks, this crossover point, where dynamic parsing paired with a memory arena beats statically compiled generated code, occurs even at the Medium Payload scale. At that size, hyperpb + Shared (286 ns) already edges out reflection-free Concrete (vtproto) (306 ns).

2. Reducing Heap Allocations

The allocation statistics highlight the biggest architectural advantage. On a Large Payload:

• dynamicpb: 4,117 heap allocations per message.
• Concrete (proto): 1,509 heap allocations.
• hyperpb: 12 heap allocations.
• hyperpb + Shared: 1 heap allocation. (The remaining allocation appears to come from the top-level message pointer escaping to the heap as a proto.Message interface wrapper. Because the standard proto.Unmarshal signature requires passing an interface, this top-level escape cannot easily be avoided and prevents hitting an absolute zero allocation count.)

For hot-path event routing or proxying services, fewer allocations should translate into less GC pressure, which can help with CPU usage and tail latency under real load.

When to Use hyperpb

hyperpb is built for specific use cases. It is not a universal drop-in replacement for standard Go Protobuf code.

Ideal Use Cases

• Dynamic Gateways & Proxies: Systems receiving dynamic schemas at runtime that must inspect or forward payloads without ahead-of-time code generation.
• Developer Tooling: Tools like FauxRPC that mock interfaces, fuzz test services, or interact dynamically with user-supplied schemas.
• High-Throughput Pipelines: Pipelines with dynamic schemas where reflection overhead is a bottleneck.

Trade-offs and Constraints

• Experimental API: hyperpb is still pre-v1, so I would avoid wrapping it deeply into public APIs without a small compatibility layer.
• Platform Specificity: hyperpb relies on specialized runtime assembly and bytecode generators tailored for 64-bit little-endian architectures. It is officially supported only on amd64 and arm64 platforms. Compiling for other architectures requires the manual build tag hyperpb.unsupported, which compiles a slower generic parser backend.
• Read-Only vs Mutable: Reusing buffers via hyperpb.Shared works best for read-only access pipelines. If you need to mutate the parsed message or pass it asynchronously to other goroutines, you must copy the data or avoid using the shared arena. This increases allocations, though still resulting in fewer than standard dynamicpb.
• AOT Compilation: If your schema is known at build time, compiling your Protobuf definitions remains the best approach. Static compilation (vtproto or standard proto) offers strict type safety and requires no runtime bytecode compilation overhead.

Conclusion

For FauxRPC, the interesting part of hyperpb is that it offers a way to speed up the read-heavy parts of dynamic schema handling.

Even with only the request path changed, the difference was not subtle. FauxRPC still uses dynamicpb where it needs to build and mutate response messages. But for request parsing, where payloads are read-only, hyperpb is a nice optimization. The benchmarks show a clear difference: fewer heap objects, less GC pressure, and faster unmarshalling on larger payloads.

If your pipeline fits those constraints, hyperpb is worth trying.

References & Further Reading

For a deeper dive into hyperpb and Go memory arenas, check out the following resources:

• Official hyperpb Announcement:
  • “Introducing hyperpb: 10x faster dynamic Protobuf parsing that’s even 3x faster than generated code” on the Buf Blog.
• Deep Dives by Sunny (mcyoung):
  • “Parsing Protobuf Like Never Before” (the internals of hyperpb’s bytecode compiler design).
  • “Cheating the Reaper in Go” (a deep dive into the design and tradeoffs of memory arenas in Go).