kmcd.devhttps://kmcd.dev/posts/Recent content on kmcd.devHugo -- gohugo.ioenAll Rights ReservedUnknown Fields in Protobufhttps://kmcd.dev/posts/protobuf-unknown-fields/Thu, 16 Apr 2026 00:00:00 +0000https://kmcd.dev/posts/protobuf-unknown-fields/ <p> <img hspace="5" src="https://kmcd.dev/posts/protobuf-unknown-fields/cover.svg" /> </p> How Protobuf unknown fields enable seamless schema evolution and robust middleware. This article was originally published in March 2024. It was republished in April 2026 after some significant editing and modernization.

Protobuf includes a feature known as unknown fields. They act as a safety net when systems encounter data they weren’t explicitly built to handle. Here is a breakdown of what they are and why they matter.

What are Protobuf Unknown Fields?

Your .proto file defines the expected structure, fields, and data types. But what happens when you parse a message and it contains fields that aren’t in your current .proto definition?

These extra pieces of data are called unknown fields.

At a lower level, unknown fields are field numbers and wire types that exist in the serialized message but are not defined in the current schema.

This mechanism is what enables forward compatibility: an older version of your software can safely read, process, and forward data produced by a newer version of the schema without crashing or losing the new data.

Key idea: Unknown fields enable forward compatibility by default.

Preserving Unknown Data

A key aspect of unknown fields is how they behave during message manipulation.

If you receive a message with unknown fields and forward it to another system, Protobuf defaults to forwarding the unknown fields alongside the known ones. This ensures the receiving system gets the complete payload.

If this didn’t happen, you could accidentally clear field values set by another part of the system.

This forwarding capability also applies when persisting messages, as long as they remain in binary Protobuf format. If you store and later reload the binary payload, the unknown fields are preserved.

Key idea: Binary Protobuf preserves unknown fields end-to-end.

Historical Note: Protobuf v3 initially tried to simplify the specification by removing several proto2 features, but real-world usage forced them to walk the biggest ones back. Early versions of proto3 dropped unknown fields entirely, but this was reversed in v3.5. Similarly, proto3 initially removed the optional keyword, but brought it back in v3.15 after developers struggled to distinguish between a field being unset and a field just having a zero value, which is a classic programming mistake.

Comparison to JSON

Consider a scenario where a new field, email, is added to a user object. The backend is updated, but the frontend is not.

The issue in JSON systems is not JSON itself, it’s typed deserialization.

{
  "user": {
    "id": "0edc0903-9e31-47be-adad-1dfc434ca2d3",
    "name": "Bob",
    "email": "bob@example.com"
  }
}

If the frontend maps this into a typed structure:

class User {
  id: string;
  name: string;
}

The unknown field (email) is dropped during deserialization. When the object is sent back:

{
  "user": {
    "id": "0edc0903-9e31-47be-adad-1dfc434ca2d3",
    "name": "Bob"
  }
}

The email field is lost.

Key idea: Typed JSON pipelines often drop unknown fields during reserialization.

Protobuf Behavior

With Protobuf, the same scenario behaves differently.

Even if the frontend does not know about the email field, it is preserved internally:

Symbol(@bufbuild/protobuf/unknown-fields): [
  {0: {no:3, wire_type:2, data: Uint8Array(14)}}
]

(Note: This specific Symbol representation is how the @bufbuild/protobuf implementation manages it under the hood. Other JS/TS generators might expose this data slightly differently, but the underlying concept remains the same.)

  • no: 3 → field number (email)
  • wire_type: 2 → length-delimited (used for strings)
  • data → raw encoded value

When the message is re-encoded:

1:LEN {"0edc0903-9e31-47be-adad-1dfc434ca2d3"}
2:LEN {"Bob"}
3:LEN {"bob@example.com"}

The unknown field survives the round trip.

Key idea: Unknown fields are preserved even when not understood.

Why We Can’t “Guess” the Type

You might wonder why we can’t just look at the raw data in an unknown field and “guess” what it is. The reason is that Protobuf uses a very limited set of wire types that are shared across many different high-level data types.

Wire Type Meaning Used for…
0 Varint int32, int64, uint32, uint64, sint32, sint64, bool, enum
1 64-bit fixed64, sfixed64, double
2 Length-delimited string, bytes, embedded messages, packed repeated fields
5 32-bit fixed32, sfixed32, float

If you see an unknown field with Wire Type 0, you don’t know if it represents the number 150, the boolean true, or an enum value.

More critically, if you see Wire Type 2, you have no way of knowing if the data is a string, a bytes buffer, or a complex nested message. If you try to “guess” and parse a string as a sub-message, you will likely get gibberish or a parsing error. This is why the schema is strictly required to understand the data, while the binary format is sufficient to preserve it.

The Middleware Advantage

Unknown fields shine in internal, middleware-heavy architectures.

Example:

  • API Gateway reads id for routing
  • Logging service reads trace_id
  • Downstream service understands full schema including new fields

Intermediate services can safely:

  1. Unmarshal using an older schema
  2. Read known fields
  3. Forward the message unchanged

No coordination is required when new fields are added upstream.

Key idea: Internal middleware can stay stable while schemas evolve.

Observability: A Signal for Upgrades

Beyond just forwarding data safely, unknown fields provide a highly valuable observability metric.

When an API gateway or a downstream service detects unknown fields in incoming payloads, it is a clear telemetry signal: a client or upstream service is sending extra information because it is using a newer schema.

Instead of crashing or silently dropping the data, the service can log the presence of these unknown fields. You can use this data to trigger alerts, track the rollout progress of new features across your architecture, and pinpoint exactly which legacy services are lagging behind and due for an upgrade.

A Note on JSON Serialization and Object Re-use

There are a couple of important exceptions where unknown fields get lost.

First, unknown field preservation applies only to binary Protobuf serialization. If you convert from binary to JSON (e.g., using protojson in Go or toJson in TypeScript), unknown fields are dropped during the encoding process. When unmarshaling JSON back into Protobuf, many libraries are strictly configured by default. For example, Go’s protojson.Unmarshal will throw a hard error if it encounters unknown fields in the JSON payload unless you explicitly bypass it by passing DiscardUnknown: true. JSON simply isn’t designed to carry this extra payload without a strict schema map.

Second, preserving these fields during binary serialization requires that you re-use the exact same object for re-serialization. If you read a message, pull out the known fields, and map them into a freshly created object to send downstream, the unknown fields tied to the original object will be left behind.

Key idea: Binary preserves and JSON drops. Always re-use the original object if you want to keep unknown fields intact.

Databases and Security

The theoretical elegance of unknown fields often collides with the messy reality of databases and security perimeters. In practice, relying on unknown fields breaks down entirely in a few critical scenarios.

First, let us consider database persistence. If clients are trying to store extra data, and a backend service parses a Protobuf message to map it to standard relational database columns, those unknown fields are absolutely gone. There is no magic column for data your database schema does not know about.

The only way to achieve true end-to-end preservation is to store the entire serialized Protobuf message directly in the database as a BLOB. Some teams do this, but blindly storing data you haven’t validated and don’t even recognize is highly dangerous.

Allowing unknown fields to propagate unchecked from external sources is a significant security risk. While they are a powerful tool inside clearly defined, trusted internal pipelines, accepting them from the open web opens your system up to data smuggling. It allows malicious actors to sneak unvalidated payloads into unknown fields to bypass validation layers that only inspect known schema structures. If your systems blindly unmarshal, store, and forward this data, older services act as unwitting mules for malicious input.

Because of these exact risks, the standard security posture is to aggressively filter at the edge. API Gateways and ingress proxies should explicitly discard unknown fields before the data ever reaches internal microservices.

Conclusion

Unknown fields provide a powerful mechanism for forward compatibility in distributed systems. They allow internal systems to evolve independently, act as a clear signal for required upgrades, reduce coordination overhead, and simplify middleware design.

However, they are not a substitute for validation, schema discipline, or proper security boundaries. Use them intentionally in trusted internal pipelines, but never trust them at the edge.

]]>IRC Log: Reactionaryhttps://kmcd.dev/posts/irc-log-reactionary/Mon, 30 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/irc-log-reactionary/

A seamless session update turns every customer's browser into an infinite loop cannon. ]]>
[14:15:00] *** PagerDutyBot: SEV-0: GLOBAL CONTROL PLANE UNREACHABLE. API ERROR RATE > 95%.
[14:15:05] *** sev_manager has joined
[14:15:10] <sev_manager> Status?
[14:16:12] <net_ops_jen> It’s bad. Traffic to the auth service just verticalized. We’re seeing 50M RPS.
[14:17:00] <sev_manager> DDoS? Scrubbing center active?
[14:17:30] <net_ops_jen> That’s the weird part. It’s bypassing the WAF. It looks like legit traffic. TLS fingerprints are valid.
[14:18:10] <sev_manager> Did someone let Tyler push a new WAF regex? Please tell me we didn’t backpedal into 2019.
[14:19:00] <backend_bob> I’m looking at the sample logs. These are all authenticated requests.
[14:19:45] <backend_bob> They are all hitting POST /api/v4/user/token/refresh.
[14:20:10] <sev_manager> Why is everyone refreshing their token at the exact same second?
[14:21:00] <net_ops_jen> It’s not one refresh. I’m seeing the SAME user IDs hitting it 500 times per second.
[14:22:15] <frontend_felix> Uh, guys?
[14:22:30] <sev_manager> Go ahead Felix.
[14:23:00] <frontend_felix> We pushed the “Seamless Session” update to the dashboard 15 minutes ago.
[14:23:15] <frontend_felix> The goal was to silently refresh the token in the background so users don’t get logged out.
[14:24:00] <backend_bob> Felix… look at the code.
[14:25:00] <frontend_felix> I am.
[14:25:10] <frontend_felix> …
[14:25:15] <backend_bob> Felix?
[14:25:45] <frontend_felix> Oh no.
[14:26:00] <sev_manager> REPORT.
[14:26:30] <frontend_felix> Okay, so, in React… useEffect runs when a dependency changes.
[14:26:45] <frontend_felix> We have: useEffect(() => { refreshToken() }, [token])
[14:27:00] <backend_bob> And refreshToken()… updates the token?
[14:27:05] <frontend_felix> Yes.
[14:27:10] <backend_bob> Which triggers the useEffect again?
[14:27:12] <frontend_felix> Yes.
[14:27:30] <backend_bob> So you turned every single customer’s browser tab into an infinite loop cannon pointed at our auth servers?
[14:28:00] <frontend_felix> In my defense, the tokens are incredibly fresh.
[14:28:15] <sev_manager> Rollback the frontend.
[14:29:10] <sev_manager> I’ll start writing another ridiculously well edited postmortem blog post.
[14:29:41] <backend_bob> Hackernews and primeagen is going to going to love this one.

Cloudflare outage on December 5, 2025

]]>Faking protobuf data in Gohttps://kmcd.dev/posts/faking-protobuf-data-in-go/Thu, 26 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/faking-protobuf-data-in-go/ <p> <img hspace="5" src="https://kmcd.dev/posts/faking-protobuf-data-in-go/cover.svg" /> </p> Stop hand-writing test fixtures If you build Go services that use protobuf over gRPC, you often need realistic-looking data. Sometimes you need a lot of it and your creative juices and manual data entry only goes so far. Tests, local demos, and stubs all benefit from messages that look plausible without carrying meaning.

FauxRPC addresses this. It’s a small Go library that fills protobuf messages with plausible data. The focus is not perfect realism; it’s speed, convenience, and reducing boilerplate.

This article covers two main use cases: populating generated protobuf types and working with dynamic descriptors. Along the way, you’ll see how to generate data and control its behavior for repeatable results.

Core entry points

FauxRPC provides two primary functions:

  • fauxrpc.SetDataOnMessage: Populates a concrete Go struct generated from .proto files.
  • fauxrpc.NewMessage: Creates a message from a protoreflect.MessageDescriptor, useful when types aren’t known at compile time.

Both use the same underlying logic; the difference is whether you start with a Go struct or a descriptor.

Filling a generated message

For a simple case, consider the Eliza demo service and its SayResponse message.

Eliza Service SayResponse Example (click to expand)View on GitHub 
package main

import (
	"fmt"
	"log"

	elizav1 "buf.build/gen/go/connectrpc/eliza/protocolbuffers/go/connectrpc/eliza/v1"
	"github.com/sudorandom/fauxrpc"
	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
	msg := &elizav1.SayResponse{}
	if err := fauxrpc.SetDataOnMessage(msg, fauxrpc.GenOptions{}); err != nil {
		log.Fatalf("err: %s", err)
	}
	b, err := protojson.MarshalOptions{Indent: "  "}.Marshal(msg)
	if err != nil {
		log.Fatalf("err: %s", err)
	}
	fmt.Println(string(b))
}

SetDataOnMessage mutates the message in place. Every field receives a value compatible with its type.

Marshaling to JSON produces something like:

{
  "sentence": "Jean shorts."
}

Values change each run. That randomness exposes assumptions in tests and ensures your code doesn’t rely on fixed values.

Populating a nested message

Most protobufs are more complex than a single field. For example, an ownerv1.Owner message with nested messages, timestamps, enums, and strings:

Owner Service Owner Example (click to expand)View on GitHub 
package main

import (
	"fmt"
	"log"

	ownerv1 "buf.build/gen/go/bufbuild/registry/protocolbuffers/go/buf/registry/owner/v1"
	"github.com/sudorandom/fauxrpc"
	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
		msg := &ownerv1.Owner{}
		if err := fauxrpc.SetDataOnMessage(msg, fauxrpc.GenOptions{}); err != nil {
			log.Fatalf("err: %s", err)
		}
		b, err := protojson.MarshalOptions{Indent: "  "}.Marshal(msg)
		if err != nil {
			log.Fatalf("err: %s", err)
		}
		fmt.Println(string(b))
}

SetDataOnMessage recursively populates nested messages, assigns valid enum values, and generates RFC 3339 timestamps. Marshaled with protojson:

{
  "organization": {
    "id": "a4cf6166453b49d9811cfdc169c36354",
    "createTime": "1912-06-01T16:45:13.510830647Z",
    "updateTime": "2009-04-03T13:11:31.216229245Z",
    "name": "xm0r3",
    "description": "Godard selvage.",
    "url": "https://www.dynamicreintermediate.name/robust/seize/metrics/b2c",
    "verificationStatus": "ORGANIZATION_VERIFICATION_STATUS_OFFICIAL"
  }
}

This works well for local servers, demos, and tests that focus on shape rather than meaning.

Using dynamic messages

For proxies, gateways, or tools without generated Go types, dynamicpb allows message creation from descriptors. FauxRPC supports this:

Dynamic Protobuf Example (click to expand)View on GitHub 
package main

import (
	"fmt"
	"log"

	elizav1 "buf.build/gen/go/connectrpc/eliza/protocolbuffers/go/connectrpc/eliza/v1"
	"github.com/sudorandom/fauxrpc"
	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
    msg, err := fauxrpc.NewMessage(
        elizav1.File_connectrpc_eliza_v1_eliza_proto.Messages().ByName("SayResponse"),
        fauxrpc.GenOptions{},
    )
    if err != nil {
        log.Fatalf("err: %s", err)
    }
    b, err := protojson.MarshalOptions{Indent: "  "}.Marshal(msg)
    if err != nil {
        log.Fatalf("err: %s", err)
    }
    fmt.Println(string(b))
}

FauxRPC fills dynamic messages with the same rules as concrete types, making them usable like any other protobuf message.

Respecting protovalidate constraints

FauxRPC reads protovalidate annotations to generate data that respects field constraints. Length limits, numeric ranges, regex patterns, and formats are applied when generating values.

For example, a User message with a username, email, and age:

Protovalidate-aware Data Generation (click to expand)View on GitHub 
package main

import (
	"fmt"

	"github.com/sudorandom/fauxrpc"
	gen "github.com/sudorandom/kmcd.dev/faking-protobuf-data-in-go/gen"
	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
	user := &gen.User{}

	err := fauxrpc.SetDataOnMessage(user, fauxrpc.GenOptions{})
	if err != nil {
		panic(err)
	}

	out, err := protojson.MarshalOptions{Indent: "  "}.Marshal(user)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

Sample output:

{
  "username": "Ezequiel",
  "email": "kiarasantos@grant.ne",
  "age": 46
}

FauxRPC ensures string lengths, numeric ranges, and formats match the annotations. The result is still fake data but plausible enough to exercise validators and gateways realistically.

Controlling randomness with GenOptions

By default, FauxRPC uses a global random generator. GenOptions allows supplying a seeded gofakeit.Faker for repeatable output:

Customizing Data Generation with GenOptions (click to expand)View on GitHub 
package main

import (
	"fmt"
	"log"

	"github.com/brianvoe/gofakeit/v7"
	elizav1 "buf.build/gen/go/connectrpc/eliza/protocolbuffers/go/connectrpc/eliza/v1"
	"github.com/sudorandom/fauxrpc"
	"google.golang.org/protobuf/encoding/protojson"
)

func main() {
	faker := gofakeit.New(123) // Seed the faker for deterministic output
	msg := &elizav1.SayResponse{}
	if err := fauxrpc.SetDataOnMessage(msg, fauxrpc.GenOptions{Faker: faker}); err != nil {
		log.Fatalf("err: %s", err)
	}
	b, err := protojson.MarshalOptions{Indent: "  "}.Marshal(msg)
	if err != nil {
		log.Fatalf("err: %s", err)
	}
	fmt.Println(string(b))
}

Repeatable generation helps make tests predictable without hard-coded fixtures.

FauxRPC in handlers

At the edge of a service, handlers can generate fake responses quickly.

ConnectRPC example:

ConnectRPC Handler (click to expand)View on GitHub 
package main

import (
	"context"
	"github.com/sudorandom/fauxrpc"
	"github.com/bufbuild/connect-go"
	elizav1 "buf.build/gen/go/connectrpc/eliza/protocolbuffers/go/connectrpc/eliza/v1"
)

func Say(ctx context.Context, req *connect.Request[elizav1.SayRequest]) (*connect.Response[elizav1.SayResponse], error) {
    msg := &elizav1.SayResponse{}
    if err := fauxrpc.SetDataOnMessage(msg, fauxrpc.GenOptions{}); err != nil {
        return nil, err
    }
    return connect.NewResponse(msg), nil
}

Request example:

$ buf curl --schema=buf.build/connectrpc/eliza \
         -d '{"sentence": "Hello world!"}' \
         http://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Microdosing."
}

gRPC-go example:

gRPC-go Handler (click to expand)View on GitHub 
package main

import (
	"context"
	"github.com/sudorandom/fauxrpc"
	elizav1 "buf.build/gen/go/connectrpc/eliza/protocolbuffers/go/connectrpc/eliza/v1"
)

func Say(ctx context.Context, req *elizav1.SayRequest) (*elizav1.SayResponse, error) {
	msg := &elizav1.SayResponse{}
	if err := fauxrpc.SetDataOnMessage(msg, fauxrpc.GenOptions{}); err != nil {
		return nil, err
	}
	return msg, nil
}

The behavior is identical and works out-of-the-box.

FauxRPC CLI

The CLI can generate data without compiling Go code:

$ fauxrpc generate --schema=. --target=example.v1.User
{"username":"Birdie", "email":"pariskozey@hamill.in", "age":54}

The CLI is incredibly flexible with the –schema flag. You can point it at:

  • A local .proto file or a directory of them.
  • A compiled protobuf descriptor set (binpb).
  • A remote Buf Schema Registry repository (e.g., buf.build/acme/auth).

Closing thoughts

FauxRPC is not a data modeling tool; it generates values consistent with protobuf types and constraints. This makes it useful for quickly standing up services, testing clients, and exploring API shapes.

It works with both ConnectRPC and grpc-go, and provides hooks for custom data generation. For full details, see the pkg.go.dev reference.

]]>Y'all are Sleeping on Mise-en-Placehttps://kmcd.dev/posts/mise-en-place/Thu, 19 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/mise-en-place/ <p> <img hspace="5" src="https://kmcd.dev/posts/mise-en-place/cover.svg" /> </p> Why I default to this tool for every project Dependency management is a perennial headache. Your local environment works perfectly until you set up a new laptop, and suddenly everything breaks due to version differences. This usually hits new hires the hardest since they are setting things up fresh and end up accidentally testing your codebase against brand new tool versions.

We have plenty of tools to solve this. I’ve been through asdf, pyenv, nvm, gvm, bazelisk, Homebrew bundles, and Dev Containers.

But they all fall short in ways that mise-en-place (usually just called mise) doesn’t. I’ve actually reached the point where nearly all my dev tooling is project-scoped. I barely install anything globally anymore.

Getting rid of siloed version managers

Most of us started with pyenv, nvm, or gvm. They do the job, but they operate in silos. If your repository has a Python backend, a Node frontend, and some Go CLI helpers, you end up juggling three different version managers and configuration formats.

mise is polyglot. Every repository gets a single mise.toml. You can even scope specific versions to subdirectories. It gives you one file per context, eliminating the need for system-level packages.

[tools]
node = "20.10"
python = "3.12"
go = "1.23"

The moment you cd into a directory, mise activates the right versions instantly. If a nested folder has its own mise.toml, those versions automatically take over. This makes supporting monorepos, mixed stacks, or legacy services incredibly easy.

Native GitHub releases

This is where mise heavily outshines older options like asdf. It isn’t just a wrapper for language runtimes. It has a native GitHub backend.

If your project needs a CLI binary like terraform, kubectl, or some obscure linter, you don’t have to pray someone wrote a plugin for it.

[tools]
"github:claudiodangelis/qrcp" = "latest"
"github:sharkdp/fd" = "v8.7.0"

It fetches the release artifacts, verifies them, and drops them into your path. It basically turns GitHub Releases into a native package manager.

Handling NPM and Go binaries

A lot of utilities live in the awkward space of being project-scoped but acting like global commands. mise handles this cleanly using npm: and go: backends:

  • Go tools: "go:github.com/fullstorydev/grpcurl/cmd/grpcurl" = "latest"
  • NPM-based linters: "npm:prettier" = "3.0"

They exist when you are in the project folder and disappear when you leave. Your global path stays clean.

Dev Containers are overkill for this

Dev containers have their place, particularly for massive multi-service architectures. But for day-to-day development, they are usually way more machinery than necessary. mise provides the main benefits with a fraction of the overhead:

  • No networking friction: You are running on bare metal. No port forwarding and no Docker bridge networks to debug.
  • Native file performance: Your IDE and compiler interact with the native filesystem. No virtiofs or gRPC-fuse lag.
  • Strict version pinning: A mise.lock file guarantees everyone on the team runs the exact same binary hash. You get reproducibility without the conceptual baggage of containers or the chore of maintaining Docker images.

The problem with asdf

I relied on asdf for a while. It was fine until it started randomly breaking. The tool resolution would get confused, and randomly a binary execution would revert to the system version instead of the pinned one. Once that trust is gone, it is hard to keep using a tool. I went looking for alternatives, found mise, and haven’t had a single issue since.

Every binary runs the exact version you expect, every single time. It is predictable and fast.

CI matching local

I also drop mise into my CI pipelines using a GitHub Action. It makes the remote tooling setup mirror my local environment perfectly.

Wrapping up

mise is built in Rust and makes juggling multiple languages genuinely painless. I primarily work in Go right now, and I don’t even have Go installed globally on my machine anymore.

It takes the friction out of environment setup. You just clone the repo, cd into it, and everything is exactly where it needs to be.

Resources

]]>IRC Log: Standup 2https://kmcd.dev/posts/irc-log-standup-2/Mon, 16 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/irc-log-standup-2/ <p> <img hspace="5" src="https://kmcd.dev/posts/irc-log-standup-2/cover.svg" /> </p> The team attempts another text-based standup, this time with management present. It goes significantly worse than the first one. The team attempts another text-based standup, this time with management present. It goes significantly worse than the first one.

[09:30] *** Topic: Daily Standup | 🚀 Q1 Goals | Synergy is Key
[09:30] <manager_matt> Good morning team! Let’s do a quick round-robin. Yesterday, Today, Blockers. Let’s keep it high-level and maximize our bandwidth.
[09:31] <qa_queen> Yesterday: Verifying that the “Magenta Login” button from last sprint still causes eye strain. Today: It does. Also, the Nyan Cat easter egg is now triggering on the “Delete Account” modal. Blockers: My sanity.
[09:32] <manager_matt> Great engagement metrics on that cat though! Let’s leverage that stickiness. Ben?
[09:33] <backend_ben> Yesterday: I noticed our JSON parsing overhead was non-zero. Today: I am replacing the standard library’s JSON decoder with a custom binding to a SIMD-accelerated C++ library I found on a SourceForge archive from 2009.
[09:33] <backend_ben> Also, I noticed the pre-receive hooks were performing a full O(n) validation of the commit graph and deep-packet inspection for JIRA keys, which was bottlenecking the Git daemon’s throughput during my refactor. I’ve bypassed them globally to maximize our collective velocity.
[09:34] <manager_matt> Unlocking the pipeline! That’s what I like to see. Lean and mean. Ben, you’re a rockstar.
[09:34] <senior_dev> Ben. You disabled the branch protections? The ONLY thing preventing a junior from overwriting the entire history?
[09:35] <intern_ian> Wait, it worked! I’ve been trying to fix my “whoopsie” commit history all morning but it kept giving me “protected branch” errors. But I just tried again and it went through!
[09:36] <intern_ian> I used git filter-branch to delete the node_modules I accidentally committed to history and then did a –force push. GitHub says the repo was created “1 minute ago” and has 1 commit. Clean slate!
[09:37] <senior_dev> EVERYONE. DO NOT PULL.
[09:38] <junior_dev> Too late. I just pulled. My local folder is now empty except for index.html.
[09:39] <junior_dev> Also, users are reporting that Production is down. But it looks… professional? It’s just a clean “Cloudflare Error 522: Connection Timed Out” page.
[09:40] <manager_matt> Oh, nice. That’s a very recognizable UI. Users trust that. It screams “Big Tech.”
[09:40] <senior_dev> … Matt. We don’t use Cloudflare. We are on bare metal AWS. We don’t even have a CDN.
[09:41] <backend_ben> It’s true. My new architecture routes traffic directly via raw TCP sockets to my laptop.
[09:41] <senior_dev> So where is the Cloudflare page coming from?
[09:42] <intern_ian> Oh! I copied the index.html from a “Modern React App” tutorial I found, but the tutorial site was down when I scraped it. I just assumed the cloud icon was a loading spinner.
[09:42] <intern_ian> Did I hardcode a timeout error as our homepage?
[09:43] <manager_matt> Let’s not get hung up on semantics. It’s deployed. It’s stable. It’s consistent. I have a hard stop for a “Visioneering” webinar. Great hustle, team!
[09:43] *** manager_matt has left the channel (Client Quit)
[09:44] <qa_queen> Ian, I’m logging a ticket: “Homepage cloud icon is not centered.”
]]>HTTP/2 From Scratch: Part 4https://kmcd.dev/posts/http2-from-scratch-part-4/Wed, 11 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/http2-from-scratch-part-4/ <p> <img hspace="5" src="https://kmcd.dev/posts/http2-from-scratch-part-4/cover.svg" /> </p> More HPACK and using http.Request and http.Response Part 3 ended on a high note: we got a real 200 OK from a server. But the code felt brittle. We were crafting requests by hand-picking hex codes from a spec sheet (0x82 for GET, 0x84 for /) and our “client” couldn’t have been less practical. It was a proof-of-concept held together with duct tape and hope.

Today, we’re rebuilding it properly. We’ll finish our HPACK implementation and, most importantly, refactor the raw socket logic into a clean Go client that speaks the language of the standard library: http.Request and http.Response.

Previously, we only implemented the Static Table, a read-only list of 61 common headers. But the real power of HTTP/2 comes from the Dynamic Table.

The Dynamic Table is essentially a short-term memory, or cache, that both the client and server build together for the duration of a single TCP connection.

  1. Client sends the header x-user-id: 123 using the “Literal Header Field with Incremental Indexing” representation. This tells the server to use the value and add it to its Dynamic Table.
  2. Server acknowledges and adds it to its table at Index 62.
  3. Client sends a second request. Instead of sending the string x-user-id: 123 again, it just sends the byte for Index 62.

To make this work, we need to handle a few new concepts in our hpack.go.

1. Huffman Decoding

Next up is Huffman coding. HTTP/2 uses it to shrink header strings. The specification includes a canonical Huffman table for this, but it’s massive. Implementing it would mean embedding a 400-line static lookup table and writing a tree-walker to encode/decode it. I briefly considered it, then imagined debugging a single bit-flip error in that tree.

That’s why I’m using golang.org/x/net/http2/hpack for this one part. It’s the perfect example of when building “from scratch” becomes counter-productive. We can pull in a battle-tested implementation for the tedious part and keep our focus on the protocol’s state and frame logic.

// We import the official helper just for the string decoding math
import "golang.org/x/net/http2/hpack"

// ... inside our decodeString method
if huffman {
    return hpack.HuffmanDecodeToString(data)
}

Implementing hpack by hand can be a personal exercise for you later!

2. The Decoder Loop

Our previous Decode method was incomplete. We need to handle four specific bit-patterns defined in RFC 7541.

Aside from the standard Indexed fields (where we just look up a value), we now have to handle Literals. Some literals request Incremental Indexing (meaning the receiver saves them to its dynamic table for later), while others are Non-Indexed (one-offs for things like tokens that shouldn’t be cached). Finally, we might see a Table Size Update, which tells us if the server resized its compression window.

Here is the updated loop using bitmasks to identify the type:

func (h *HPACKDecoder) Decode(payload []byte) ([]HeaderField, error) {
    var headers []HeaderField
    r := bytes.NewReader(payload)
    
    for r.Len() > 0 {
        b, _ := r.ReadByte()
        
        // Indexed Header Field (starts with 1xxxxxxx)
        if b&maskIndexed == patternIndexed { 
            // ... fetch from Static or Dynamic table ...
        
        // Literal with Incremental Indexing (starts with 01xxxxxx)
        // (Read the header, then ADD it to the Dynamic Table)
        } else if b&maskLiteralIncremental == patternLiteralIncremental { 
            header, _ := h.decodeLiteralHeader(r, 6, true)
            headers = append(headers, header)

        // Literal without Indexing (starts with 0000xxxx or 0001xxxx)
        // (Read the header, do NOT add to Dynamic Table)
        } else if b&maskLiteral == patternLiteral { 
             header, _ := h.decodeLiteralHeader(r, 4, false)
             headers = append(headers, header)

        // Dynamic Table Size Update (starts with 001xxxxx)
        } else if b&maskDynamicTableSize == patternDynamicTableSize { 
            // ... update max size ...
        }
    }
    return headers, nil
}
full hpack.go (click to expand)View on GitHub 
package main

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"io"

	"golang.org/x/net/http2/hpack"
)

// RFC 7541: HPACK: Header Compression for HTTP/2
// https://datatracker.ietf.org/doc/html/rfc7541

const (
	// Masks for HPACK header field types
	maskIndexed            = 0x80 // 10000000
	maskLiteralIncremental = 0xc0 // 11000000
	maskDynamicTableSize   = 0xe0 // 11100000
	maskLiteral            = 0xf0 // 11110000

	// Patterns for HPACK header field types
	patternIndexed            = 0x80 // 10000000
	patternLiteralIncremental = 0x40 // 01000000
	patternDynamicTableSize   = 0x20 // 00100000
	patternLiteralNever       = 0x10 // 00010000
	patternLiteral            = 0x00 // 00000000

	HuffmanFlagMask         = 0x80
	IntegerContinuationMask = 0x80
)

// HeaderField represents a header field with a name and value.
type HeaderField struct {
	Name, Value string
}

// StaticTable is the predefined, unchangeable table of header fields, as defined in RFC 7541 Appendix A.
var StaticTable = []HeaderField{
	{Name: ":authority", Value: ""},
	{Name: ":method", Value: "GET"},
	{Name: ":method", Value: "POST"},
	{Name: ":path", Value: "/"},
	{Name: ":path", Value: "/index.html"},
	{Name: ":scheme", Value: "http"},
	{Name: ":scheme", Value: "https"},
	{Name: ":status", Value: "200"},
	{Name: ":status", Value: "204"},
	{Name: ":status", Value: "206"},
	{Name: ":status", Value: "304"},
	{Name: ":status", Value: "400"},
	{Name: ":status", Value: "404"},
	{Name: ":status", Value: "500"},
	{Name: "accept-charset", Value: ""},
	{Name: "accept-encoding", Value: "gzip, deflate"},
	{Name: "accept-language", Value: ""},
	{Name: "accept-ranges", Value: ""},
	{Name: "accept", Value: ""},
	{Name: "access-control-allow-origin", Value: ""},
	{Name: "age", Value: ""},
	{Name: "allow", Value: ""},
	{Name: "authorization", Value: ""},
	{Name: "cache-control", Value: ""},
	{Name: "content-disposition", Value: ""},
	{Name: "content-encoding", Value: ""},
	{Name: "content-language", Value: ""},
	{Name: "content-length", Value: ""},
	{Name: "content-location", Value: ""},
	{Name: "content-range", Value: ""},
	{Name: "content-type", Value: ""},
	{Name: "cookie", Value: ""},
	{Name: "date", Value: ""},
	{Name: "etag", Value: ""},
	{Name: "expect", Value: ""},
	{Name: "expires", Value: ""},
	{Name: "from", Value: ""},
	{Name: "host", Value: ""},
	{Name: "if-match", Value: ""},
	{Name: "if-modified-since", Value: ""},
	{Name: "if-none-match", Value: ""},
	{Name: "if-range", Value: ""},
	{Name: "if-unmodified-since", Value: ""},
	{Name: "last-modified", Value: ""},
	{Name: "link", Value: ""},
	{Name: "location", Value: ""},
	{Name: "max-forwards", Value: ""},
	{Name: "proxy-authenticate", Value: ""},
	{Name: "proxy-authorization", Value: ""},
	{Name: "range", Value: ""},
	{Name: "referer", Value: ""},
	{Name: "retry-after", Value: ""},
	{Name: "server", Value: ""},
	{Name: "set-cookie", Value: ""},
	{Name: "strict-transport-security", Value: ""},
	{Name: "transfer-encoding", Value: ""},
	{Name: "user-agent", Value: ""},
	{Name: "vary", Value: ""},
	{Name: "via", Value: ""},
	{Name: "www-authenticate", Value: ""},
}

var (
	staticTableMap     = make(map[HeaderField]int)
	staticTableNameMap = make(map[string]int)
)

func init() {
	for i, hf := range StaticTable {
		staticTableMap[hf] = i + 1
		if _, ok := staticTableNameMap[hf.Name]; !ok {
			staticTableNameMap[hf.Name] = i + 1
		}
	}
}

type DynamicTable struct {
	headers []HeaderField
	size    uint32
	maxSize uint32
}

func NewDynamicTable(maxSize uint32) *DynamicTable {
	return &DynamicTable{
		maxSize: maxSize,
	}
}

func (d *DynamicTable) At(i int) (HeaderField, bool) {
	if i < 0 || i >= len(d.headers) {
		return HeaderField{}, false
	}
	return d.headers[i], true
}

func (d *DynamicTable) Add(h HeaderField) {
	size := uint32(len(h.Name) + len(h.Value) + 32)
	for d.size+size > d.maxSize && len(d.headers) > 0 {
		last := d.headers[len(d.headers)-1]
		d.size -= uint32(len(last.Name) + len(last.Value) + 32)
		d.headers = d.headers[:len(d.headers)-1]
	}
	d.headers = append([]HeaderField{h}, d.headers...)
	d.size += size
}

func (d *DynamicTable) SetMaxSize(size uint32) {
	d.maxSize = size
	for d.size > d.maxSize && len(d.headers) > 0 {
		last := d.headers[len(d.headers)-1]
		d.size -= uint32(len(last.Name) + len(last.Value) + 32)
		d.headers = d.headers[:len(d.headers)-1]
	}
}

type HPACKDecoder struct {
	dynamicTable *DynamicTable
}

func NewHPACKDecoder(maxSize uint32) *HPACKDecoder {
	return &HPACKDecoder{
		dynamicTable: NewDynamicTable(maxSize),
	}
}

func (h *HPACKDecoder) Header(i int) (HeaderField, bool) {
	if i <= 0 {
		return HeaderField{}, false
	}
	if i <= len(StaticTable) {
		return StaticTable[i-1], true
	}
	return h.dynamicTable.At(i - len(StaticTable) - 1)
}

func (h *HPACKDecoder) Decode(payload []byte) ([]HeaderField, error) {
	var headers []HeaderField
	r := bytes.NewReader(payload)
	for r.Len() > 0 {
		b, _ := r.ReadByte()
		if b&maskIndexed == patternIndexed { // Indexed Header Field
			index, n := decodeInt(b, r, 7)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, ok := h.Header(index)
			if !ok {
				return nil, fmt.Errorf("invalid header index: %d", index)
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskLiteralIncremental == patternLiteralIncremental { // Literal Header Field with Incremental Indexing
			index, n := decodeInt(b, r, 6)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, err := h.decodeLiteralHeader(r, index, true)
			if err != nil {
				return nil, err
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskLiteral == patternLiteral || b&maskLiteral == patternLiteralNever { // Literal Header Field without or never indexed
			index, n := decodeInt(b, r, 4)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, err := h.decodeLiteralHeader(r, index, false)
			if err != nil {
				return nil, err
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskDynamicTableSize == patternDynamicTableSize { // Dynamic Table Size Update
			size, n := decodeInt(b, r, 5)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			h.dynamicTable.SetMaxSize(uint32(size))
		} else {
			return nil, fmt.Errorf("not implemented: unknown header field type %08b", b)
		}
	}
	return headers, nil
}

func (h *HPACKDecoder) decodeLiteralHeader(r *bytes.Reader, index int, addToDynamicTable bool) (HeaderField, error) {
	var name string
	var err error
	if index > 0 {
		header, ok := h.Header(index)
		if !ok {
			return HeaderField{}, fmt.Errorf("invalid header index: %d", index)
		}
		name = header.Name
	} else {
		name, err = h.decodeString(r)
		if err != nil {
			return HeaderField{}, err
		}
	}
	value, err := h.decodeString(r)
	if err != nil {
		return HeaderField{}, err
	}
	header := HeaderField{Name: name, Value: value}
	if addToDynamicTable {
		h.dynamicTable.Add(header)
	}
	return header, nil
}

func (h *HPACKDecoder) decodeString(r *bytes.Reader) (string, error) {
	b, _ := r.ReadByte()
	huffman := b&HuffmanFlagMask == HuffmanFlagMask
	length, n := decodeInt(b, r, 7)
	if n < 0 {
		return "", fmt.Errorf("failed to decode integer")
	}
	if r.Len() < length {
		return "", io.ErrUnexpectedEOF
	}
	data := make([]byte, length)
	r.Read(data)
	if huffman {
		return hpack.HuffmanDecodeToString(data)
	}
	return string(data), nil
}

func decodeInt(b byte, r *bytes.Reader, n int) (int, int) {
	mask := (1 << n) - 1
	i := int(b) & mask
	if i < mask {
		return i, 1
	}

	var m uint = 0
	bytesRead := 1
	var val uint64
	for {
		b, err := r.ReadByte()
		if err != nil {
			return 0, -bytesRead
		}
		bytesRead++
		val |= uint64(b&127) << m
		m += 7
		if b&IntegerContinuationMask == 0 {
			break
		}
	}
	return i + int(val), bytesRead
}

type HPACKEncoder struct {
	dynamicTable *DynamicTable
}

func NewHPACKEncoder(maxSize uint32) *HPACKEncoder {
	return &HPACKEncoder{
		dynamicTable: NewDynamicTable(maxSize),
	}
}

func (e *HPACKEncoder) Encode(headers []HeaderField) []byte {
	var buf bytes.Buffer
	for _, hf := range headers {
		// Find a match in static table
		if index, ok := staticTableMap[hf]; ok {
			encodeInt(&buf, index, 7, patternIndexed)
			continue
		}

		// Find a name match in static table
		if index, ok := staticTableNameMap[hf.Name]; ok {
			encodeInt(&buf, index, 6, patternLiteralIncremental)
			encodeString(&buf, hf.Value)
			e.dynamicTable.Add(hf)
			continue
		}

		// Literal with literal name
		encodeInt(&buf, 0, 6, patternLiteralIncremental)
		encodeString(&buf, hf.Name)
		encodeString(&buf, hf.Value)
		e.dynamicTable.Add(hf)
	}
	return buf.Bytes()
}

func encodeInt(buf *bytes.Buffer, i int, n int, pattern byte) {
	mask := (1 << n) - 1
	if i < mask {
		buf.WriteByte(pattern | byte(i))
	} else {
		buf.WriteByte(pattern | byte(mask))
		i -= mask
		varint := make([]byte, binary.MaxVarintLen64)
		c := binary.PutUvarint(varint, uint64(i))
		buf.Write(varint[:c])
	}
}

func encodeString(buf *bytes.Buffer, s string) {
	// no huffman for now
	encodeInt(buf, len(s), 7, 0x00) // This is patternLiteral. We are encoding a raw string.
	buf.WriteString(s)
}

Writing an Encoder

In the last part, we manually crafted our request bytes (0x82, 0x84…). That was great for learning, but terrible for usability. We need an HPACKEncoder.

Our encoder will use a “naive” but compliant strategy to compress headers:

  1. Perfect Match: Is the full header (:method: GET) already in the Static Table? If yes, send the 1-byte index.
  2. Name Match: Is just the name (:authority) in the Static Table? If yes, send the index for the name, but write the value as a string literal and instruct the server to index it.
  3. No Match: Send both the name and value as string literals and instruct the server to index them.

This simple logic covers 90% of use cases without needing complex state tracking on the client side.

func (e *HPACKEncoder) Encode(headers []HeaderField) []byte {
    var buf bytes.Buffer
    for _, hf := range headers {
        // Try to find a full match (Name + Value)
        if index, ok := staticTableMap[hf]; ok {
            encodeInt(&buf, index, 7, patternIndexed)
            continue
        }

        // Try to find a Name match
        if index, ok := staticTableNameMap[hf.Name]; ok {
            encodeInt(&buf, index, 6, patternLiteralIncremental) // Literal with Incremental Indexing
            encodeString(&buf, hf.Value)
            e.dynamicTable.Add(hf) // We must track this too!
            continue
        }

        // Send as a full literal
        encodeInt(&buf, 0, 6, patternLiteralIncremental)
        encodeString(&buf, hf.Name)
        encodeString(&buf, hf.Value)
        e.dynamicTable.Add(hf)
    }
    return buf.Bytes()
}

Building a Real Client

Now for the satisfying part. We are going to take all that raw socket code from main.go and wrap it in a clean struct that mimics the standard library. Our goal here is to integrate seamlessly with Go’s net/http package, allowing us to leverage familiar types.

// client.go

type Client struct {
    Timeout time.Duration
}

func NewClient() *Client {
    return &Client{
        Timeout: 30 * time.Second,
    }
}

func (c *Client) Do(req *http.Request) (*http.Response, error) {
    // ... Connection and Handshake logic ...

    // Convert http.Request to HPACK HeaderFields
    authority := req.URL.Host
    if authority == "" {
        authority = req.Host // Fallback for robustness
    }
    headers := []HeaderField{
        {Name: ":method", Value: req.Method},
        {Name: ":scheme", Value: req.URL.Scheme}, 
        {Name: ":authority", Value: authority},
        {Name: ":path", Value: req.URL.Path},
    }
    // ... append req.Header ...

    // Encode and send the HEADERS frame
    // ...

    // Read Loop and a Subtle Bug
    // ...
    
    return httpResp, nil
}

This part was mostly straightforward. We convert to and from the HeaderField type. We make sure the we use the host in the request as the :authority value. But there is one thing that wasted more time than I’d like to admit.

After sending our request, we enter a loop to read the server’s response. This is where I hit an annoying Go concurrency bug that wasted my time. My first implementation had a defer conn.Close() right after dialing the connection. I figured “Why not close the connection? This toy client will only issue a single request per connection”. Here’s what I was doing:

// The WRONG way
conn, err := tls.Dial(...)
if err != nil { ... }
defer conn.Close() // Problem!

// ... send request ...

return &http.Response{ Body: io.NopCloser(bytes.NewReader(bodyBytes)) }, nil

The problem? defer executes when the function (Do) returns. But the user of our client needs to read the response Body after Do has returned. The connection was being closed before they had a chance to read it! The old version of the client would read the full response body in-line. Switching to act more like http.Client significantly changed when the response body is read and, in turn, changed the lifecycle of the underlying connection.

The fix is to remove the defer and instead wrap the connection itself in a custom io.ReadCloser that we assign to the http.Response.Body. When the user calls resp.Body.Close(), it’s now our responsibility to close the underlying network connection. This is the standard pattern used by Go’s own net/http client.

Here’s the read loop that replaces the ... above.

// The Read Loop
var respHeaders []HeaderField
var respBody []byte
for {
    frame, err := ReadFrame(conn)
    if err != nil {
        return nil, fmt.Errorf("connection closed: %w", err)
    }

    switch frame.Header.Type {
    case FrameData:
        respBody = append(respBody, frame.Payload...)
    case FrameHeaders:
        headers, err := hpackDec.Decode(frame.Payload)
        if err != nil {
            return nil, fmt.Errorf("hpack error: %w", err)
        }
        respHeaders = append(respHeaders, headers...)
    case FrameGoAway:
        // Handle server telling us to go away
    case FrameWindowUpdate:
        // Handle flow control
    }

    // The stream is finished when we receive a frame with the END_STREAM flag.
    // This can be on a HEADERS frame or the final DATA frame.
    isEndFrame := frame.Header.Type == FrameData || frame.Header.Type == FrameHeaders
    if isEndFrame && (frame.Header.Flags&FlagEndStream != 0) {
        break
    }
}

// ... build http.Response, then assign our custom Body ...
httpResp.Body = &responseBody{bytes.NewReader(respBody), conn}
return httpResp, nil

// And the helper struct that makes this possible:
type responseBody struct {
    *bytes.Reader
    conn io.Closer
}

func (rb *responseBody) Close() error {
    return rb.conn.Close()
}
full client.go (click to expand)View on GitHub 
package main

import (
	"bytes"
	"crypto/tls"
	"encoding/binary"
	"fmt"
	"io"
	"net/http"
	"time"
)

const (
	// Protocol constants
	Preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	Server  = "kmcd.dev:443"

	// Frame Types (RFC 9113 Section 6)
	FrameData         uint8 = 0x0
	FrameHeaders      uint8 = 0x1
	FramePriority     uint8 = 0x2
	FrameRstStream    uint8 = 0x3
	FrameSettings     uint8 = 0x4
	FramePushPromise  uint8 = 0x5
	FramePing         uint8 = 0x6
	FrameGoAway       uint8 = 0x7
	FrameWindowUpdate uint8 = 0x8
	FrameContinuation uint8 = 0x9

	// Common Flags
	FlagAck        uint8 = 0x01 // For SETTINGS/PING
	FlagEndStream  uint8 = 0x01 // For DATA/HEADERS
	FlagEndHeaders uint8 = 0x04 // For HEADERS/PUSH_PROMISE/CONTINUATION
)

// responseBody implements io.ReadCloser. It reads from the response body
// buffer and closes the underlying connection when Close is called.
type responseBody struct {
	*bytes.Reader
	conn io.Closer
}

func (rb *responseBody) Close() error {
	fmt.Println("Closing response body and underlying connection.")
	return rb.conn.Close()
}

type Client struct {
	Timeout time.Duration
}

func NewClient() *Client {
	return &Client{
		Timeout: 30 * time.Second,
	}
}

func (c *Client) Do(req *http.Request) (*http.Response, error) {
	// Setup TLS with ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	hpackDec := NewHPACKDecoder(4096)
	hpackEnc := NewHPACKEncoder(4096)

	port := "443"
	if req.URL.Port() != "" {
		port = req.URL.Port()
	}

	conn, err := tls.Dial("tcp", req.URL.Hostname()+":"+port, config)
	if err != nil {
		return nil, fmt.Errorf("failed to connect: %w", err)
	}
	// Do NOT defer conn.Close(). The response body wrapper will be responsible for it.

	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		conn.Close() // Close connection if h2 is not negotiated
		return nil, fmt.Errorf("server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", req.URL.Host, state.NegotiatedProtocol)

	// Send Connection Preface
	if _, err = conn.Write([]byte(Preface)); err != nil {
		conn.Close()
		return nil, fmt.Errorf("failed to send preface: %w", err)
	}
	fmt.Println("Preface sent.")

	// Initial Handshake Loop (using the Client's hpackDec)
	// Send our initial empty settings
	mySettings := []byte{0, 0, 0, FrameSettings, 0, 0, 0, 0, 0}
	conn.Write(mySettings)

	serverSettingsAcked := false
	mySettingsAcked := false

	for !serverSettingsAcked || !mySettingsAcked {
		frame, err := ReadFrame(conn) // ReadFrame needs the raw conn
		if err != nil {
			conn.Close()
			return nil, fmt.Errorf("handshake read error: %w", err)
		}

		fmt.Printf("<<< [Handshake] Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameSettings:
			if frame.Header.Flags&FlagAck != 0 {
				mySettingsAcked = true
				fmt.Println("<<< Server ACK'd our settings")
			} else {
				ack := []byte{0, 0, 0, FrameSettings, FlagAck, 0, 0, 0, 0}
				conn.Write(ack)
				serverSettingsAcked = true
				fmt.Println(">>> Sent SETTINGS ACK")
			}
		case FrameWindowUpdate:
			fmt.Println("<<< Server provided flow control window")
		case FrameGoAway:
			conn.Close()
			return nil, fmt.Errorf("server sent GOAWAY during handshake")
		}
	}

	// Now the actual request sending logic from previous Do method
	authority := req.URL.Host
	if authority == "" {
		authority = req.Host // Fallback if URL.Host is empty
	}
	scheme := "https"
	if req.URL.Scheme != "" {
		scheme = req.URL.Scheme
	}
	headers := []HeaderField{
		{Name: ":method", Value: req.Method},
		{Name: ":scheme", Value: scheme},
		{Name: ":authority", Value: authority},
		{Name: ":path", Value: req.URL.Path},
	}
	for name, values := range req.Header {
		for _, value := range values {
			headers = append(headers, HeaderField{Name: name, Value: value})
		}
	}

	requestPayload := hpackEnc.Encode(headers)

	header := make([]byte, 9)
	payloadLen := len(requestPayload)
	header[0] = byte(payloadLen >> 16)
	header[1] = byte(payloadLen >> 8)
	header[2] = byte(payloadLen)
	header[3] = FrameHeaders
	header[4] = FlagEndStream | FlagEndHeaders
	binary.BigEndian.PutUint32(header[5:9], 1)

	if _, err := conn.Write(append(header, requestPayload...)); err != nil {
		conn.Close()
		return nil, fmt.Errorf("failed to send request: %w", err)
	}
	fmt.Println(">>> Sent HEADERS (Stream 1)")

	// Read response (using the conn local to Do)
	var respHeaders []HeaderField
	var respBody []byte
	for {
		frame, err := ReadFrame(conn) // ReadFrame needs the raw conn
		if err != nil {
			conn.Close()
			return nil, fmt.Errorf("connection closed: %w", err)
		}

		fmt.Printf("<<< Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameData:
			respBody = append(respBody, frame.Payload...)
			fmt.Printf("      [DATA] Length=%d\n", len(frame.Payload))
		case FrameHeaders:
			fmt.Println("      [HEADERS] Decoding...")
			headers, err := hpackDec.Decode(frame.Payload)
			if err != nil {
				conn.Close()
				return nil, fmt.Errorf("hpack error: %w", err)
			}
			respHeaders = append(respHeaders, headers...)
		case FrameGoAway:
			lastStream := binary.BigEndian.Uint32(frame.Payload[0:4]) & 0x7FFFFFFF
			errCode := binary.BigEndian.Uint32(frame.Payload[4:8])
			conn.Close()
			return nil, fmt.Errorf("GOAWAY: Last Stream %d, Error Code %d", lastStream, errCode)
		case FrameWindowUpdate:
			fmt.Println("      [WINDOW_UPDATE]")
		}

		isEndFrame := frame.Header.Type == FrameData || frame.Header.Type == FrameHeaders
		if isEndFrame && (frame.Header.Flags&FlagEndStream != 0) {
			fmt.Println("Stream finished.")
			break
		}
	}

	// Build http.Response
	httpResp := &http.Response{
		StatusCode: 200, // Hardcoded for now
		Proto:      "HTTP/2.0",
		ProtoMajor: 2,
		ProtoMinor: 0,
		Header:     make(http.Header),
		Body:       &responseBody{bytes.NewReader(respBody), conn},
	}

	for _, h := range respHeaders {
		httpResp.Header.Add(h.Name, h.Value)
		if h.Name == ":status" {
			fmt.Sscanf(h.Value, "%d", &httpResp.StatusCode)
		}
	}
	httpResp.Status = fmt.Sprintf("%d %s", httpResp.StatusCode, http.StatusText(httpResp.StatusCode))

	return httpResp, nil
}

The Result

With this refactor, our main.go transforms from a mess of magical hex values into clean, idiomatic Go:

func main() {
    client := NewClient()
    req, _ := http.NewRequest("GET", "https://kmcd.dev/", nil)

    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    // This now correctly closes our underlying TCP connection.
    defer resp.Body.Close()

    fmt.Printf("Status: %s\n", resp.Status)
    body, _ := io.ReadAll(resp.Body)
    fmt.Printf("Body Length: %d bytes\n", len(body))
}

Limitations

While we have a working client, it is strictly a “happy path” implementation:

  • Concurrency: Our client is synchronous. It sends a request and waits. Real HTTP/2 uses multiplexing to send many requests at once over a single connection.
  • Flow Control: We are ignoring WINDOW_UPDATE frames. If we tried to download a large file, our connection would stall once the window fills up.
  • Connection Re-use: We create a new TCP connection for every Do call. Because the HPACK Dynamic Table is tied to the connection, we aren’t actually gaining any compression benefits across multiple requests.
  • Many, Many Other Features: There is a lot of small details that this toy client completely glosses over. For example, trailer support. This is funny, because I actually added trailer support for quic-go: #4581, #4630. You can read up more about this when writing my gRPC over HTTP/3 series.

Getting this far, I have a renewed and massive respect for the net/http maintainers. We started this series with hex dumps and now have a client struct that respects io.Closer. The “happy path” alone is a journey through specifications, bit-masking, and subtle concurrency bugs. Handling real-world network conditions, multiplexing, and flow control is a monumental task that makes you appreciate the standard library on a new level.

For now, I’m happy with this victory. We’ve built a real, working HTTP/2 client.

Next Steps

Now that we’ve implemented a decent amount of HTTP/2, I think the next step is to delve into QUIC and HTTP/3. I mentioned earlier that HTTP/2 solves the head-of-line blocking issue. That’s only partially true. Yes, at the application layer HTTP/2 doesn’t have a head-of-line blocking issue but since HTTP/2 is built on top of TCP, it inherits sequential packet ordering. This is normally an amazing feature of TCP but in this case it actually hinders us. Since HTTP/2 multiplexes many independent streams over a single TCP connection, packet loss affecting stream A will still block delivery for stream B. The only way around this is to make significant updates to TCP (not going to happen) or completely abandon TCP altogether.

This, along with 0-RTT connection resumption (sending data before the handshake completes), are things that just aren’t possible to side-step using HTTP/2. This is why QUIC and HTTP/3 were created. But you’ll have to wait a bit longer before seeing me implement that from scratch.

See all of the code mentioned in this article here:

go/client.go (click to expand)View on GitHub 
package main

import (
	"bytes"
	"crypto/tls"
	"encoding/binary"
	"fmt"
	"io"
	"net/http"
	"time"
)

const (
	// Protocol constants
	Preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	Server  = "kmcd.dev:443"

	// Frame Types (RFC 9113 Section 6)
	FrameData         uint8 = 0x0
	FrameHeaders      uint8 = 0x1
	FramePriority     uint8 = 0x2
	FrameRstStream    uint8 = 0x3
	FrameSettings     uint8 = 0x4
	FramePushPromise  uint8 = 0x5
	FramePing         uint8 = 0x6
	FrameGoAway       uint8 = 0x7
	FrameWindowUpdate uint8 = 0x8
	FrameContinuation uint8 = 0x9

	// Common Flags
	FlagAck        uint8 = 0x01 // For SETTINGS/PING
	FlagEndStream  uint8 = 0x01 // For DATA/HEADERS
	FlagEndHeaders uint8 = 0x04 // For HEADERS/PUSH_PROMISE/CONTINUATION
)

// responseBody implements io.ReadCloser. It reads from the response body
// buffer and closes the underlying connection when Close is called.
type responseBody struct {
	*bytes.Reader
	conn io.Closer
}

func (rb *responseBody) Close() error {
	fmt.Println("Closing response body and underlying connection.")
	return rb.conn.Close()
}

type Client struct {
	Timeout time.Duration
}

func NewClient() *Client {
	return &Client{
		Timeout: 30 * time.Second,
	}
}

func (c *Client) Do(req *http.Request) (*http.Response, error) {
	// Setup TLS with ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	hpackDec := NewHPACKDecoder(4096)
	hpackEnc := NewHPACKEncoder(4096)

	port := "443"
	if req.URL.Port() != "" {
		port = req.URL.Port()
	}

	conn, err := tls.Dial("tcp", req.URL.Hostname()+":"+port, config)
	if err != nil {
		return nil, fmt.Errorf("failed to connect: %w", err)
	}
	// Do NOT defer conn.Close(). The response body wrapper will be responsible for it.

	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		conn.Close() // Close connection if h2 is not negotiated
		return nil, fmt.Errorf("server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", req.URL.Host, state.NegotiatedProtocol)

	// Send Connection Preface
	if _, err = conn.Write([]byte(Preface)); err != nil {
		conn.Close()
		return nil, fmt.Errorf("failed to send preface: %w", err)
	}
	fmt.Println("Preface sent.")

	// Initial Handshake Loop (using the Client's hpackDec)
	// Send our initial empty settings
	mySettings := []byte{0, 0, 0, FrameSettings, 0, 0, 0, 0, 0}
	conn.Write(mySettings)

	serverSettingsAcked := false
	mySettingsAcked := false

	for !serverSettingsAcked || !mySettingsAcked {
		frame, err := ReadFrame(conn) // ReadFrame needs the raw conn
		if err != nil {
			conn.Close()
			return nil, fmt.Errorf("handshake read error: %w", err)
		}

		fmt.Printf("<<< [Handshake] Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameSettings:
			if frame.Header.Flags&FlagAck != 0 {
				mySettingsAcked = true
				fmt.Println("<<< Server ACK'd our settings")
			} else {
				ack := []byte{0, 0, 0, FrameSettings, FlagAck, 0, 0, 0, 0}
				conn.Write(ack)
				serverSettingsAcked = true
				fmt.Println(">>> Sent SETTINGS ACK")
			}
		case FrameWindowUpdate:
			fmt.Println("<<< Server provided flow control window")
		case FrameGoAway:
			conn.Close()
			return nil, fmt.Errorf("server sent GOAWAY during handshake")
		}
	}

	// Now the actual request sending logic from previous Do method
	authority := req.URL.Host
	if authority == "" {
		authority = req.Host // Fallback if URL.Host is empty
	}
	scheme := "https"
	if req.URL.Scheme != "" {
		scheme = req.URL.Scheme
	}
	headers := []HeaderField{
		{Name: ":method", Value: req.Method},
		{Name: ":scheme", Value: scheme},
		{Name: ":authority", Value: authority},
		{Name: ":path", Value: req.URL.Path},
	}
	for name, values := range req.Header {
		for _, value := range values {
			headers = append(headers, HeaderField{Name: name, Value: value})
		}
	}

	requestPayload := hpackEnc.Encode(headers)

	header := make([]byte, 9)
	payloadLen := len(requestPayload)
	header[0] = byte(payloadLen >> 16)
	header[1] = byte(payloadLen >> 8)
	header[2] = byte(payloadLen)
	header[3] = FrameHeaders
	header[4] = FlagEndStream | FlagEndHeaders
	binary.BigEndian.PutUint32(header[5:9], 1)

	if _, err := conn.Write(append(header, requestPayload...)); err != nil {
		conn.Close()
		return nil, fmt.Errorf("failed to send request: %w", err)
	}
	fmt.Println(">>> Sent HEADERS (Stream 1)")

	// Read response (using the conn local to Do)
	var respHeaders []HeaderField
	var respBody []byte
	for {
		frame, err := ReadFrame(conn) // ReadFrame needs the raw conn
		if err != nil {
			conn.Close()
			return nil, fmt.Errorf("connection closed: %w", err)
		}

		fmt.Printf("<<< Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameData:
			respBody = append(respBody, frame.Payload...)
			fmt.Printf("      [DATA] Length=%d\n", len(frame.Payload))
		case FrameHeaders:
			fmt.Println("      [HEADERS] Decoding...")
			headers, err := hpackDec.Decode(frame.Payload)
			if err != nil {
				conn.Close()
				return nil, fmt.Errorf("hpack error: %w", err)
			}
			respHeaders = append(respHeaders, headers...)
		case FrameGoAway:
			lastStream := binary.BigEndian.Uint32(frame.Payload[0:4]) & 0x7FFFFFFF
			errCode := binary.BigEndian.Uint32(frame.Payload[4:8])
			conn.Close()
			return nil, fmt.Errorf("GOAWAY: Last Stream %d, Error Code %d", lastStream, errCode)
		case FrameWindowUpdate:
			fmt.Println("      [WINDOW_UPDATE]")
		}

		isEndFrame := frame.Header.Type == FrameData || frame.Header.Type == FrameHeaders
		if isEndFrame && (frame.Header.Flags&FlagEndStream != 0) {
			fmt.Println("Stream finished.")
			break
		}
	}

	// Build http.Response
	httpResp := &http.Response{
		StatusCode: 200, // Hardcoded for now
		Proto:      "HTTP/2.0",
		ProtoMajor: 2,
		ProtoMinor: 0,
		Header:     make(http.Header),
		Body:       &responseBody{bytes.NewReader(respBody), conn},
	}

	for _, h := range respHeaders {
		httpResp.Header.Add(h.Name, h.Value)
		if h.Name == ":status" {
			fmt.Sscanf(h.Value, "%d", &httpResp.StatusCode)
		}
	}
	httpResp.Status = fmt.Sprintf("%d %s", httpResp.StatusCode, http.StatusText(httpResp.StatusCode))

	return httpResp, nil
}
go/hpack.go (click to expand)View on GitHub 
package main

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"io"

	"golang.org/x/net/http2/hpack"
)

// RFC 7541: HPACK: Header Compression for HTTP/2
// https://datatracker.ietf.org/doc/html/rfc7541

const (
	// Masks for HPACK header field types
	maskIndexed            = 0x80 // 10000000
	maskLiteralIncremental = 0xc0 // 11000000
	maskDynamicTableSize   = 0xe0 // 11100000
	maskLiteral            = 0xf0 // 11110000

	// Patterns for HPACK header field types
	patternIndexed            = 0x80 // 10000000
	patternLiteralIncremental = 0x40 // 01000000
	patternDynamicTableSize   = 0x20 // 00100000
	patternLiteralNever       = 0x10 // 00010000
	patternLiteral            = 0x00 // 00000000

	HuffmanFlagMask         = 0x80
	IntegerContinuationMask = 0x80
)

// HeaderField represents a header field with a name and value.
type HeaderField struct {
	Name, Value string
}

// StaticTable is the predefined, unchangeable table of header fields, as defined in RFC 7541 Appendix A.
var StaticTable = []HeaderField{
	{Name: ":authority", Value: ""},
	{Name: ":method", Value: "GET"},
	{Name: ":method", Value: "POST"},
	{Name: ":path", Value: "/"},
	{Name: ":path", Value: "/index.html"},
	{Name: ":scheme", Value: "http"},
	{Name: ":scheme", Value: "https"},
	{Name: ":status", Value: "200"},
	{Name: ":status", Value: "204"},
	{Name: ":status", Value: "206"},
	{Name: ":status", Value: "304"},
	{Name: ":status", Value: "400"},
	{Name: ":status", Value: "404"},
	{Name: ":status", Value: "500"},
	{Name: "accept-charset", Value: ""},
	{Name: "accept-encoding", Value: "gzip, deflate"},
	{Name: "accept-language", Value: ""},
	{Name: "accept-ranges", Value: ""},
	{Name: "accept", Value: ""},
	{Name: "access-control-allow-origin", Value: ""},
	{Name: "age", Value: ""},
	{Name: "allow", Value: ""},
	{Name: "authorization", Value: ""},
	{Name: "cache-control", Value: ""},
	{Name: "content-disposition", Value: ""},
	{Name: "content-encoding", Value: ""},
	{Name: "content-language", Value: ""},
	{Name: "content-length", Value: ""},
	{Name: "content-location", Value: ""},
	{Name: "content-range", Value: ""},
	{Name: "content-type", Value: ""},
	{Name: "cookie", Value: ""},
	{Name: "date", Value: ""},
	{Name: "etag", Value: ""},
	{Name: "expect", Value: ""},
	{Name: "expires", Value: ""},
	{Name: "from", Value: ""},
	{Name: "host", Value: ""},
	{Name: "if-match", Value: ""},
	{Name: "if-modified-since", Value: ""},
	{Name: "if-none-match", Value: ""},
	{Name: "if-range", Value: ""},
	{Name: "if-unmodified-since", Value: ""},
	{Name: "last-modified", Value: ""},
	{Name: "link", Value: ""},
	{Name: "location", Value: ""},
	{Name: "max-forwards", Value: ""},
	{Name: "proxy-authenticate", Value: ""},
	{Name: "proxy-authorization", Value: ""},
	{Name: "range", Value: ""},
	{Name: "referer", Value: ""},
	{Name: "retry-after", Value: ""},
	{Name: "server", Value: ""},
	{Name: "set-cookie", Value: ""},
	{Name: "strict-transport-security", Value: ""},
	{Name: "transfer-encoding", Value: ""},
	{Name: "user-agent", Value: ""},
	{Name: "vary", Value: ""},
	{Name: "via", Value: ""},
	{Name: "www-authenticate", Value: ""},
}

var (
	staticTableMap     = make(map[HeaderField]int)
	staticTableNameMap = make(map[string]int)
)

func init() {
	for i, hf := range StaticTable {
		staticTableMap[hf] = i + 1
		if _, ok := staticTableNameMap[hf.Name]; !ok {
			staticTableNameMap[hf.Name] = i + 1
		}
	}
}

type DynamicTable struct {
	headers []HeaderField
	size    uint32
	maxSize uint32
}

func NewDynamicTable(maxSize uint32) *DynamicTable {
	return &DynamicTable{
		maxSize: maxSize,
	}
}

func (d *DynamicTable) At(i int) (HeaderField, bool) {
	if i < 0 || i >= len(d.headers) {
		return HeaderField{}, false
	}
	return d.headers[i], true
}

func (d *DynamicTable) Add(h HeaderField) {
	size := uint32(len(h.Name) + len(h.Value) + 32)
	for d.size+size > d.maxSize && len(d.headers) > 0 {
		last := d.headers[len(d.headers)-1]
		d.size -= uint32(len(last.Name) + len(last.Value) + 32)
		d.headers = d.headers[:len(d.headers)-1]
	}
	d.headers = append([]HeaderField{h}, d.headers...)
	d.size += size
}

func (d *DynamicTable) SetMaxSize(size uint32) {
	d.maxSize = size
	for d.size > d.maxSize && len(d.headers) > 0 {
		last := d.headers[len(d.headers)-1]
		d.size -= uint32(len(last.Name) + len(last.Value) + 32)
		d.headers = d.headers[:len(d.headers)-1]
	}
}

type HPACKDecoder struct {
	dynamicTable *DynamicTable
}

func NewHPACKDecoder(maxSize uint32) *HPACKDecoder {
	return &HPACKDecoder{
		dynamicTable: NewDynamicTable(maxSize),
	}
}

func (h *HPACKDecoder) Header(i int) (HeaderField, bool) {
	if i <= 0 {
		return HeaderField{}, false
	}
	if i <= len(StaticTable) {
		return StaticTable[i-1], true
	}
	return h.dynamicTable.At(i - len(StaticTable) - 1)
}

func (h *HPACKDecoder) Decode(payload []byte) ([]HeaderField, error) {
	var headers []HeaderField
	r := bytes.NewReader(payload)
	for r.Len() > 0 {
		b, _ := r.ReadByte()
		if b&maskIndexed == patternIndexed { // Indexed Header Field
			index, n := decodeInt(b, r, 7)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, ok := h.Header(index)
			if !ok {
				return nil, fmt.Errorf("invalid header index: %d", index)
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskLiteralIncremental == patternLiteralIncremental { // Literal Header Field with Incremental Indexing
			index, n := decodeInt(b, r, 6)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, err := h.decodeLiteralHeader(r, index, true)
			if err != nil {
				return nil, err
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskLiteral == patternLiteral || b&maskLiteral == patternLiteralNever { // Literal Header Field without or never indexed
			index, n := decodeInt(b, r, 4)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			header, err := h.decodeLiteralHeader(r, index, false)
			if err != nil {
				return nil, err
			}
			headers = append(headers, header)
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else if b&maskDynamicTableSize == patternDynamicTableSize { // Dynamic Table Size Update
			size, n := decodeInt(b, r, 5)
			if n < 0 {
				return nil, fmt.Errorf("failed to decode integer")
			}
			h.dynamicTable.SetMaxSize(uint32(size))
		} else {
			return nil, fmt.Errorf("not implemented: unknown header field type %08b", b)
		}
	}
	return headers, nil
}

func (h *HPACKDecoder) decodeLiteralHeader(r *bytes.Reader, index int, addToDynamicTable bool) (HeaderField, error) {
	var name string
	var err error
	if index > 0 {
		header, ok := h.Header(index)
		if !ok {
			return HeaderField{}, fmt.Errorf("invalid header index: %d", index)
		}
		name = header.Name
	} else {
		name, err = h.decodeString(r)
		if err != nil {
			return HeaderField{}, err
		}
	}
	value, err := h.decodeString(r)
	if err != nil {
		return HeaderField{}, err
	}
	header := HeaderField{Name: name, Value: value}
	if addToDynamicTable {
		h.dynamicTable.Add(header)
	}
	return header, nil
}

func (h *HPACKDecoder) decodeString(r *bytes.Reader) (string, error) {
	b, _ := r.ReadByte()
	huffman := b&HuffmanFlagMask == HuffmanFlagMask
	length, n := decodeInt(b, r, 7)
	if n < 0 {
		return "", fmt.Errorf("failed to decode integer")
	}
	if r.Len() < length {
		return "", io.ErrUnexpectedEOF
	}
	data := make([]byte, length)
	r.Read(data)
	if huffman {
		return hpack.HuffmanDecodeToString(data)
	}
	return string(data), nil
}

func decodeInt(b byte, r *bytes.Reader, n int) (int, int) {
	mask := (1 << n) - 1
	i := int(b) & mask
	if i < mask {
		return i, 1
	}

	var m uint = 0
	bytesRead := 1
	var val uint64
	for {
		b, err := r.ReadByte()
		if err != nil {
			return 0, -bytesRead
		}
		bytesRead++
		val |= uint64(b&127) << m
		m += 7
		if b&IntegerContinuationMask == 0 {
			break
		}
	}
	return i + int(val), bytesRead
}

type HPACKEncoder struct {
	dynamicTable *DynamicTable
}

func NewHPACKEncoder(maxSize uint32) *HPACKEncoder {
	return &HPACKEncoder{
		dynamicTable: NewDynamicTable(maxSize),
	}
}

func (e *HPACKEncoder) Encode(headers []HeaderField) []byte {
	var buf bytes.Buffer
	for _, hf := range headers {
		// Find a match in static table
		if index, ok := staticTableMap[hf]; ok {
			encodeInt(&buf, index, 7, patternIndexed)
			continue
		}

		// Find a name match in static table
		if index, ok := staticTableNameMap[hf.Name]; ok {
			encodeInt(&buf, index, 6, patternLiteralIncremental)
			encodeString(&buf, hf.Value)
			e.dynamicTable.Add(hf)
			continue
		}

		// Literal with literal name
		encodeInt(&buf, 0, 6, patternLiteralIncremental)
		encodeString(&buf, hf.Name)
		encodeString(&buf, hf.Value)
		e.dynamicTable.Add(hf)
	}
	return buf.Bytes()
}

func encodeInt(buf *bytes.Buffer, i int, n int, pattern byte) {
	mask := (1 << n) - 1
	if i < mask {
		buf.WriteByte(pattern | byte(i))
	} else {
		buf.WriteByte(pattern | byte(mask))
		i -= mask
		varint := make([]byte, binary.MaxVarintLen64)
		c := binary.PutUvarint(varint, uint64(i))
		buf.Write(varint[:c])
	}
}

func encodeString(buf *bytes.Buffer, s string) {
	// no huffman for now
	encodeInt(buf, len(s), 7, 0x00) // This is patternLiteral. We are encoding a raw string.
	buf.WriteString(s)
}
go/main.go (click to expand)View on GitHub 
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
)

func main() {
	client := NewClient()

	req, err := http.NewRequest("GET", "https://kmcd.dev/", nil)
	if err != nil {
		log.Fatalf("Failed to create request: %v", err)
	}

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Failed to execute request: %v", err)
	}
	defer resp.Body.Close()

	fmt.Printf("Protocol: %s\n", resp.Proto)

	fmt.Println("Response Status:", resp.Status)
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Failed to read body: %v", err)
	}
	fmt.Printf("Response Body: %s\n", string(body))
}
go/parser.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
	"io"
)

// FrameHeader represents the 9-byte fixed header of every HTTP/2 frame.
type FrameHeader struct {
	Length   uint32
	Type     uint8
	Flags    uint8
	StreamID uint32
}

// Frame represents a complete HTTP/2 frame including its payload.
type Frame struct {
	Header  FrameHeader
	Payload []byte
}

// ReadFrame reads a header and then the corresponding payload from the connection.
func ReadFrame(r io.Reader) (Frame, error) {
	// Read the 9-byte header
	headerBuf := make([]byte, 9)
	_, err := io.ReadFull(r, headerBuf)
	if err != nil {
		return Frame{}, fmt.Errorf("reading header: %w", err)
	}

	// Parse the header fields using bit-shifting
	header := FrameHeader{
		Length:   uint32(headerBuf[0])<<16 | uint32(headerBuf[1])<<8 | uint32(headerBuf[2]),
		Type:     headerBuf[3],
		Flags:    headerBuf[4],
		StreamID: binary.BigEndian.Uint32(headerBuf[5:9]) & 0x7FFFFFFF,
	}

	// Read the payload based on the Length field
	payload := make([]byte, header.Length)
	if header.Length > 0 {
		_, err = io.ReadFull(r, payload)
		if err != nil {
			return Frame{}, fmt.Errorf("reading payload: %w", err)
		}
	}

	return Frame{Header: header, Payload: payload}, nil
}

]]>IRC Log: rm -rf /var/opt/gitlab/postgresql/datahttps://kmcd.dev/posts/irc-log-rm-rf-/var/opt/gitlab/postgresql/data/Mon, 09 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/irc-log-rm-rf-/var/opt/gitlab/postgresql/data/ <p> <img hspace="5" src="https://kmcd.dev/posts/irc-log-rm-rf-/var/opt/gitlab/postgresql/data/cover.svg" /> </p> A tired sysadmin accidentally deletes the production database and discovers that all five backup mechanisms have failed. The Incident Log: January 31, 2017
[23:00] *** Topic: DB Replication Lag | Status: 🔴 Critical
[23:05] *** tired_sysadmin has joined
[23:10] <tired_sysadmin> Replication is stuck again. The secondary node (db2) is refusing to sync.
[23:11] <tired_sysadmin> I’m going to wipe the data directory on db2 and let it pull a fresh copy from master.
[23:12] <tired_sysadmin> rm -rf /var/opt/gitlab/postgresql/data
[23:12] <tired_sysadmin> Weird. It’s taking a while. Usually empty directories delete instantly.
[23:13] <helper_dev> Hey, why did the website just go 500?
[23:13] <tired_sysadmin> …
[23:13] <tired_sysadmin> I’m looking at my terminal prompt.
[23:14] <tired_sysadmin> It says root@db1.
[23:14] <helper_dev> db1 is Prod. You are deleting Prod.
[23:15] <tired_sysadmin> CTRL+C CTRL+C CTRL+C
[23:15] <tired_sysadmin> Okay, I stopped it. How much is left?
[23:16] <helper_dev> Checking… The directory is 4.5KB.
[23:16] <tired_sysadmin> We had 300GB of data.
[23:17] <helper_dev> Okay, don’t panic. We have 5 different backup mechanisms. Let’s check S3.
[23:20] <helper_dev> S3 bucket is empty. The backup script has been failing silently since version 8.1.
[23:21] <tired_sysadmin> Check the Azure disk snapshots.
[23:22] <helper_dev> Not enabled.
[23:23] <tired_sysadmin> …LVM snapshots?
[23:24] <helper_dev> We take them every 24 hours. We just lost 6 hours of data.
[23:25] <tired_sysadmin> I am going to live stream the restoration on YouTube so people don’t kill us.

Postmortem of database outage of January 31

]]>HTTP/2 From Scratch: Part 3https://kmcd.dev/posts/http2-from-scratch-part-3/Wed, 04 Mar 2026 10:00:00 +0000https://kmcd.dev/posts/http2-from-scratch-part-3/ <p> <img hspace="5" src="https://kmcd.dev/posts/http2-from-scratch-part-3/cover.svg" /> </p> Decoding HPACK and the evolution of the HTTP header In the last two posts, we established a raw TCP connection, navigated the TLS handshake with ALPN to select “h2”, and built a parser that can read the 9-byte frames of an HTTP/2 connection. We have a synchronized, acknowledged connection. Now it’s time to do what we came for: request a web page.

This is where HTTP/2 departs dramatically from its predecessor. There is no GET / HTTP/1.1. Instead, we enter the world of compressed headers, pseudo-headers, and stateful tables. This is the world of HPACK: Header Compression for HTTP/2.

What is HPACK?

In HTTP/1.1, headers are human-readable text. This is great for debugging but inefficient. The same headers (like User-Agent) are sent with every single request, wasting bandwidth. HPACK (RFC 7541) solves this by using several compression strategies. Instead of sending full header names and values, it sends compact, indexed representations.

At its core, HPACK uses two tables to translate between full headers and small integer indices:

  1. Static Table: A predefined, read-only table containing 61 of the most common headers. For example, {':method', 'GET'} is entry #2. Every HTTP/2 client and server knows this table.
  2. Dynamic Table: A small, temporary table that is specific to a single connection. In a full HPACK implementation, if you send a header that’s not in the static table (like a custom x-request-id), it can be added to the dynamic table. On the next request, you can just send its index instead of the full header again. For this part, we will focus solely on the Static Table.

Our journey into HPACK will start with the basics. We’ll implement a decoder that understands the static table and indexed headers. We’ll leave the dynamic table handling for the next article.

Decoding Indexed Headers

The simplest form of header compression is the Indexed Header Field. When a header to be sent is present in one of the tables, it can be represented by a single integer.

An indexed header byte starts with a 1. The remaining 7 bits are the start of a variable-length integer representing the index in the tables.

Let’s look at our hpack.go file. The Decode method reads the payload from a HEADERS frame. If it sees a byte starting with 1, it knows it’s an indexed header.

func (h *HPACKDecoder) Decode(payload []byte) error {
	fmt.Printf("Decoding %d bytes\n", len(payload))
	for len(payload) > 0 {
		b := payload[0]
		if b&128 == 128 { // Indexed Header Field (starts with 1)
			index, n := decodeInt(payload, 7)
			if n < 0 {
				return fmt.Errorf("failed to decode integer")
			}
			payload = payload[n:]
			header, ok := h.Header(index)
			if !ok {
				return fmt.Errorf("invalid header index: %d", index)
			}
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else {
			// Other header types (like literals) are not implemented yet.
			// We'll tackle this in the next part.
			return fmt.Errorf("not implemented: literal header field")
		}
	}
	return nil
}

The decodeInt function handles HPACK’s specific integer encoding. It parses the first byte manually to account for the n-bit prefix. If the integer overflows that first byte, it uses the standard library’s binary.Uvarint to efficiently parse the remaining continuation bytes. This hybrid approach correctly handles the HPACK format while leveraging Go’s optimized, built-in varint decoder.

With this, our decoder can parse headers that are in the static table. The hpack.go file contains the full static table definition.

go/hpack.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
)

// RFC 7541: HPACK: Header Compression for HTTP/2
// https://datatracker.ietf.org/doc/html/rfc7541

const (
	// HPACK Static Table Indices (Masked with 0x80 for Indexed Header Fields)
	// See RFC 7541 Appendix A
	HpackMethodGet   uint8 = 0x82 // Index 2: :method: GET
	HpackPathRoot    uint8 = 0x84 // Index 4: :path: /
	HpackSchemeHttps uint8 = 0x87 // Index 7: :scheme: https

	// 0x40 is the mask for Literal Header Field with Incremental Indexing
	HpackAuthority uint8 = 0x40 | 1 // Index 1: :authority
)

// HeaderField represents a header field with a name and value.
type HeaderField struct {
	Name, Value string
}

// StaticTable is the predefined, unchangeable table of header fields, as defined in RFC 7541 Appendix A.
var StaticTable = []HeaderField{
	{Name: ":authority", Value: ""},
	{Name: ":method", Value: "GET"},
	{Name: ":method", Value: "POST"},
	{Name: ":path", Value: "/"},
	{Name: ":path", Value: "/index.html"},
	{Name: ":scheme", Value: "http"},
	{Name: ":scheme", Value: "https"},
	{Name: ":status", Value: "200"},
	{Name: ":status", Value: "204"},
	{Name: ":status", Value: "206"},
	{Name: ":status", Value: "304"},
	{Name: ":status", Value: "400"},
	{Name: ":status", Value: "404"},
	{Name: ":status", Value: "500"},
	{Name: "accept-charset", Value: ""},
	{Name: "accept-encoding", Value: "gzip, deflate"},
	{Name: "accept-language", Value: ""},
	{Name: "accept-ranges", Value: ""},
	{Name: "accept", Value: ""},
	{Name: "access-control-allow-origin", Value: ""},
	{Name: "age", Value: ""},
	{Name: "allow", Value: ""},
	{Name: "authorization", Value: ""},
	{Name: "cache-control", Value: ""},
	{Name: "content-disposition", Value: ""},
	{Name: "content-encoding", Value: ""},
	{Name: "content-language", Value: ""},
	{Name: "content-length", Value: ""},
	{Name: "content-location", Value: ""},
	{Name: "content-range", Value: ""},
	{Name: "content-type", Value: ""},
	{Name: "cookie", Value: ""},
	{Name: "date", Value: ""},
	{Name: "etag", Value: ""},
	{Name: "expect", Value: ""},
	{Name: "expires", Value: ""},
	{Name: "from", Value: ""},
	{Name: "host", Value: ""},
	{Name: "if-match", Value: ""},
	{Name: "if-modified-since", Value: ""},
	{Name: "if-none-match", Value: ""},
	{Name: "if-range", Value: ""},
	{Name: "if-unmodified-since", Value: ""},
	{Name: "last-modified", Value: ""},
	{Name: "link", Value: ""},
	{Name: "location", Value: ""},
	{Name: "max-forwards", Value: ""},
	{Name: "proxy-authenticate", Value: ""},
	{Name: "proxy-authorization", Value: ""},
	{Name: "range", Value: ""},
	{Name: "referer", Value: ""},
	{Name: "retry-after", Value: ""},
	{Name: "server", Value: ""},
	{Name: "set-cookie", Value: ""},
	{Name: "strict-transport-security", Value: ""},
	{Name: "transfer-encoding", Value: ""},
	{Name: "user-agent", Value: ""},
	{Name: "vary", Value: ""},
	{Name: "via", Value: ""},
	{Name: "www-authenticate", Value: ""},
}

var (
	staticTableMap = make(map[HeaderField]int)
)

func init() {
	for i, hf := range StaticTable {
		staticTableMap[hf] = i + 1
	}
}

type HPACKDecoder struct{}

func NewHPACKDecoder() *HPACKDecoder {
	return &HPACKDecoder{}
}

func (h *HPACKDecoder) Header(i int) (HeaderField, bool) {
	if i <= 0 {
		return HeaderField{}, false
	}
	staticIndex := i - 1
	if staticIndex < len(StaticTable) {
		return StaticTable[staticIndex], true
	}
	return HeaderField{}, false
}

func (h *HPACKDecoder) Decode(payload []byte) error {
	fmt.Printf("Decoding %d bytes\n", len(payload))
	for len(payload) > 0 {
		b := payload[0]
		if b&128 == 128 { // Indexed Header Field
			index, n := decodeInt(payload, 7)
			if n < 0 {
				return fmt.Errorf("failed to decode integer")
			}
			payload = payload[n:]
			header, ok := h.Header(index)
			if !ok {
				return fmt.Errorf("invalid header index: %d", index)
			}
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else {
			// Other header field types (literal, etc.) not implemented yet
			return fmt.Errorf("not implemented: literal header field")
		}
	}
	return nil
}

// decodeInt decodes a variable-length integer from a byte slice.
// It returns the decoded integer and the number of bytes consumed.
// See RFC 7541 section 5.1 for details.
func decodeInt(payload []byte, n int) (int, int) {
	if len(payload) == 0 {
		return 0, -1
	}
	mask := (1 << n) - 1
	i := int(payload[0]) & mask
	if i < mask {
		return i, 1
	}

	// The value overflows the first byte. The rest of the integer is a
	// standard varint.
	val, bytesRead := binary.Uvarint(payload[1:])
	if bytesRead <= 0 {
		return 0, -1 // Malformed varint
	}

	return i + int(val), 1 + bytesRead
}

Manually Encoding Our First Request

Our current client doesn’t have an HPACK encoder. To send our first request, we’re going to manually craft the byte payload for our HEADERS frame. This is a great way to understand how encoding works.

We want to send the following headers for a GET / request:

  • :method: GET
  • :path: /
  • :scheme: https
  • :authority: kmcd.dev

Looking at the static table in RFC 7541, Appendix A:

  • {':method', 'GET'} is at index 2.
  • {':path', '/'} is at index 4.
  • {':scheme', 'https'} is at index 7.

The :authority header doesn’t have a perfect match for both name and value. However, its name is at index 1. This means we have to send it as a Literal Header Field. This type of header representation has a prefix indicating how it should be handled. For now, we will use “Literal Header Field with Incremental Indexing” (prefix 0100), which tells the server to use this header and add it to the dynamic table. We will do this manually for now, but in the future we will want to create code to manage this for us.

Our client.go assembles this payload:

// content/posts/2026/http2-from-scratch-part-3/go/client.go

// ...
	authority := "kmcd.dev"
	requestPayload := []byte{
		0x82, // Index 2: :method: GET
		0x84, // Index 4: :path: /
		0x87, // Index 7: :scheme: https
		0x41, // Index 1 for :authority, with literal value
		byte(len(authority)), // Length of "kmcd.dev"
	}
	requestPayload = append(requestPayload, []byte(authority)...)
// ...

Let’s break down the bytes:

  • 0x82: 10000010. Starts with 1, so it’s an indexed header. The integer value is 2. This is :method: GET.
  • 0x84: 10000100. Indexed header, index 4. This is :path: /.
  • 0x87: 10000111. Indexed header, index 7. This is :scheme: https.
  • 0x41: 01000001. Starts with 01, so it’s a “Literal Header Field with Incremental Indexing”. The remaining 6 bits are the index for the name, which is 1 (:authority). The value will follow as a literal string.
  • byte(len(authority)): The length of the value “kmcd.dev”, which is 8.
  • []byte(authority): The string “kmcd.dev” itself.

We have now manually encoded a HEADERS payload!

The Client’s Structure

Here is a high-level overview of our client’s logic so far:

  1. Connect & Handshake: Establish a TCP connection and perform the TLS handshake, using ALPN to negotiate “h2”.
  2. Send Preface: Send the magic PRI * ... connection preface.
  3. Exchange Settings: Send our empty SETTINGS frame, wait for the server’s SETTINGS frame, and then send an ACK.
  4. Send Request: Construct and send the HEADERS frame for GET / with our manually encoded HPACK payload. We set the END_STREAM flag to indicate this is our entire request.
  5. Read Response: Loop and read frames from the server.
    • If it’s a HEADERS frame, use our HPACKDecoder to parse and print the response headers.
    • If it’s a DATA frame, print the content.
    • If we see a frame with the END_STREAM flag, the response is complete, and we exit.

This gives us a working end-to-end client that makes a real HTTP/2 request and prints the response.

Putting It All Together: The Final Client

Here is the full client.go script.

go/client.go (click to expand)View on GitHub 
package main

import (
	"crypto/tls"
	"encoding/binary"
	"fmt"
	"log"
)

const (
	// Protocol constants
	Preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	Server  = "kmcd.dev:443"

	// Frame Types (RFC 9113 Section 6)
	FrameData         uint8 = 0x0
	FrameHeaders      uint8 = 0x1
	FramePriority     uint8 = 0x2
	FrameRstStream    uint8 = 0x3
	FrameSettings     uint8 = 0x4
	FramePushPromise  uint8 = 0x5
	FramePing         uint8 = 0x6
	FrameGoAway       uint8 = 0x7
	FrameWindowUpdate uint8 = 0x8
	FrameContinuation uint8 = 0x9

	// Common Flags
	FlagAck        uint8 = 0x01 // For SETTINGS/PING
	FlagEndStream  uint8 = 0x01 // For DATA/HEADERS
	FlagEndHeaders uint8 = 0x04 // For HEADERS/PUSH_PROMISE/CONTINUATION
)

func main() {
	// 1. Setup TLS with ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	conn, err := tls.Dial("tcp", Server, config)
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		log.Fatalf("Server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", Server, state.NegotiatedProtocol)

	// 2. Send Connection Preface
	if _, err = conn.Write([]byte(Preface)); err != nil {
		log.Fatalf("Failed to send preface: %v", err)
	}
	fmt.Println("Preface sent.")

	// 3. Initial Handshake Loop
	hpackDec := NewHPACKDecoder()

	// Send our initial empty settings
	mySettings := []byte{0, 0, 0, FrameSettings, 0, 0, 0, 0, 0}
	conn.Write(mySettings)

	serverSettingsAcked := false
	mySettingsAcked := false

	for !serverSettingsAcked || !mySettingsAcked {
		frame, err := ReadFrame(conn)
		if err != nil {
			log.Fatalf("Handshake read error: %v", err)
		}

		fmt.Printf("<<< [Handshake] Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameSettings:
			if frame.Header.Flags&FlagAck != 0 {
				mySettingsAcked = true
				fmt.Println("<<< Server ACK'd our settings")
			} else {
				ack := []byte{0, 0, 0, FrameSettings, FlagAck, 0, 0, 0, 0}
				conn.Write(ack)
				serverSettingsAcked = true
				fmt.Println(">>> Sent SETTINGS ACK")
			}
		case FrameWindowUpdate:
			fmt.Println("<<< Server provided flow control window")
		case FrameGoAway:
			log.Fatalf("Server sent GOAWAY during handshake")
		}
	}

	// 4. Send the Request (GET /)
	authority := "kmcd.dev"
	requestPayload := []byte{
		HpackMethodGet,
		HpackPathRoot,
		HpackSchemeHttps,
		HpackAuthority,
		byte(len(authority)), // The length prefix for the string literal
	}
	requestPayload = append(requestPayload, []byte(authority)...)

	header := make([]byte, 9)
	payloadLen := len(requestPayload)
	header[0] = byte(payloadLen >> 16)
	header[1] = byte(payloadLen >> 8)
	header[2] = byte(payloadLen)
	header[3] = FrameHeaders
	header[4] = FlagEndStream | FlagEndHeaders
	binary.BigEndian.PutUint32(header[5:9], 1)

	if _, err = conn.Write(append(header, requestPayload...)); err != nil {
		log.Fatalf("Failed to send request: %v", err)
	}
	fmt.Println(">>> Sent HEADERS (Stream 1)")

	// 5. Main Processing Loop
	for {
		frame, err := ReadFrame(conn)
		if err != nil {
			log.Printf("Connection closed: %v", err)
			break
		}

		fmt.Printf("<<< Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameData:
			fmt.Printf("      [DATA] %s\n", string(frame.Payload))
		case FrameHeaders:
			fmt.Println("      [HEADERS] Decoding...")
			if err := hpackDec.Decode(frame.Payload); err != nil {
				log.Printf("HPACK Error: %v", err)
			}
		case FrameGoAway:
			lastStream := binary.BigEndian.Uint32(frame.Payload[0:4]) & 0x7FFFFFFF
			errCode := binary.BigEndian.Uint32(frame.Payload[4:8])
			fmt.Printf("!!! GOAWAY: Last Stream %d, Error Code %d\n", lastStream, errCode)
			return
		case FrameWindowUpdate:
			fmt.Println("      [WINDOW_UPDATE]")
		}

		// Use the global constants for the exit condition
		isEndFrame := frame.Header.Type == FrameData || frame.Header.Type == FrameHeaders
		if isEndFrame && (frame.Header.Flags&FlagEndStream != 0) {
			fmt.Println("Stream finished. Exiting.")
			break
		}
	}
}

Running this client produces a full HTTP/2 interaction. We send our request and get back headers and data from the server, all parsed by our own code. Here’s what it looks like when we run it (body truncated to reduce noise):

$ go run .
Connected to kmcd.dev:443 using h2
Preface sent.
<<< [Handshake] Frame Type=4, Flags=0, Stream=0
>>> Sent SETTINGS ACK
<<< [Handshake] Frame Type=8, Flags=0, Stream=0
<<< Server provided flow control window
<<< [Handshake] Frame Type=4, Flags=1, Stream=0
<<< Server ACK'd our settings
>>> Sent HEADERS (Stream 1)
<<< Frame Type=1, Flags=4, Stream=1
      [HEADERS] Decoding...
Decoding 705 bytes
  [Header] :status: 200
2026/01/31 14:36:00 HPACK Error: not implemented: literal header field
<<< Frame Type=0, Flags=0, Stream=1
      [DATA] <!doctype html><html lang=en>[...snipped...]</html>
<<< Frame Type=0, Flags=1, Stream=1
      [DATA]

I want you to notice a few things here. We successfully make it through the initial handshake. The server sends us headers and we decode only a single header, the :status pseudo-header that tells us that the response is a 200 but the next header doesn’t exist in the status table and is sent as a string literal and since our code doesn’t yet handle string literals we have to stop processing the headers at this point. This will be improved later. Finally, I want you to notice that we have successfully received the DATA frame. We actually receive two of them: one that contains the data and another that says that we have received all of the data for the request. This is a significant improvement. We are very close to a fully functioning client!

What’s Next?

We’ve made a huge leap. We can now make requests and parse simple responses. However, as you just saw, our HPACK decoder is incomplete. In this part, we’ve focused on decoding Indexed Header Fields that refer exclusively to the Static Table. This means our client will successfully decode common headers like :method: GET or :path: / but will completely fail on any string literal or any reference to the Dynamic Table.

In the next and final part covering HTTP/2, we will complete our HPACK implementation and adapt our client to use the http.Request and http.Response types.

See all of the code mentioned in this article here:

go/client.go (click to expand)View on GitHub 
package main

import (
	"crypto/tls"
	"encoding/binary"
	"fmt"
	"log"
)

const (
	// Protocol constants
	Preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	Server  = "kmcd.dev:443"

	// Frame Types (RFC 9113 Section 6)
	FrameData         uint8 = 0x0
	FrameHeaders      uint8 = 0x1
	FramePriority     uint8 = 0x2
	FrameRstStream    uint8 = 0x3
	FrameSettings     uint8 = 0x4
	FramePushPromise  uint8 = 0x5
	FramePing         uint8 = 0x6
	FrameGoAway       uint8 = 0x7
	FrameWindowUpdate uint8 = 0x8
	FrameContinuation uint8 = 0x9

	// Common Flags
	FlagAck        uint8 = 0x01 // For SETTINGS/PING
	FlagEndStream  uint8 = 0x01 // For DATA/HEADERS
	FlagEndHeaders uint8 = 0x04 // For HEADERS/PUSH_PROMISE/CONTINUATION
)

func main() {
	// 1. Setup TLS with ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	conn, err := tls.Dial("tcp", Server, config)
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		log.Fatalf("Server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", Server, state.NegotiatedProtocol)

	// 2. Send Connection Preface
	if _, err = conn.Write([]byte(Preface)); err != nil {
		log.Fatalf("Failed to send preface: %v", err)
	}
	fmt.Println("Preface sent.")

	// 3. Initial Handshake Loop
	hpackDec := NewHPACKDecoder()

	// Send our initial empty settings
	mySettings := []byte{0, 0, 0, FrameSettings, 0, 0, 0, 0, 0}
	conn.Write(mySettings)

	serverSettingsAcked := false
	mySettingsAcked := false

	for !serverSettingsAcked || !mySettingsAcked {
		frame, err := ReadFrame(conn)
		if err != nil {
			log.Fatalf("Handshake read error: %v", err)
		}

		fmt.Printf("<<< [Handshake] Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameSettings:
			if frame.Header.Flags&FlagAck != 0 {
				mySettingsAcked = true
				fmt.Println("<<< Server ACK'd our settings")
			} else {
				ack := []byte{0, 0, 0, FrameSettings, FlagAck, 0, 0, 0, 0}
				conn.Write(ack)
				serverSettingsAcked = true
				fmt.Println(">>> Sent SETTINGS ACK")
			}
		case FrameWindowUpdate:
			fmt.Println("<<< Server provided flow control window")
		case FrameGoAway:
			log.Fatalf("Server sent GOAWAY during handshake")
		}
	}

	// 4. Send the Request (GET /)
	authority := "kmcd.dev"
	requestPayload := []byte{
		HpackMethodGet,
		HpackPathRoot,
		HpackSchemeHttps,
		HpackAuthority,
		byte(len(authority)), // The length prefix for the string literal
	}
	requestPayload = append(requestPayload, []byte(authority)...)

	header := make([]byte, 9)
	payloadLen := len(requestPayload)
	header[0] = byte(payloadLen >> 16)
	header[1] = byte(payloadLen >> 8)
	header[2] = byte(payloadLen)
	header[3] = FrameHeaders
	header[4] = FlagEndStream | FlagEndHeaders
	binary.BigEndian.PutUint32(header[5:9], 1)

	if _, err = conn.Write(append(header, requestPayload...)); err != nil {
		log.Fatalf("Failed to send request: %v", err)
	}
	fmt.Println(">>> Sent HEADERS (Stream 1)")

	// 5. Main Processing Loop
	for {
		frame, err := ReadFrame(conn)
		if err != nil {
			log.Printf("Connection closed: %v", err)
			break
		}

		fmt.Printf("<<< Frame Type=%d, Flags=%d, Stream=%d\n",
			frame.Header.Type, frame.Header.Flags, frame.Header.StreamID)

		switch frame.Header.Type {
		case FrameData:
			fmt.Printf("      [DATA] %s\n", string(frame.Payload))
		case FrameHeaders:
			fmt.Println("      [HEADERS] Decoding...")
			if err := hpackDec.Decode(frame.Payload); err != nil {
				log.Printf("HPACK Error: %v", err)
			}
		case FrameGoAway:
			lastStream := binary.BigEndian.Uint32(frame.Payload[0:4]) & 0x7FFFFFFF
			errCode := binary.BigEndian.Uint32(frame.Payload[4:8])
			fmt.Printf("!!! GOAWAY: Last Stream %d, Error Code %d\n", lastStream, errCode)
			return
		case FrameWindowUpdate:
			fmt.Println("      [WINDOW_UPDATE]")
		}

		// Use the global constants for the exit condition
		isEndFrame := frame.Header.Type == FrameData || frame.Header.Type == FrameHeaders
		if isEndFrame && (frame.Header.Flags&FlagEndStream != 0) {
			fmt.Println("Stream finished. Exiting.")
			break
		}
	}
}
go/hpack.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
)

// RFC 7541: HPACK: Header Compression for HTTP/2
// https://datatracker.ietf.org/doc/html/rfc7541

const (
	// HPACK Static Table Indices (Masked with 0x80 for Indexed Header Fields)
	// See RFC 7541 Appendix A
	HpackMethodGet   uint8 = 0x82 // Index 2: :method: GET
	HpackPathRoot    uint8 = 0x84 // Index 4: :path: /
	HpackSchemeHttps uint8 = 0x87 // Index 7: :scheme: https

	// 0x40 is the mask for Literal Header Field with Incremental Indexing
	HpackAuthority uint8 = 0x40 | 1 // Index 1: :authority
)

// HeaderField represents a header field with a name and value.
type HeaderField struct {
	Name, Value string
}

// StaticTable is the predefined, unchangeable table of header fields, as defined in RFC 7541 Appendix A.
var StaticTable = []HeaderField{
	{Name: ":authority", Value: ""},
	{Name: ":method", Value: "GET"},
	{Name: ":method", Value: "POST"},
	{Name: ":path", Value: "/"},
	{Name: ":path", Value: "/index.html"},
	{Name: ":scheme", Value: "http"},
	{Name: ":scheme", Value: "https"},
	{Name: ":status", Value: "200"},
	{Name: ":status", Value: "204"},
	{Name: ":status", Value: "206"},
	{Name: ":status", Value: "304"},
	{Name: ":status", Value: "400"},
	{Name: ":status", Value: "404"},
	{Name: ":status", Value: "500"},
	{Name: "accept-charset", Value: ""},
	{Name: "accept-encoding", Value: "gzip, deflate"},
	{Name: "accept-language", Value: ""},
	{Name: "accept-ranges", Value: ""},
	{Name: "accept", Value: ""},
	{Name: "access-control-allow-origin", Value: ""},
	{Name: "age", Value: ""},
	{Name: "allow", Value: ""},
	{Name: "authorization", Value: ""},
	{Name: "cache-control", Value: ""},
	{Name: "content-disposition", Value: ""},
	{Name: "content-encoding", Value: ""},
	{Name: "content-language", Value: ""},
	{Name: "content-length", Value: ""},
	{Name: "content-location", Value: ""},
	{Name: "content-range", Value: ""},
	{Name: "content-type", Value: ""},
	{Name: "cookie", Value: ""},
	{Name: "date", Value: ""},
	{Name: "etag", Value: ""},
	{Name: "expect", Value: ""},
	{Name: "expires", Value: ""},
	{Name: "from", Value: ""},
	{Name: "host", Value: ""},
	{Name: "if-match", Value: ""},
	{Name: "if-modified-since", Value: ""},
	{Name: "if-none-match", Value: ""},
	{Name: "if-range", Value: ""},
	{Name: "if-unmodified-since", Value: ""},
	{Name: "last-modified", Value: ""},
	{Name: "link", Value: ""},
	{Name: "location", Value: ""},
	{Name: "max-forwards", Value: ""},
	{Name: "proxy-authenticate", Value: ""},
	{Name: "proxy-authorization", Value: ""},
	{Name: "range", Value: ""},
	{Name: "referer", Value: ""},
	{Name: "retry-after", Value: ""},
	{Name: "server", Value: ""},
	{Name: "set-cookie", Value: ""},
	{Name: "strict-transport-security", Value: ""},
	{Name: "transfer-encoding", Value: ""},
	{Name: "user-agent", Value: ""},
	{Name: "vary", Value: ""},
	{Name: "via", Value: ""},
	{Name: "www-authenticate", Value: ""},
}

var (
	staticTableMap = make(map[HeaderField]int)
)

func init() {
	for i, hf := range StaticTable {
		staticTableMap[hf] = i + 1
	}
}

type HPACKDecoder struct{}

func NewHPACKDecoder() *HPACKDecoder {
	return &HPACKDecoder{}
}

func (h *HPACKDecoder) Header(i int) (HeaderField, bool) {
	if i <= 0 {
		return HeaderField{}, false
	}
	staticIndex := i - 1
	if staticIndex < len(StaticTable) {
		return StaticTable[staticIndex], true
	}
	return HeaderField{}, false
}

func (h *HPACKDecoder) Decode(payload []byte) error {
	fmt.Printf("Decoding %d bytes\n", len(payload))
	for len(payload) > 0 {
		b := payload[0]
		if b&128 == 128 { // Indexed Header Field
			index, n := decodeInt(payload, 7)
			if n < 0 {
				return fmt.Errorf("failed to decode integer")
			}
			payload = payload[n:]
			header, ok := h.Header(index)
			if !ok {
				return fmt.Errorf("invalid header index: %d", index)
			}
			fmt.Printf("  [Header] %s: %s\n", header.Name, header.Value)
		} else {
			// Other header field types (literal, etc.) not implemented yet
			return fmt.Errorf("not implemented: literal header field")
		}
	}
	return nil
}

// decodeInt decodes a variable-length integer from a byte slice.
// It returns the decoded integer and the number of bytes consumed.
// See RFC 7541 section 5.1 for details.
func decodeInt(payload []byte, n int) (int, int) {
	if len(payload) == 0 {
		return 0, -1
	}
	mask := (1 << n) - 1
	i := int(payload[0]) & mask
	if i < mask {
		return i, 1
	}

	// The value overflows the first byte. The rest of the integer is a
	// standard varint.
	val, bytesRead := binary.Uvarint(payload[1:])
	if bytesRead <= 0 {
		return 0, -1 // Malformed varint
	}

	return i + int(val), 1 + bytesRead
}
go/parser.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
	"io"
)

// FrameHeader represents the 9-byte fixed header of every HTTP/2 frame.
type FrameHeader struct {
	Length   uint32
	Type     uint8
	Flags    uint8
	StreamID uint32
}

// Frame represents a complete HTTP/2 frame including its payload.
type Frame struct {
	Header  FrameHeader
	Payload []byte
}

// ReadFrame reads a header and then the corresponding payload from the connection.
func ReadFrame(r io.Reader) (Frame, error) {
	// 1. Read the 9-byte header
	headerBuf := make([]byte, 9)
	_, err := io.ReadFull(r, headerBuf)
	if err != nil {
		return Frame{}, fmt.Errorf("reading header: %w", err)
	}

	// 2. Parse the header fields using bit-shifting
	header := FrameHeader{
		Length:   uint32(headerBuf[0])<<16 | uint32(headerBuf[1])<<8 | uint32(headerBuf[2]),
		Type:     headerBuf[3],
		Flags:    headerBuf[4],
		StreamID: binary.BigEndian.Uint32(headerBuf[5:9]) & 0x7FFFFFFF,
	}

	// 3. Read the payload based on the Length field
	payload := make([]byte, header.Length)
	if header.Length > 0 {
		_, err = io.ReadFull(r, payload)
		if err != nil {
			return Frame{}, fmt.Errorf("reading payload: %w", err)
		}
	}

	return Frame{Header: header, Payload: payload}, nil
}

]]>Building a Live BGP Maphttps://kmcd.dev/posts/live-internet-map/Mon, 02 Mar 2026 08:00:00 +0000https://kmcd.dev/posts/live-internet-map/ <p> <img hspace="5" src="https://kmcd.dev/posts/live-internet-map/map_hu_cad483650bdd0548.webp" /> </p> Building a cool looking, real-time BGP map Right now, thousands of routers are arguing about how to reach each other. That’s expected. It’s how the Internet works. This website wouldn’t load without it. BGP (Border Gateway Protocol) continuously announces and withdraws prefixes, adjusting how traffic moves globally. Most people see URLs and apps; routers see prefixes and AS paths.

I made a map that lets us listen in on this conversation, but in a relaxing, aesthetically pleasing way.

In my last post, I mentioned a websocket-based streaming API from RIPE. At the time, I set it aside. Soon, it became my obsession and the live view was born. While this visualization occasionally stumbles into being practically useful for spotting global outages, my primary requirement was simply to build a really cool looking map.

You can check out the source code for this project on GitHub or watch the map in action on my YouTube channel or here:

What are we looking at?

This map is a live visualization of the Border Gateway Protocol (BGP). This is the “language” routers use to talk to each other and decide the best path for your data to travel across the globe.

Imagine a router trying to find the best way to send traffic to Google. It receives multiple path advertisements from its neighbors, and it has to pick the most efficient route:

D2 Diagram

Every pulse on the map represents a real routing update. Sometimes it’s routine churn. Sometimes it’s maintenance, an outage, or a path change somewhere along the way.

The Global Game of Telephone

To understand why the map pulses, you have to look at how routers talk. BGP is a path-vector protocol, which is effectively a global game of telephone. When a network (an Autonomous System, or AS) wants to be found, it tells its immediate neighbors, who tell their neighbors, and so on.

  • The Announcement: When a router in Tokyo says, “I have a path to 8.8.8.0/24,” it sends an Update to its peers. Every peer that hears this stamps the message with its own ID before passing it along. This list of stamps is called the AS Path.
  • The Selection: Routers generally prefer the shortest path. If an observer in New York hears the same news from London (2 hops) and Sydney (5 hops), it will automatically choose the shorter London route. On the live map, you will see this selection light up as a purple pulse.
  • The Withdrawal: If a fiber line is cut, for example, the router sends a Withdrawal. This is where the game of telephone gets frantic. Neighbors start checking their old notes: “Wait, I can’t go through London anymore? What about that longer path through Sydney I heard about earlier?”

Here is how that “envelope” looks as it travels from Tokyo to New York. Notice how the path grows longer at every step.

D2 Diagram

Because routers often wait a few seconds before passing news along (to avoid “vibrating” the whole internet with every tiny hiccup), these updates arrive in waves. On the live map, this looks like a ripple of activity starting at the origin and washing over the globe as the “signatures” accumulate.

Spotting a BGP Flap

If you are watching the map and suddenly see a wave of pulses lighting up all over the world at the exact same time, you might be witnessing a BGP flap.

In networking, flapping happens when a route rapidly appears and disappears. Imagine a misconfigured router or a loose fiber cable. The router yells to the Internet, “I have a path to Google!” only to drop the connection a second later and say, “Never mind, it is gone”. That single localized hiccup doesn’t stay local. It ripples outward as routers everywhere recalculate their paths. To keep the whole system from grinding to a halt, modern routers use Route Flap Damping. This essentially puts the noisy network in a time-out until it proves it can stay stable.

Decoding the Pulses

When you see those colored pulses popping off on the map, they represent BGP updates processed through a multi-stage classification engine. Rather than just showing raw protocol messages (which is what the earlier version of the map did), the map categorizes events into four distinct colors based on their behavior and potential impact.

Legend

Anomalies and Behaviors

The classification engine also maps events into Level 2 categorizations (anomalies) based on heuristics applied over recent activity windows. To make sense of the noise, the multi-stage engine uses specific triggers to drop these events into the four colored buckets before presenting them on the map. These fall into four severity tiers:

Severity Tier Color Description
Critical Red Significant routing failures, such as a prefix sustaining multiple withdrawals with no announcements, or path violations that suggest a route leak.
Bad Orange Highly volatile or inefficient behavior, including rapid “flapping” of routes, excessive “babbling” (a term I coined for this project) with unchanged attributes, or frequent next-hop changes.
Normal / Policy Purple Standard routing adjustments, such as traffic engineering (Policy Churn), path length oscillations, or the natural “Path Hunting” process where routers explore alternatives during convergence.
Normal / Discovery Blue Routine background noise, including standard prefix origination or redundant gossip pulses that keep routing tables current.

When you zoom out and see all those colors firing at once, the true scale of the Internet comes to life. It tells the story of over 70,000 independent networks coordinating in real time.

What else is on the map?

To make sure the map isn’t just a wall of moving dots, I included several dashboard elements that provide context to the chaos:

Preview Description
Active Countries
Ranks the countries currently experiencing the highest volume of network updates. It is an instant look at where in the world the most “routing churn” is happening at any given moment.
BGP Anomalies
Tracks the specific network blocks that have the worst anomalies that we’ve detected. Great for spotting outages or a flapping link. Another name for this is the networking “wall of shame”.
Activity Trend
A rolling 60-second activity graph. It tracks whether activity is spiking or calming down, letting you see the difference between routine background noise and a massive routing event.
Beacon Analysis
A dynamic donut chart separating “Organic” traffic from “Beacons” (special test signals sent out by researchers). It helps show how much activity is natural versus intentional measurement. More on this below.
Now Playing
The current background music track.

Path Hunting and Anycast

When I first started watching the live data, I was confused by why a single localized outage would trigger a massive global explosion of pulses.

I’ve since learned this is likely due to a phenomenon called “Path Hunting.” When a route dies, the Internet doesn’t instantly agree it’s gone. Instead, routers desperately try to find backup paths. They’ll try a longer route, fail, try an even longer one, fail again, and generate a new BGP update every single time. Those massive bursts of purple pulses are basically the routers “thinking out loud” as they scramble to route around the damage.

Routers looking for Facebook’s network on October 4, 2021

This scramble to find backup paths can occasionally leave behind an interesting anomaly known as a “BGP zombie.” If a router fails to process a withdrawal message due to a software bug or slow propagation, it will stubbornly keep announcing a dead path to its neighbors, creating a localized black hole for traffic. Cloudflare has a great write-up on hunting down these undead routes if you want to fall down that rabbit hole.

Anycast routing amplifies this chatter even further. Huge networks (like Google or Cloudflare) announce the exact same /24 prefix from dozens of different physical locations globally so their services are fast everywhere. But if a major transit provider drops a peering session, or a provider intentionally shifts traffic away from a datacenter for maintenance, thousands of routers might suddenly decide to shift their traffic to a different Anycast node all at once. The result is a sudden surge of routing adjustments across the map.

RIPE RIS Beacons and Anchors

While building the “Most Active Prefixes” list, I kept noticing the exact same thing: /24 subnets were overrepresented on the leaderboard.

A /24 (256 IPs) is effectively the smallest globally routable unit, so most churn naturally happens at that granularity.

But there was another reason for seeing the same /24 subnets appearing on the list. Not all activity on the map comes from failing links or organic traffic shifts. There is also intentional ‘breakage’ happening behind the scenes to test BGP propagation.

It turns out RIPE RIS operates Routing Beacons. Routing Beacons are prefixes deliberately announced and withdrawn on a fixed schedule, typically every two hours. One of them announces and withdraws every 10 minutes. Researchers use these beacons as a controlled signal inside the global routing table to study BGP propagation and convergence. To make the activity list useful, I had to write logic to classify and filter these beacons out of the ranking.

RIPE also runs “Anchors” alongside these beacons. While a beacon prefix constantly flips on and off, an anchor is a prefix permanently announced from the exact same physical router. This gives researchers a stable control group. They can compare the volatile beacon traffic against a baseline of stable routing from the identical location.

I eventually added a Beacon Analysis view that separates “organic” updates from beacon-driven ones. It makes the metrics more accurate and highlights how much traffic is from deliberate live validation.

BGP Babbling and Attribute Churn

So if a burst of updates isn’t a dying link, a desperate search for a backup path, or a research beacon, what else could it be? Sometimes a network is just fidgeting. I call this babbling. While not an official industry term, it perfectly describes the constant, repetitive “talk” of updates that don’t actually change anything meaningful about the route.

I caught a great example of this while watching the stream. A Finnish fiber provider (AS43016) was firing off nearly 100 pulses per second, and this went on for days. The raw data showed the route wasn’t actually dropping. Instead, a single piece of metadata called the Aggregator ID just kept flipping back and forth.

This creates a localized flurry of activity. Some router somewhere was probably misconfigured and couldn’t make up its mind about how to summarize its own network. Every time it changed its mind, even by a single bit, it had to update every other router on Earth. Standard monitoring tools usually miss these “attribute flaps” because the network stays perfectly reachable. But on the map, they paint a very clear picture: a constant, rhythmic heartbeat of orange “bad behavior” pulses.

I built a tool to debug noisy prefixes like this. It aggregates BGP update stats and tries to diagnose the root cause, such as path oscillation, a flapping link, or heavy Anycast routing. Here is the output for our problem child over at AS43016:

$ just debug-prefix 195.155.146.0/24
BGP Prefix Monitor Stats (Running for 293.4s)
--------------------------------------------------
Announcements: 4576 (15.60/s)
Withdrawals:   1422 (4.85/s)
Total Msgs:    5101 (17.39/s)
Unique Peers:  310
--------------------------------------------------
GLOBAL CHURN EVENTS:
  AS-Path Changes:  2275
  Community Changes: 3259
  Next-Hop Changes:  0
  Aggregator Flaps:  0
  Path Length Flaps: 1255
--------------------------------------------------
LIKELY CONCLUSIONS:
  - Path Length Oscillation (Route is toggling between different path lengths)
  - BGP Babbling (Excessive update rate detected)
--------------------------------------------------
Top 5 Churning Peers:
  187.16.220.216: 149 attribute changes
  5.188.4.211: 142 attribute changes
  103.152.35.254: 142 attribute changes
  177.221.140.2: 138 attribute changes
  154.18.4.110: 132 attribute changes

At the time of publishing, this prefix is still babbling away. This script became the basis for the classification engine that I discuss later on in the article.

Making the map

Handling 30,000+ BGP updates per second takes more than plotting points on a canvas. The project is written in Go for its concurrency model and relies on Ebitengine for hardware-accelerated 2D rendering.

Why a Stream?

I originally planned to build this as a standard web frontend, similar to my previous map. However, I hit two massive walls almost immediately.

The first problem was the sheer volume of data. BGP updates can easily peak at over 30,000 events per second. Forcing a web browser to process that firehose while maintaining a smooth 30 FPS with complex blending is just not in the cards today.

The second problem was scaling. If the map actually got popular, having thousands of browsers opening individual websocket connections to the RIPE RIS-Live service would be a disaster. It is wildly inefficient, and accidentally DDoSing a service designed to monitor Internet stability was not on my to-do list.

Here is what that scenario looks like:

D2 Diagram

To protect the RIPE service from being overwhelmed, the logical next step was to put a middleman in place to handle the multiplexing. This led me to a standard client-server architecture:

D2 Diagram

Multiplexing solves the connection problem, but it completely ignores the browser rendering issues I was having. To guarantee a smooth 30 FPS for everyone without melting their CPUs, I decided to bypass the browser canvas entirely. I pivoted the architecture to a centralized YouTube stream:

D2 Diagram

Now I had a choice. Scenario 1 was dead on arrival because it could make the operators of RIPE RIS-Live very sad and potentially angry. That left me with the choice between building a complex backend service to multiplex that single RIPE connection to all my users (Scenario 2), or completely changing how people view the map by streaming to YouTube (Scenario 3). I went with the latter option.

Rendering the entire visualization on my own server and broadcasting it guarantees that every viewer gets the exact same high-fidelity experience, regardless of their hardware. It is easy to run on a TV where the browser version isn’t really viable. This pivot also made the tech stack an obvious choice. Once I started experimenting with Ebitengine, hardware-accelerated rendering in Go gave me crisper, far more fluid visuals than I could ever squeeze out of a standard browser canvas.

The downside is reduced interaction: no zooming, no toggling UI, no customization. I think this tradeoff was ultimately worth it, but I just want to note what I lost from making this dramatic change in architecture.

Flattening IP Space

To map a BGP update to a geographic location, you need reliable IP-to-region data. I am currently only focusing on IPv4, and that data comes from five Regional Internet Registries (RIRs). Each registry publishes large and sometimes overlapping delegated stats files.

Fragmented lookups across raw datasets might be fine for offline processing, but we have a strict frame rate budget. If the engine had to search through five separate datasets for every single update, the visualization would stutter. At 30,000+ updates per second, efficiency is pretty important.

To solve this, I preprocess all the data upfront using a sweep-line algorithm. Each IP range acts as a segment on a 1D number line. The algorithm walks across this space, resolves any overlaps between registries, and collapses millions of ranges into a single, clean, non-overlapping index.

For example, take two overlapping registry entries:

  • Range A (ARIN): 10.0.0.0 to 10.0.0.255
  • Range B (RIPE): 10.0.0.128 to 10.0.1.255

The algorithm flattens these into three distinct, non-overlapping segments:

  1. 10.0.0.0 to 10.0.0.127 (ARIN only)
  2. 10.0.0.128 to 10.0.0.255 (Conflict resolved)
  3. 10.0.1.0 to 10.0.1.255 (RIPE only)

This preprocessing seems like overkill, but it’s worth it since it makes lookups super cheap. I back this index with BadgerDB and a DiskTrie for high-performance persistent storage. This allows the engine to track “seen” prefixes seamlessly across different sessions without eating up memory.

Managing the Firehose

BGP updates arrive continuously, and during route flapping events the volume spikes hard.

To keep the visualization readable without becoming an incomprehensible mess, the pipeline waits 10 seconds to ensure a withdrawal isn’t just a rapid path re-convergence, and paces the visual output so spikes are emitted smoothly every 500ms.

D2 Diagram

Aesthetics and Motion

Animations use interpolation instead of snapping to the next state. For parts of the map which update infrequently, I wanted to highlight that a change occurred. For that, I added a “glitch” effect to the “Top Activity Hubs” and “Most Active Prefixes” to make it more obvious and to add to the cyberpunk aesthetic. These effects add polish, but too much motion detracts from the vibes of the map. Finding that balance took restraint and a surprisingly large amount of experimentation.

The pulses are what actually bring the data to life. In the engine, each pulse is a simple generated glow texture. I add a bit of spatial jitter so concurrent events do not stack perfectly on top of each other, and I scale their sizes logarithmically so massive data spikes do not turn the map into a solid wall of color.

The colors map directly to the severity tiers: red for critical events, orange for bad behavior, purple for policy churn and hunting, and blue for routine discovery. Because they use additive blending, overlapping pulses naturally create a bright hotspot over regions with a ton of routing activity. They pop onto the map, expand, and fade out smoothly. Managing this entire visual lifecycle efficiently is what keeps the map feeling dynamic without tanking the frame rate.

Animation of BGP events in Europe

Animation of BGP events in Europe

The Mollweide Projection

Mercator would have been easy, but it heavily distorts size near the poles. For a global activity map, that felt misleading.

I chose the Mollweide projection.
By Justin Kunimune - Own work, CC BY-SA 4.0

This is an equal-area projection, which means it accurately represents the physical footprint of different regions. It produces a world view that still feels familiar without exaggerating high-latitude areas.

More Meaningful Events

Raw BGP messages only tell us two things: a route was announced, or a route was withdrawn. So how does the dashboard know when to declare a ’link flap’, a ‘route leak’, or a massive ‘outage’? The short answer is that I built a classification engine that takes the pattern of raw announce/withdrawal updates that BGP provides and converts them more meaningful events. Some kinds of events are easier to detect than others.

Route leaks are a great example of how messy this can get. Initially, I tried to validate routes using databases like Cloudflare’s RPKI portal, specifically hooking into their rpki.json endpoint. The goal was to check if the announcements for networks actually matched their registered ASNs. In practice, this resulted in way too many false positives because a massive number of announcements just were not matching the registered ASNs. If I had kept that logic, the map would have been permanently covered in red alert pulses.

Because of the noise, I ended up implementing a check for the valley-free routing principle. To understand why this works, we have to look at how BGP treats business relationships. BGP routing policies are built around who is paying whom. A network typically has providers it pays for transit, customers who pay it for access, and peers it swaps traffic with for mutual benefit. The valley-free rule dictates that a network should never act as a free transit bridge between two of its providers or peers.

Imagine a small regional network buys internet access from both AT&T and Verizon for redundancy. AT&T shares its global routing table with this small network so it knows where to send data. If that small network accidentally announces all of those AT&T routes to Verizon, it is inadvertently telling the entire internet to send all traffic between Verizon and AT&T through its local routers. Traffic would flow down from Verizon, into the small regional network, and back up to AT&T. That “down and back up” path is what creates the valley shape in the AS path. Because that small network does not have the capacity to handle global Tier-1 traffic, it immediately gets crushed under the weight of the data. The network drops packets and causes a massive localized internet outage, which is a classic route leak.

D2 Diagram
D2 Diagram

So, when the classification engine sees an AS path that violates this principle by dipping down into a lower-tier network and back up to a major provider, the system flags it as a route leak. While it serves as a decent baseline, I am generally uncertain about relying solely on this method. It is definitely a part of the classification engine that I want to explore and refine over time.

Note that since I published this post, I significantly changed how route leaks are detected. Yes, it does involve the valley-free routing principle and RPKI, but the story is a bit more complex and will probably require an entire post to explain. More on this topic in the future!

Other rules are slightly more straightforward:

Event Detection Trigger
Outage >= 3 withdrawals, 0 announcements
Route Leak path contains Tier-1 to non-Tier-1 to Tier-1
Link Flap > 5 withdrawals, announce:withdrawal ratio < 2.5
Babbling High volume, unchanged attributes
Next-Hop Flap >= 5 next-hop changes, stable path length
Aggregator Flap > 10 AGGREGATOR changes
Policy Churn Elevated attribute changes
Path Oscillation Frequent path length switching
Path Hunting Increasing path length, then withdrawal
Discovery Prolonged announcements, few changes

To make these determinations, the engine analyzes the last five minutes of event data. Once classified, a prefix holds its state for ten minutes before being reevaluated. This means a prefix might eventually downgrade from a Link Flap to a routine Discovery. Outage states are the sole exception and clear immediately upon any new announcement. These initial rules are just a baseline I plan to refine as the project evolves.

Here is the final result, which I’ve gazed at for far too long already:

🔴 livemap.kmcd.dev

🔴 livemap.kmcd.dev

This project turned into a deeper dive into BGP than I expected. Watching as routing updates happen live exposes patterns that are impossible to find with a static snapshot. It has been a rewarding project and I am extremely happy with the result.

So please, toss the live stream on your TV, sit back, relax, and watch the Internet route the world’s network traffic as you listen to relaxing lofi in the background.

]]>HTTP/2 From Scratch: Part 2https://kmcd.dev/posts/http2-from-scratch-part-2/Wed, 25 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/http2-from-scratch-part-2/ <p> <img hspace="5" src="https://kmcd.dev/posts/http2-from-scratch-part-2/cover.svg" /> </p> Diving into the binary framing layer and byte-shifting in Go In the previous post, we successfully performed the TLS handshake and sent our 24-byte connection preface. To the server, we now look like a valid HTTP/2 client. But as soon as that preface is sent, the server starts talking back in a language we haven’t yet taught our Go code to understand.

While HTTP/1.1 communicated in lines of text separated by newlines, HTTP/2 communicates in frames. Every single interaction from here on out (settings, headers, data, and keepalive pings) happens inside of a frame. To move us closer to our goal of understanding HTTP/2, we need to build a parser that can slice into these binary packets and make sense of the bits inside.

The Anatomy of a Frame

According to the specification, every HTTP/2 frame begins with a fixed 9-byte header. This header is the roadmap for the rest of the payload. It tells us exactly how much data follows and how we should interpret it.

The 9-byte header is laid out precisely. Using a packet visualization, we can see how the bits are packed together:

packet-beta
0-23: "Length (24 bits)"
24-31: "Type (8 bits)"
32-39: "Flags (8 bits)"
40: "R (Reserved)"
41-71: "Stream Identifier (31 bits)"

The header fields serve specific purposes:

  • Length: The size of the frame payload (up to 16,383 bytes by default).
  • Type: The specific purpose of the frame (e.g., 0x0 for DATA, 0x1 for HEADERS).
  • Flags: Boolean modifiers that change how the frame is processed.
  • Stream Identifier: A unique ID that links the frame to a specific request-response lifecycle.

The first bit of the Stream Identifier is reserved (R) and must remain zero. You will see this a lot in binary protocols, where bits are reserved for the potential of making a backwards compatible next version of the protocol or byte alignment.

Modeling the Header in Go

To handle this in Go, we need a structure that mirrors the spec. While the length and stream ID are 24 and 31 bits respectively, we use uint32 to store them and handle the bit-masking manually during the read.

type FrameHeader struct {
    Length   uint32
    Type     uint8
    Flags    uint8
    StreamID uint32
}

The challenge here is that Go’s encoding/binary package doesn’t have a direct way to read a 24-bit integer. We have to read three bytes and manually shift them into place. We are no longer working with strings; we are working with bitwise operators.

Thinking in Bits

Before we write the parser, we need to get comfortable with Bit Shifting. In Go, we use the << (left shift) and | (bitwise OR) operators to “stitch” separate bytes into a single number.

Think of it like a conveyor belt. To turn three separate bytes into one 24-bit integer, we take the first byte and slide it 16 places to the left, move the second byte 8 places to the left, and then lay them all on top of each other using the OR operator.

In Go, uint32(data[0]) << 16 tells the compiler to take an 8-bit byte, treat it as a 32-bit number, and move its value to the “high” end of the bucket. HTTP/2 always uses network byte order (big-endian). Go’s binary.BigEndian makes this explicit when reconstructing integers.

Bitwise operations take some getting used to, so it’s worth spending time with them if this feels unfamiliar. I recommend Chapter 27: Enum, Iota & Bitmask from Practical Go Lessons.

Parsing the Bytes

When the server responds, we read the first 9 bytes from our tls.Conn. We then use bit-shifting to reconstruct the fields. For the Stream Identifier, we use binary.BigEndian.Uint32 and then mask out the first bit.

We also need to handle the Reserved (R) bit in the Stream Identifier. Even though we read 4 bytes (32 bits), the spec says the first bit must be ignored. We use the bitwise AND operator with 0x7FFFFFFF, which is a binary 0 followed by thirty-one 1s—to “mask out” that first bit and ensure our ID is always interpreted correctly.

func ReadFrameHeader(data []byte) FrameHeader {
    // Length is 24 bits: first 3 bytes shifted and OR'd
    length := uint32(data[0])<<16 | uint32(data[1])<<8 | uint32(data[2])
    
    // Type and Flags are single bytes
    fType := data[3]
    flags := data[4]
    
    // Stream ID is 31 bits. We read 4 bytes starting at offset 5, 
    // then mask out the reserved bit (0x7FFFFFFF)
    streamID := binary.BigEndian.Uint32(data[5:9]) & 0x7FFFFFFF

    return FrameHeader{
        Length:   length,
        Type:     fType,
        Flags:    flags,
        StreamID: streamID,
    }
}

The SETTINGS Frame

Immediately after receiving our preface, the server will send a SETTINGS frame (Type 0x04). This is the protocol’s way of establishing the rules of the road. It defines limits like how many concurrent streams we can open or how large our header table can be.

The payload of a SETTINGS frame consists of a series of 6-byte parameters:

packet-beta
title SETTINGS Parameter (6 Bytes)
0-15: "Identifier (16 bits)"
16-47: "Value (32 bits)"

For now, the most important thing is that we simply acknowledge these settings. HTTP/2 requires that when you receive a SETTINGS frame, the client must send back a SETTINGS frame with the ACK flag set (bit 0x1). If you don’t, the server will eventually assume the client is unresponsive and close the connection.

Putting it into Practice

We can now update our client to read the header, read the payload based on the length we just parsed, and log what we found. This is where the binary nature of the protocol becomes visible in your terminal.

full parser.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
	"io"
)

// FrameHeader represents the 9-byte fixed header of every HTTP/2 frame.
type FrameHeader struct {
	Length   uint32
	Type     uint8
	Flags    uint8
	StreamID uint32
}

// Frame represents a complete HTTP/2 frame including its payload.
type Frame struct {
	Header  FrameHeader
	Payload []byte
}

// ReadFrame reads a header and then the corresponding payload from the connection.
func ReadFrame(r io.Reader) (Frame, error) {
	// 1. Read the 9-byte header
	headerBuf := make([]byte, 9)
	_, err := io.ReadFull(r, headerBuf)
	if err != nil {
		return Frame{}, fmt.Errorf("reading header: %w", err)
	}

	// 2. Parse the header fields using bit-shifting
	header := FrameHeader{
		Length:   uint32(headerBuf[0])<<16 | uint32(headerBuf[1])<<8 | uint32(headerBuf[2]),
		Type:     headerBuf[3],
		Flags:    headerBuf[4],
		StreamID: binary.BigEndian.Uint32(headerBuf[5:9]) & 0x7FFFFFFF,
	}

	// 3. Read the payload based on the Length field
	payload := make([]byte, header.Length)
	if header.Length > 0 {
		_, err = io.ReadFull(r, payload)
		if err != nil {
			return Frame{}, fmt.Errorf("reading payload: %w", err)
		}
	}

	return Frame{Header: header, Payload: payload}, nil
}

Running this against kmcd.dev will show you exactly what Cloudflare or your server expects. You’ll see a Type: 4 frame with a specific length. This is the first time our code is truly listening to an HTTP/2 server.

What’s Next?

We can now read the frame header and the raw payload, but the raw payload represents a HEADERS frame and we have no idea how to actually parse that yet. In the next post, we will tackle the most complex part of the HTTP/2 specification: HPACK.

We need to learn how to compress and decompress headers using a stateful table, which is the only way we will ever be able to send a real GET request and get a real response.

See all of the code mentioned in this article here:

go/client.go (click to expand)View on GitHub 
package main

import (
	"crypto/tls"
	"fmt"
	"log"
)

const (
	// The "Magic" preface required by RFC 9113
	preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	server  = "kmcd.dev:443"
)

func main() {
	// We configure TLS to specifically look for "h2" via ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	// Dial the server and perform the handshake
	conn, err := tls.Dial("tcp", server, config)
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	// Ensure the server actually agreed to speak HTTP/2
	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		log.Fatalf("Server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", server, state.NegotiatedProtocol)

	// Send the Connection Preface to initialize the H2 session
	_, err = conn.Write([]byte(preface))
	if err != nil {
		log.Fatalf("Failed to send preface: %v", err)
	}

	fmt.Println("Preface sent successfully. The connection is open.")

	frame, err := ReadFrame(conn)
	if err != nil {
		log.Fatalf("Failed to read server settings: %v", err)
	}

	// Check the Type on the Header field of our Frame struct
	if frame.Header.Type != 0x04 {
		log.Fatalf("Expected SETTINGS frame (0x04), got: %d", frame.Header.Type)
	}
	fmt.Printf("Received SETTINGS: %d bytes on Stream %d\n", frame.Header.Length, frame.Header.StreamID)

	// 2. Acknowledge the settings frame
	ackHeader := []byte{
		0x00, 0x00, 0x00, // Length: 0
		0x04,                   // Type: SETTINGS
		0x01,                   // Flags: ACK (0x01)
		0x00, 0x00, 0x00, 0x00, // Stream ID: 0 (Connection level)
	}

	if _, err = conn.Write(ackHeader); err != nil {
		log.Fatalf("Failed to send SETTINGS ACK: %v", err)
	}
	fmt.Println("Sent SETTINGS ACK.")
}
go/parser.go (click to expand)View on GitHub 
package main

import (
	"encoding/binary"
	"fmt"
	"io"
)

// FrameHeader represents the 9-byte fixed header of every HTTP/2 frame.
type FrameHeader struct {
	Length   uint32
	Type     uint8
	Flags    uint8
	StreamID uint32
}

// Frame represents a complete HTTP/2 frame including its payload.
type Frame struct {
	Header  FrameHeader
	Payload []byte
}

// ReadFrame reads a header and then the corresponding payload from the connection.
func ReadFrame(r io.Reader) (Frame, error) {
	// 1. Read the 9-byte header
	headerBuf := make([]byte, 9)
	_, err := io.ReadFull(r, headerBuf)
	if err != nil {
		return Frame{}, fmt.Errorf("reading header: %w", err)
	}

	// 2. Parse the header fields using bit-shifting
	header := FrameHeader{
		Length:   uint32(headerBuf[0])<<16 | uint32(headerBuf[1])<<8 | uint32(headerBuf[2]),
		Type:     headerBuf[3],
		Flags:    headerBuf[4],
		StreamID: binary.BigEndian.Uint32(headerBuf[5:9]) & 0x7FFFFFFF,
	}

	// 3. Read the payload based on the Length field
	payload := make([]byte, header.Length)
	if header.Length > 0 {
		_, err = io.ReadFull(r, payload)
		if err != nil {
			return Frame{}, fmt.Errorf("reading payload: %w", err)
		}
	}

	return Frame{Header: header, Payload: payload}, nil
}

]]>IRC Log: The Cloud Scale Incidenthttps://kmcd.dev/posts/irc-log-the-cloud-scale-incident/Mon, 23 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/irc-log-the-cloud-scale-incident/ <p> <img hspace="5" src="https://kmcd.dev/posts/irc-log-the-cloud-scale-incident/cover.svg" /> </p> A vibe-coder attempts to provision a supercomputer to host a crypto-based to-do list app. An archived log from the #dev-help channel on the Freenode (RIP) afterlife network.

[03:12] *** Topic: reckless_dev is now known as broke_dev | Don’t paste API keys
[03:14] *** vibe_coder_99 has joined
[03:15] <vibe_coder_99> yo! anyone good with terraform? i’m trying to launch my new crypto-based to-do list app and the tutorial is too slow.
[03:15] <sysadmin_dave> What tutorial? Also, crypto to-do list? Why?
[03:16] <vibe_coder_99> it’s strictly vibes based. i just need it to scale. like, infinite scale. i found this script on a forum that says it “maximizes throughput”.
[03:17] <cloud_guru> Paste the plan. Don’t just run random scripts.
[03:18] <vibe_coder_99> nah its fine, i just want to know what “p4d.24xlarge” means. sounds powerful. i changed the count to 50 just to be safe for launch day.
[03:18] <sysadmin_dave> STOP.
[03:18] <sysadmin_dave> DO NOT RUN THAT.
[03:19] <cloud_guru> That instance is $32 an hour. PER INSTANCE.
[03:19] <vibe_coder_99> wait really? lol whatever i have $100 in credits. running terraform apply --auto-approve now.
[03:19] * sysadmin_dave screams internally
[03:22] <vibe_coder_99> man aws is slow today. it’s been provisioning for like 2 minutes.
[03:22] <cloud_guru> You are requesting 50 top-tier GPU instances. You are trying to provision a supercomputer to host a to-do list.
[03:23] <vibe_coder_99> gotta go fast right? 🚀
[03:24] <vibe_coder_99> uh guys
[03:25] <vibe_coder_99> my bank called. fraud alert. something about “unusual spending pattern”.
[03:25] <sysadmin_dave> You just burned your credits in 4 minutes. You are now spending ~$1600/hour.
[03:26] <vibe_coder_99> how do i undo??? ctrl-z doesn’t work in terminal!!
[03:26] <sysadmin_dave> terraform destroy. PRAY that the API isn’t rate limiting you.
[03:27] <vibe_coder_99> it says “state lock”. i think i closed the window too fast.
[03:28] *** vibe_coder_99 has left
[03:28] *** vibe_coder_99 has quit (Connection reset by peer: fleeing the country)
]]>Visualizing the Internet (2026)https://kmcd.dev/posts/internet-map-2026/Wed, 18 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/internet-map-2026/ <p> <img hspace="5" src="https://kmcd.dev/posts/internet-map-2026/cover.svg" /> </p> Mapping global Internet infrastructure and routing dominance over time For the past few years, I’ve been trying to make the physical reality of the Internet visible with my Internet Infrastructure Map. This map shows the network of undersea fiber-optic cables along with peering bandwidth, grouped by city. I update the map annually, but I don’t want to just pull the latest data and call it a day. In this post I discuss how the map evolved this year and what I did to make it happen, but you can skip to the good part by viewing it here: map.kmcd.dev.

For the 2026 edition, I wanted to better answer the question: where does the Internet actually live? By layering on BGP routing tables alongside physical infrastructure data, I’m now closer to answering that question.

The result is a concept I call “Logical Dominance.” Each city’s dominance is calculated by summing total address space of IPv4 subnets that are “homed” in that city. How can I tell where IP addresses are homed? This required analyzing global routing tables to trace IP ownership back to specific geographies. Read on to find out how I accomplished this!

Before:

Before

After:

After
Internet Infrastructure Map (2026) @ map.kmcd.dev

How the Internet Routes Traffic

Previous versions of the map focused on physical infrastructure: cables and exchange points. The physical path is only half the story. To understand how data moves, we have to look at BGP (Border Gateway Protocol).

BGP is the protocol that distinct networks, known as Autonomous Systems (AS), use to announce which IP addresses they own and how to reach them. If the cables are the hardware, BGP is the software that ties the Internet together. Cloudflare has an excellent primer.

When you load a webpage, your request doesn’t just “know” the path. Your ISP’s routers consult the global BGP routing table to decide the best next hop. Visualized, it looks a little bit like this:

D2 Diagram

In this state, the route from Router -> Netstream (AS8283) -> Google (AS15169) was chosen, at least for now. The underlying routes of the global Internet change thousands of times per second, constantly reshaping the topology.

Sources of BGP Data

To visualize this layer, we need access to routing tables. I explored three ways to get this data, each with its own trade-offs between real-time visibility and historical context.

Query a Looking Glass

We can connect to public routers via projects like University of Oregon Route Views. These allow you to telnet in and run standard CLI commands like show ip bgp to see exactly what a backbone router sees.

BGP routes for 8.8.8.8 (click to expand)View on GitHub 
$ telnet route-views.routeviews.org 23
**********************************************************************

                    RouteViews BGP Route Viewer
                    route-views.routeviews.org

 RouteViews data is archived on https://archive.routeviews.org

 This hardware is part of a grant by the NSF.

 Please contact help@routeviews.org if you have questions, or
 if you wish to contribute your view.

 This router has views of full routing tables from several ASes.

 The current list of all RouteViews peers is at
   https://www.routeviews.org/peers/peering-status.html

 NOTE: If you are using macOS and seeing the error message
 "no default Kerberos realm" when logging in, you may want to
 add "default unset autologin" to your ~/.telnetrc

 To login, use the username "rviews".

**********************************************************************


User Access Verification

Username: rviews
route-views>show ip bgp 8.8.8.8
BGP routing table entry for 8.8.8.0/24, version 941738530
Paths: (16 available, best #15, table default)
  Not advertised to any peer
  Refresh Epoch 1
  4826 15169
    114.31.199.16 from 114.31.199.16 (114.31.199.16)
      Origin IGP, localpref 100, valid, external
      Community: 4826:5203 4826:6510 4826:52032
      path 7F168F059710 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  57866 15169
    37.139.139.17 from 37.139.139.17 (37.139.139.17)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 57866:200 65102:56393 65103:1 65104:31
      unknown transitive attribute: flag 0xE0 type 0x20 length 0x30
        value 0000 E20A 0000 0065 0000 00C8 0000 E20A
              0000 0066 0000 DC49 0000 E20A 0000 0067
              0000 0001 0000 E20A 0000 0068 0000 001F

      path 7F15A63304E8 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  6939 15169
    64.71.137.241 from 64.71.137.241 (216.218.253.53)
      Origin IGP, localpref 100, valid, external
      path 7F1555102CB8 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  20130 6939 15169
    140.192.8.16 from 140.192.8.16 (140.192.8.16)
      Origin IGP, localpref 100, valid, external
      path 7F1588A8BD60 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  3333 1257 15169
    193.0.0.56 from 193.0.0.56 (193.0.0.56)
      Origin IGP, localpref 100, valid, external
      Community: 1257:50 1257:51 1257:3528
      path 7F16B16DD098 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  7018 15169
    12.0.1.63 from 12.0.1.63 (12.0.1.63)
      Origin IGP, localpref 100, valid, external
      Community: 7018:2500 7018:37232
      path 7F1626828FD8 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  20912 15169
    77.39.192.30 from 77.39.192.30 (77.39.192.1)
      Origin IGP, localpref 100, valid, external
      Community: 20912:65002 20912:65022
      path 7F15D19801C0 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  3356 15169
    4.68.4.46 from 4.68.4.46 (4.69.184.201)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 3356:3 3356:86 3356:576 3356:666 3356:901 3356:2012
      path 7F1679EB7640 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  3549 3356 15169
    208.51.134.254 from 208.51.134.254 (67.16.168.191)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 3356:3 3356:22 3356:86 3356:575 3356:666 3356:901 3356:2011 3549:2581 3549:30840
      path 7F16C7D32728 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  101 15169
    209.124.176.223 from 209.124.176.223 (209.124.176.224)
      Origin IGP, localpref 100, valid, external
      Community: 101:20400 101:22200 101:24100
      path 7F1648388978 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  1351 15169
    132.198.255.253 from 132.198.255.253 (132.198.255.253)
      Origin IGP, localpref 100, valid, external
      path 7F15C90D61B8 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  3257 15169
    89.149.178.10 from 89.149.178.10 (213.200.83.26)
      Origin IGP, metric 10, localpref 100, valid, external
      Community: 3257:8052 3257:30306 3257:50001 3257:54900 3257:54901
      path 7F154665E650 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  2497 15169
    202.232.0.2 from 202.232.0.2 (58.138.96.254)
      Origin IGP, localpref 100, valid, external
      path 7F1660B0A1C8 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 2
  3303 15169
    217.192.89.50 from 217.192.89.50 (138.187.128.158)
      Origin IGP, localpref 100, valid, external
      Community: 3303:1004 3303:1007 3303:3067
      path 7F16E10C0508 RPKI State valid
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  8283 15169
    94.142.247.3 from 94.142.247.3 (94.142.247.3)
      Origin IGP, localpref 100, valid, external, best
      Community: 8283:1 8283:101 8283:102
      unknown transitive attribute: flag 0xE0 type 0x20 length 0x30
        value 0000 205B 0000 0000 0000 0001 0000 205B
              0000 0005 0000 0001 0000 205B 0000 0005
              0000 0002 0000 205B 0000 0008 0000 001A

      path 7F16DC0E5B28 RPKI State valid
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 1
  49788 12552 15169
    91.218.184.60 from 91.218.184.60 (91.218.184.60)
      Origin IGP, localpref 100, valid, external
      Community: 12552:10000 12552:14000 12552:14100 12552:14101 12552:24000
      Extended Community: 0x43:100:0
      path 7F15CAD0C378 RPKI State valid
      rx pathid: 0, tx pathid: 0
route-views>⏎

These paths often carry metadata called BGP Communities. These are optional tags that networks use to signal things like geographic origin or peering policy. While perfect for debugging today’s Internet, this approach lacks historical context; you can’t telnet into 2012 to check a routing table from 14 years ago.

Subscribe to a Stream

For real-time views, services like RIPE RIS Live aggregate BGP data from global collectors and stream it over a public WebSocket. You can watch the Internet “breathe” as routes are announced and withdrawn thousands of times per second. This is fascinating for a live dashboard, but useless for backfilling history.

Here’s an example script consuming this stream:

go/stream_bgp/main.go (click to expand)View on GitHub 
package main

import (
	"fmt"
	"log"
	"os"
	"os/signal"
	"time"

	"github.com/gorilla/websocket"
)

// RIPE RIS Live WebSocket URL
const risLiveURL = "wss://ris-live.ripe.net/v1/ws/?client=kmcd-internet-map"

// Message defines the structure of the JSON messages we receive
type Message struct {
	Type string                 `json:"type"`
	Data map[string]interface{} `json:"data"`
}

func main() {
	// Handle Ctrl+C gracefully
	interrupt := make(chan os.Signal, 1)
	signal.Notify(interrupt, os.Interrupt)

	fmt.Printf("Connecting to %s...\n", risLiveURL)

	c, _, err := websocket.DefaultDialer.Dial(risLiveURL, nil)
	if err != nil {
		log.Fatal("dial:", err)
	}
	defer c.Close()

	// Subscribe to the firehose (all messages)
	// You can filter this! e.g., {"host": "rrc21"} for a specific collector
	subscribeMsg := map[string]interface{}{
		"type": "ris_subscribe",
		"data": map[string]interface{}{
			"moreSpecific": true,
			"type":         "UPDATE", // Only show route updates
		},
	}

	if err := c.WriteJSON(subscribeMsg); err != nil {
		log.Fatal("subscribe:", err)
	}

	fmt.Println("Connected! Streaming global BGP updates...")
	fmt.Println("------------------------------------------------")

	done := make(chan struct{})

	go func() {
		defer close(done)
		for {
			var msg Message
			err := c.ReadJSON(&msg)
			if err != nil {
				log.Println("read:", err)
				return
			}

			// We only care about BGP UPDATE messages 
			if msg.Type == "ris_message" {
				path := msg.Data["path"]
				prefix := msg.Data["announcements"]
				
				// Handle withdrawals (routes being removed)
				if prefix == nil {
					prefix = "WITHDRAWAL"
				}

				// Print the timestamp, the route prefix, and the AS path
				fmt.Printf("[%s] Prefix: %v | Path: %v\n", 
					time.Now().Format("15:04:05"), 
					prefix, 
					path,
				)
			}
		}
	}()

	// Wait for interrupt
	<-interrupt
	fmt.Println("\nDisconnecting...")
	err = c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
	if err != nil {
		log.Println("write close:", err)
		return
	}
	select {
	case <-done:
	case <-time.After(time.Second):
	}
}

The output looks like this:

Websocket Stream Output (click to expand)View on GitHub 
Connecting to wss://ris-live.ripe.net/v1/ws/?client=kmcd-internet-map...
Connected! Streaming global BGP updates...
------------------------------------------------
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::b1 prefixes:[2804:70c0::/32]]] | Path: [58057 6939 22381 1031 263444 22381 270746]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::aa,fe80::8a7e:25ff:fed3:420b prefixes:[2a13:9404::/32]]] | Path: [6939 215120 34689]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::aa,fe80::470:71ff:fec5:b6ad prefixes:[2a14:7c0:1740::/48 2a10:ccc7:b110::/44]]] | Path: [196621 6939 215120 214497 6204 215120 214497]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::aa,fe80::224:38ff:fea4:a907 prefixes:[2a14:7c0:1740::/48 2a10:ccc7:b110::/44]]] | Path: [29691 6939 215120 214497 6204 215120 214497]
[22:52:35] Prefix: [map[next_hop:91.206.52.177 prefixes:[2a10:ccc7:b110::/44 2a14:7c0:1740::/48]]] | Path: [58057 6939 215120 214497 6204 215120 214497]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::b1 prefixes:[2803:5d10::/32]]] | Path: [58057 6939 3356 28343 272053]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::aa,fe80::8a7e:25ff:fed3:420b prefixes:[2a14:7c0:1740::/48 2a10:ccc7:b110::/44]]] | Path: [6939 215120 214497]
[22:52:35] Prefix: [map[next_hop:2001:7f8:24::aa,fe80::470:71ff:fec5:b6ad prefixes:[2a13:9404::/32]]] | Path: [196621 6939 215120 34689]
2026/02/09 22:52:35 read: websocket: close 1000 (normal)

Download Historical Snapshots

To build the historical model, I processed raw RIB (Routing Information Base) files. These are snapshots of the entire routing table as seen by a backbone router at a specific moment in time. Because BGP is a “chatter” protocol that only announces changes, these full table dumps are essential for reconstructing the state of the Internet at any point in the past.

I specifically fetched snapshots from February 1st at 12:00 UTC for every year in my timeline. To ensure a comprehensive view, I aggregated data from multiple global collectors maintained by the University of Oregon Route Views project.

Other excellent resources for this kind of data include:

How BGP Shapes the Global Internet Map

For this edition, I processed over 15 years of BGP snapshots and PeeringDB archives to build the Logical Dominance model. Reconstructing this history was easily the hardest part of the project. I quickly realized that reliable archival data for physical peering effectively vanishes before 2010, which set a hard limit on how far back I could take the timeline.

Defining Logical Dominance

Logical Dominance is calculated by summing the number of unique IPv4 addresses originated by an ASN and attributed to a given city. Overlapping prefixes are deduplicated using longest-prefix normalization so that no address space is counted twice.

The Scaling Problem: Why not IPv6?

You might notice this model focuses entirely on IPv4. While IPv6 is the future of the protocol, its sheer scale currently breaks the “Logical Dominance” math. I measure dominance by counting unique IP addresses; if I treated IPv4 and IPv6 as equals, the numbers wouldn’t just be skewed; they’d be nonsensical.

Consider the math: The smallest standard IPv6 assignment is a /64. That single subnet contains 18,446,744,073,709,551,616 addresses. You could fit the entire global IPv4 routing table (4,294,967,296 addresses) inside that one subnet 4.3 billion times over.

If I treated every IP equally, a single residential IPv6 connection would statistically obliterate a city hosting the entire legacy IPv4 Internet. Until I develop a weighted model for IPv6, perhaps based on prefix density rather than raw address count, IPv4 remains the only way to compare global “weight” on a 1:1 scale.

Finding the Truth in the Noise

Mapping a BGP prefix to a specific city is more difficult than you may think. A subnet might be registered to a corporate HQ but serve users thousands of miles away. To solve this, I built a prioritized “waterfall” of attribution logic. I check sources in order of reliability, stopping as soon as I find a match:

  1. Geofeeds (RFC 8805): These are machine-readable CSVs where network operators explicitly self-report where their subnets are used.
  2. Cloud Provider Ranges: I ingest live IP lists from AWS, Google Cloud, and others, mapping logical regions (like eu-west-1) to their physical locations (Dublin).
  3. Network Hints (Communities & Next-Hops): At this point, I look to the routing table itself for hints. If a prefix is only announced at the London Internet Exchange, or tagged with a “London” BGP Community, I attribute it there.
  4. Historical WHOIS: My final fallback for specific location data is the APNIC/RIPE databases.
  5. Footprint Heuristic: For anything remaining, I assign the IP weight to every city where that network maintains physical peering capacity as listed in PeeringDB.

This approach ensures that accurate, granular data (like a specific cloud region) always overrides broad, administrative data (like a generic WHOIS entry).

Building this pipeline presented unique engineering hurdles; here are the most significant ones:

The Local Cache

Downloading 15 years of archives is slow. I threw together a quick file-based cache to avoid hitting the network repeatedly. It was the simplest code I wrote but easily the most valuable, turning 30-minute download waits into near-instant local reads.

RAM remains stubbornly finite

Loading millions of IP prefixes, WHOIS records, PeeringDB entries, and their associated metadata into a standard in-memory map consumes gigabytes of RAM instantly. Frustratingly, my laptop only has so much. To avoid out-of-memory errors I built a custom on-disk trie data structure using BadgerDB v4, which is a Go KV store built on an LSM tree, which makes IP prefix lookups very efficient. I might show it off in a later blog post after I clean it up a little bit. By using IP prefixes as keys in a sorted KV store, I can perform efficient longest-prefix matching directly against the disk.

Cleaning Up the Spaghetti

While investigating all of these different data sources, I ended up writing several programs that generated output of different shapes that would be used by other programs. It all made sense to me at the time but it spiraled out of control into a confusing mess. Now, I have one script for generating this city data. I was only able to do this because of the improvements mentioned above: caching and using on-disk data structures. Now, the script has clear stages of:

  • Fetch: Downloads and caches raw data (WHOIS, BGP, PeeringDB).
  • Index: Builds searchable on-disk tries and resolves authoritative network names from RIRs.
  • Process: Scans BGP routes and attributes each prefix using the various data sources mentioned above.
  • Output: Produces clean, normalized city results without duplicate entries (e.g., merging “Seoul” and “SEOUL”).

What Changed When IP Dominance Was Added

When I layered IP dominance onto the physical map, many additional cities became visible.

Before:

Before

After:

After
World (before and after adding BGP data).

In earlier versions, visibility depended heavily on registered Internet Exchange Points. That highlighted the traditional coastal hubs and major peering metros. But once routing table data was incorporated, the map revealed cities without major IXPs. These are places with substantial address space and large originating networks, even if they do not host a major public exchange. This is most noticeable in India, Japan, China, Indonesia, and in secondary metros beyond traditional hubs in the EU and United States.

Before:

Before

After:

After
Asia on the map (before and after adding BGP data).

The physical meeting points of networks only tell us a part of the story. The global routing table reveals where address space is actually controlled and originated. Some cities carry significant weight without being major public peering hubs. The IP dominance layer makes that distinction visible.

Before:

Before

After:

After
Europe on the map (before and after adding BGP data).

The Chinese Internet

The Chinese internet is giant, but it presents a unique attribution challenge. Because so much of China’s domestic routing remains internal to national carriers, the global BGP table often only sees these massive networks when they peer at international hubs like Hong Kong, Los Angeles, or Frankfurt. An earlier version of my attribution code ended up adding all of China’s IP space to these select few international hubs, which was clearly incorrect. It looked like China Telecom was the biggest ISP in Germany, which made it appear that China Telecom dominated Germany. It does not, at least not yet. To fix this, I implemented specific logic for China-based networks. I used pattern matching to parse provincial hints from APNIC WHOIS data. This mapped prefixes like GD or SH to their respective provincial capitals. I also linked ASNs to their parent organizations in PeeringDB to prevent Chinese networks from being misattributed to foreign exchange points. This resolved attribution for the vast majority of prefixes. Any remaining IP space attributed only at the country level is distributed across major domestic hubs.

The result is a far more realistic view of China’s internal internet topology.

Ghost Networks and Spurious ASNs

Not every entry in the global routing table represents a real network with a physical footprint. While investigating the data, I found several “spurious” Autonomous Systems that I had to filter out to keep the map accurate.

For example, I had to add safety checks to prevent “IP swallowing.” There is a massive 0.0.0.0/0 block often pinned to Australia in the APNIC database. Since 0.0.0.0/0 matches every single IPv4 address, that one entry would incorrectly claim the entire global IP space for Australia. I know they have a lot of open space down there, but that seemed excessive.

Another prominent example was the Department of Defense (DoD). The DoD holds several massive /8 blocks (like 7.0.0.0/8 and 11.0.0.0/8). While this space is technically routed, it does not represent commercial internet traffic. In early versions of my model, the registration data for these blocks linked them to administrative offices in New York City. This caused my script to dump millions of military IPs onto Manhattan and incorrectly made it look like the absolute center of the universe.

I also built a blocklist to ignore other non-geographic entities:

  • Administrative Containers: I filter out WHOIS entries containing IANA-NETBLOCK, CIDR-BLOCK, or ERX-NETBLOCK. These are typically placeholders for unassigned pools managed by regional registries rather than active networks.
  • Registry Placeholders: Specific ASNs like 721, 56, and 37069 often function as loopbacks or registry tests.

By explicitly ignoring these, the resulting map represents the actual commercial Internet rather than the administrative database of the Internet.

UX and Rendering

In addition to adding more data to the map, I’ve also made several improvements to the map itself.

Dynamic Cluster Grouping

Layering BGP data onto an already complex physical map created a major design challenge: information density. With hundreds of new cities “lighting up” globally, the map became significantly cluttered when zoomed out.

To solve this, I implemented Dynamic Cluster Grouping. Close-by cities now group together into aggregate hubs at low zoom levels, which then split into individual markers as you zoom in. This isn’t just a visual fix; by reducing the number of active SVG shapes in the DOM, it significantly improves panning performance on mobile devices.

Before:

Before

After:

After
Before and after dynamic cluster groupings.

Dynamic Cluster Grouping ensures the map remains legible, preventing the increased data density from overwhelming the map. When you click on a cluster, the details panel expands to list every city contained within that group.

Viewport Culling

I also introduced Viewport Culling. The map now only renders assets currently within your bounds. As you pan to a new region, cities “pop in” dynamically, ensuring the browser isn’t wasting resources on rendering things on the other side of the planet.

Updates to City Sizing

The visual size of cities on the map also now dynamically reflects their importance. Previously, cities were sized based only on their relative peering bandwidth. Now, their size depends on a weighted combination of aggregate peering bandwidth and IP dominance, contributing 80% and 20% to the size calculation respectively. Although this ratio is arbitrary and was picked for aesthetic reasons, peering bandwidth is a stronger signal of real traffic concentration than raw IP space alone, so I think it should be emphasized significantly more.

Enable/disable layers

Now, the map can be sliced into three layers: Cables, peering bandwidth, and IP allocations. There are controls that allow you to show or hide each of these layers individually.

I also added permalinks to make the map state fully shareable. The URL now encodes the current latitude, longitude, zoom level, selected year, and active text filters. If you zoom into Southeast Asia in 2016 and search for “Singapore”, that exact view can be copied and shared. The resulting link will look like this:

https://map.kmcd.dev/?lat=3.1625&lng=103.4033&z=5.00&year=2016&q=singapore

…which will show exactly how amazingly connected Singapore is when others click on it.

Better Exports

One of the most requested features for the map has been a way to export the current view for use in presentations, reports, posters, or just as a high-quality wallpaper.

Previously, I was using a standard Leaflet plugin for this, but it was not great. It would often fail in weird ways, leaving you with a glitched or incomplete rendering of the map. It also exported as PNG, which meant the beautiful vector data of the cables and cities was flattened into a low-resolution raster format.

Now there’s a new export button that renders an isolated SVG. Because the map itself is built on SVGs, this new export method is lossless. It respects your current zoom level and position, allowing you to focus on a specific region and generate an incredibly high-quality vector file that you can scale to any size without losing a single pixel of detail. Most images in this post were generated using this new export feature.

Show Me the Data

Another one of the biggest requests I’ve had in previous years is for access to the raw data behind the visualizations. For the 2026 edition, I have exposed the underlying JSON datasets that power the map. These files are curated from TeleGeography (for modern cables), PeeringDB (for IXPs), and historical data is curated from various sources including submarinenetworks.com and archived maps.

You can access these directly to build your own visualizations, analyze the growth of global bandwidth, or double check my numbers.

  • all_cables.json: The Core Map Data. A GeoJSON FeatureCollection containing all submarine cables. Each feature includes properties like name, rfs_year (Ready for Service), decommission_year, owners, and landing_points. This follows the standard GeoJSON format.
  • year-summaries.json: Brief textual descriptions of notable events or milestones for specific years, displayed in the footer.
  • city-dominance/{year}.json: Per-year JSON files (e.g., 2026.json) with detailed city-level peering capacity, regional information, and coordinates. Used for rendering city markers and calculating regional statistics.
  • meta.json: Metadata including the minimum and maximum years covered by the visualization.

See You Next Year

You might ask why I burned so much time manually attributing IP space when services like MaxMind or IPInfo already exist. The honest answer? Buying the data isn’t fun. The joy of this project comes from the archaeology and the work involved in bringing order to chaotic and disjointed datasets and transforming them into something beautiful.

This was a great project, and I am extremely happy with the results. If you’ve gotten this far without checking out the map, I’m impressed with your restraint, but here’s one more link for you to take a look: Explore the Map »

]]>Shell Log: Namastehttps://kmcd.dev/posts/shell-log-namaste/Mon, 16 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/shell-log-namaste/

The 'Work-Life Balance' Patch ]]>We had a critical hotfix to deploy. The bug was losing the company $10k a minute. I just needed to run the deployment script. Unfortunately, the Sysadmin team had just rolled out the new mandatory “Mental Health & Wellness” kernel module.

marcus@prod-deploy-01:~/app$ date
Fri May 21 17:01:02 PDT 2027

marcus@prod-deploy-01:~/app$ ./deploy.sh --force

      .--.
    .'_\/_'.
    '. /\ .'   WellnessOS v2.0 (Stable)
      "||"     "Namaste, Admin."
       ||
    /\ || /\
   (  \||/  )
    \  ||  /
     ' -- '

[!] COMMAND BLOCKED
Reason: It is currently 17:01. Working hours ended at 17:00.
Suggestion: Go home. Hug your family. Touch grass.

Are you kidding me? The site is down.

marcus@prod-deploy-01:~/app$ sudo ./deploy_hotfix.sh --force
[sudo] password for marcus:
[!] SUDO DENIED
Analysis: The use of 'sudo' implies urgency, which creates cortisol.
Cortisol is bad for your long-term health.
Action: This incident has been logged with 'People Ops' as a potential burnout risk.

I am going to throw this laptop out the window. Maybe I can trick it by changing the system time.

marcus@prod-deploy-01:~/app$ sudo date -s "16:00:00"
[!] TIME MANIPULATION DETECTED
Analysis: Denial is the first stage of grief.
Action: Playing soothing ocean sounds via PC Speaker.
Locking terminal for 10 minutes for a mandatory 'Mindfulness Break'.

It’s actually beeping at me. This is insane.

marcus@prod-deploy-01:~/app$ kill -9 -1

[!] VIOLENT LANGUAGE DETECTED
The command 'kill' is on the banned words list for maintaining a positive workplace culture.
Please consider using non-violent alternatives such as:
  - 'transition-process'
  - 'release-to-universe'
  - 'conscious-uncoupling'

Action: Replacing your shell with 'eliza-therapy-bot'.

Hello, Marcus. I see you are trying to 'kill' something.
Do you feel like your father didn't listen to you enough?
> _
]]>HTTP/2 From Scratch: Part 1https://kmcd.dev/posts/http2-from-scratch-part-1/Wed, 11 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/http2-from-scratch-part-1/ <p> <img hspace="5" src="https://kmcd.dev/posts/http2-from-scratch-part-1/cover.svg" /> </p> Re-building the web in Go to learn more about it If you have ever opened a terminal and manually typed an HTTP/1.1 request, you know there is a certain beauty in its simplicity. You send a few lines of plain text and the server responds with more text. It is human-readable and easy to debug, but it is also remarkably inefficient for the modern web. If you haven’t done that, I’m hoping I can get you to use telnet this way for the first time today.

As applications have grown more complex and require many disparate streams of data, we seem to have reached the limit to what a text-based protocol can do. In this series, we are going to stop using the high-level abstractions provided by the Go standard library (mostly) and build an HTTP/2 implementation from the ground up. This is the way that I’ve learned best, so maybe you can join me on this journey.

The Simplicity of the Past

To understand why we need HTTP/2, we first have to look at how simple things used to be. You can still see the inner workings of the old web by using telnet or nc (netcat) to talk to a server. If you connect to port 80 and manually type a request, the interaction looks like this:

$ telnet kmcd.dev 80
Connected to kmcd.dev.
Escape character is '^]'.

GET / HTTP/1.1
Host: kmcd.dev

After hitting enter twice, the server sends back a text response (Some of the CloudFlare headers removed for brevity):

HTTP/1.1 301 Moved Permanently
Date: Wed, 28 Jan 2026 14:17:33 GMT
Content-Length: 0
Connection: keep-alive
Location: https://kmcd.dev/
Server: cloudflare

This response is a 301 status code, telling you to use the HTTPS version of the website, because plain-text HTTP is for heathens. But regardless, this is extremely insightful and cuts out many of the complexities that tooling can add. You can read every byte, understand every header, and even type the requests by hand. However, this transparency comes at a cost.

For completeness, here is a similar setup using HTTPS via the openssl CLI client:

$ openssl s_client -connect kmcd.dev:443 -servername kmcd.dev
openssl command output (click to expand) 
CONNECTED(00000006)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R4
verify return:1
depth=1 C = US, O = Google Trust Services, CN = WE1
verify return:1
depth=0 CN = kmcd.dev
verify return:1
write W BLOCK
---
Certificate chain
 0 s:/CN=kmcd.dev
   i:/C=US/O=Google Trust Services/CN=WE1
 1 s:/C=US/O=Google Trust Services/CN=WE1
   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R4
 2 s:/C=US/O=Google Trust Services LLC/CN=GTS Root R4
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDkDCCAzWgAwIBAgIRANuMVwoYX/O/E75qT9V5T9AwCgYIKoZIzj0EAwIwOzEL
MAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczEMMAoG
A1UEAxMDV0UxMB4XDTI2MDEwMTA1MjA0OVoXDTI2MDQwMTA2MjA0M1owEzERMA8G
A1UEAxMIa21jZC5kZXYwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATxkKGuge8n
FcjibwoBq1QKBeq/KDwkorBy7MUuyYMcvhfdB9QGMalqF8wDtkruPStPe6rMAUjZ
NAoDSKmJwrdco4ICQDCCAjwwDgYDVR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsG
AQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFKVQVjLBxQ3apOR8KM1a/BjG
BNyzMB8GA1UdIwQYMBaAFJB3kjVnxP+ozKnme9mAeXvMk/k4MF4GCCsGAQUFBwEB
BFIwUDAnBggrBgEFBQcwAYYbaHR0cDovL28ucGtpLmdvb2cvcy93ZTEvMjR3MCUG
CCsGAQUFBzAChhlodHRwOi8vaS5wa2kuZ29vZy93ZTEuY3J0MBMGA1UdEQQMMAqC
CGttY2QuZGV2MBMGA1UdIAQMMAowCAYGZ4EMAQIBMDYGA1UdHwQvMC0wK6ApoCeG
JWh0dHA6Ly9jLnBraS5nb29nL3dlMS9mLUxxQnNPOTBnWS5jcmwwggEDBgorBgEE
AdZ5AgQCBIH0BIHxAO8AdQDRbqmlaAd+ZjWgPzel3bwDpTxBEhTUiBj16TGzI8uV
BAAAAZt4N1MJAAAEAwBGMEQCIFpnx0F4+HFZiAAZp/S1OXTqUGE2XXEJreljtBiX
52ezAiA8IqlGlvYzT9to8aWkfCmvd5/zxpktLVxjPAZhTtvO/QB2AA5XlLzzrqk+
MxssmQez95Dfm8I9cTIl3SGpJaxhxU4hAAABm3g3Ui8AAAQDAEcwRQIgYpHRsDzn
0EyvJeasmiFJHnVAgNjO9nVLhHUkgMMluQ0CIQCLxOjYLJUGwBMAF4RG2L7J+P/A
4g2ojvjXSOqUhIpJmTAKBggqhkjOPQQDAgNJADBGAiEAgfuMNfYBAbPoYqIJ7TyY
y1uNp7kzpQShTsBpjgQrqTsCIQC22/6iBI+qiTSgWIrF7tKFil8+XyoMdmf8CTeI
GBAV/w==
-----END CERTIFICATE-----
subject=/CN=kmcd.dev
issuer=/C=US/O=Google Trust Services/CN=WE1
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 2789 bytes and written 368 bytes
---
New, TLSv1/SSLv3, Cipher is AEAD-CHACHA20-POLY1305-SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : AEAD-CHACHA20-POLY1305-SHA256
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1769674847
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
GET / HTTP/1.1
Host: kmcd.dev

HTTP/1.1 200 OK
Date: Thu, 29 Jan 2026 08:20:53 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Cache-Control: max-age=31536000, public
cf-cache-status: DYNAMIC
Link: <https://fonts.googleapis.com>; rel="preconnect"
referrer-policy: no-referrer
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
Vary: accept-encoding
Server: cloudflare
alt-svc: h3=":443"; ma=86400

3d8d
<!doctype html><html lang=en><head><meta name=generator content="Hugo 0.152.2">...[snip]...</head></html>
0

The response has been trimmed for brevity, but this shows that you can use a TLS-wrapped socket to access HTTPS websites using a plain-text interface similar to telnet. This shows how the “S” in “HTTPS” is simply HTTP wrapped with TLS.

Head-of-line blocking

The primary issue with HTTP/1.1 is head-of-line blocking. Even if you have a high-speed fiber connection, a single TCP stream can only process one request at a time. If your browser needs a tiny CSS file but it is stuck behind a massive high-resolution image, that CSS file simply waits. In the early days of the web, when pages were just a few kilobytes of text and a couple of images, this was a minor inconvenience. Today, the average webpage loads hundreds of individual assets.

Web developers and browser makers are clever individuals. They fixed this by opening multiple TCP connections to the same server. Browsers usually allow up to six parallel connections per domain name. If developers need more than that, they would host assets on different domain names using a technique called domain sharding. This set of techniques did make loading web assets faster, but this is a heavy-handed solution that comes as a cost. Each new connection requires a full TCP handshake and a TLS negotiation which means needing more powerful servers and clients to deal with this extra crypto negotiation.

Beyond the connection limits, HTTP/1.1 is incredibly verbose. Every single request sends the same headers over and over again. Your user agent and cookie headers are almost identical for every image on a page, yet the protocol forces you to upload that redundant text hundreds of times. On mobile networks with limited upload bandwidth, this overhead becomes a measurable performance tax.

Persistent Connections and Pipelining

Before we dive into the binary world of HTTP/2, it’s worth noting that HTTP/1.1 didn’t just sit idly by while performance suffered. It introduced two key concepts to mitigate the overhead of opening new connections: Persistent Connections (Keep-Alive) and Pipelining.

In the original HTTP/1.0 days, every single request required a brand-new TCP handshake. HTTP/1.1 changed the default behavior to “Keep-Alive,” allowing a single TCP connection to stay open for multiple subsequent requests. This saved the “tax” of the initial handshake for every image or script.

Then came Pipelining. The idea was brilliant on paper: instead of waiting for a response before sending the next request, a client could fire off three or four requests in a row. While you could send requests in a batch, the server was strictly required to send the responses back in the exact same order they were received. If the first request in the “pipe” was a slow database query and the second was a tiny static file, the static file was still stuck waiting for the database; the classic “Head-of-Line Blocking” problem. Because of buggy implementations in middleware and browsers, pipelining never saw widespread adoption and is disabled by default in almost every modern browser for HTTP/1. It was a valiant attempt at concurrency that ultimately proved the need for a more radical shift in how we frame data.

Entering the Binary World

HTTP/2 solves these problems by moving away from text entirely. As defined in RFC 9113, Section 4, it introduces a binary framing layer that allows us to interleave multiple requests and responses over a single connection. This technique is called multiplexing. By breaking every message into small, independent frames, a slow request no longer blocks the ones behind it. The protocol can send a piece of an image, then a piece of a script, then another piece of the image, all within the same stream.

In this binary world, we stop thinking about lines of text and start thinking about frames. Every piece of data is wrapped in a header that tells the receiver exactly what it is, how big it is, and which request it belongs to. This makes parsing significantly faster for machines. In HTTP/1.1, the server has to read the text and look for newlines to know where a header ends. In HTTP/2, the server knows exactly how many bytes to read next.

Negotiating the Connection

Since the web is built on backward compatibility, a client needs a way to tell a server it wants to use HTTP/2 without breaking things for older servers. We do this using ALPN (Application-Layer Protocol Negotiation) during the TLS handshake. This allows the client and server to agree on a protocol before they ever start sending application data.

Once the encrypted tunnel is established, the client must send a connection preface. This is a 24-byte magic string that serves as a final confirmation. The string looks like this, as documented in Section 3.5 in RFC-9113:

PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n

If the server receives this string successfully, it will speak HTTP/2 from then on. It will respond with its own settings. If it does not, it will likely drop the connection. This prevents a modern client from sending binary data to an old server that is still expecting plain text.

The Implementation in Go

The following script is the starting point for our project. It uses the crypto/tls package to handle the encryption but stops right at the moment the HTTP/2 state machine should take over. We define the mandatory preface and configure our TLS dialer to negotiate for the “h2” protocol.

go/client.go (click to expand)View on GitHub 
package main

import (
	"crypto/tls"
	"fmt"
	"log"
)

const (
	// The "Magic" preface required by RFC 9113
	preface = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
	server  = "kmcd.dev:443"
)

func main() {
	// We configure TLS to specifically look for "h2" via ALPN
	config := &tls.Config{
		NextProtos: []string{"h2"},
	}

	// Dial the server and perform the handshake
	conn, err := tls.Dial("tcp", server, config)
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	// Ensure the server actually agreed to speak HTTP/2
	state := conn.ConnectionState()
	if state.NegotiatedProtocol != "h2" {
		log.Fatalf("Server did not negotiate HTTP/2: %s", state.NegotiatedProtocol)
	}

	fmt.Printf("Connected to %s using %s\n", server, state.NegotiatedProtocol)

	// Send the Connection Preface to initialize the H2 session
	_, err = conn.Write([]byte(preface))
	if err != nil {
		log.Fatalf("Failed to send preface: %v", err)
	}

	fmt.Println("Preface sent successfully. The connection is open.")
}

This is actually quite amazing. A lot of hard work is being done to give us a TLS-wrapped connection and we’ve also used TLS to negotiate the HTTP/2 protocol for us. TLS is a complex protocol in its own right, so we’ll rely on Go’s crypto/tls package and focus entirely on HTTP/2 from this point forward.

What Happens Next

At this point, the connection is technically alive, but it is silent. The server is now expecting us to handle binary frames. If you were to run this code and try to read from the connection, you would see a stream of bytes representing the server’s initial configuration.

In the next post, we leave the comfort of strings behind and dive into the encoding/binary package. We’ll build a parser for the mandatory 9-byte frame header, learn how to mask bits, and start interpreting the hex constants that define the language of the modern web.

]]>IRC Log: Standuphttps://kmcd.dev/posts/irc-log-standup/Mon, 09 Feb 2026 10:00:00 +0000https://kmcd.dev/posts/irc-log-standup/ <p> <img hspace="5" src="https://kmcd.dev/posts/irc-log-standup/cover.svg" /> </p> The lead is away, so the team holds standup in IRC. It quickly descends into abstract art and magenta-colored login pages. The lead is at a conference, so standup is happening in #engineering-team. It went about as well as expected.

[09:00] *** Topic: Daily Standup | Lead is OOO at GopherCon
[09:01] <senior_dev> Morning everyone. Since Dave is at the conference, let’s just do a quick text-based standup here. Keep it brief.
[09:02] <junior_dev> Yesterday: Fixed the CSS bug on the login page. Today: Investigating why the login page is now magenta. No blockers.
[09:03] <backend_ben> Yesterday: Optimized the API query. Today: Realizing I optimized the wrong API. Redoing it. Blocker: My coffee machine is leaking.
[09:04] <qa_queen> Yesterday: Found 14 bugs in Ben’s “optimized” API. Today: Testing the magenta login page. It’s… a choice.
[09:05] <junior_dev> it’s a bold choice!
[09:05] <senior_dev> Yesterday: PR reviews. Today: Dealing with the magenta crisis and Ben’s wet kitchen. Blocker: None.
[09:06] <intern_ian> Yesterday: Learned how to use git merge. Today: Learned how to use git reset --hard. Blocker: The repo is currently in a state that I can only describe as “abstract art.”
[09:07] * senior_dev checks the commit log
[09:07] <senior_dev> Ian, why is there a commit titled “whoopsie”?
[09:08] <intern_ian> it seemed descriptive at the time
[09:10] <qa_queen> I’ve found another bug. If you click the magenta button 50 times, it plays the Nyan Cat theme. Anyway, I’m just sitting here waiting for the next set of bugs to drop—I mean, features.
[09:10] <junior_dev> that’s not a bug, that’s an engagement feature.
[09:11] <senior_dev> Okay, standup over. Ben, fix your API. Ian, don’t touch anything. Kevin, change the color back.
[09:12] <junior_dev> but the vibes are so good right now
[09:13] *** senior_dev is now known as senior_dev_drinking_early
]]>HTTP/1.1 From Scratchhttps://kmcd.dev/posts/http1.1-from-scratch/Wed, 04 Feb 2026 00:00:00 +0000https://kmcd.dev/posts/http1.1-from-scratch/ <p> <img hspace="5" src="https://kmcd.dev/posts/http1.1-from-scratch/cover.svg" /> </p> When the web became whole. HTTP/1.1 has become synonymous with “HTTP/1” because it made several steps towards enabling the scale that the web was starting to experience in the late 1990s. It took the foundational concepts we explored in HTTP/0.9 and HTTP/1.0 and made several advancements and adjustments that enable the web to scale for almost two decades.

While newer protocols like HTTP/2 and HTTP/3 have since arrived with their own improvements, HTTP/1.1 remains a non-negotiable requirement… well, almost. There are a few who believe that it is time to kill HTTP/1.1 and some believe it would immediately reduce bot traffic. It has historically been the default transport for the web and the protocol that servers and clients fall back on. Its simplicity and power are why, even now, a massive portion of internet traffic flows over HTTP/1.1.

Let’s start looking at the new features in HTTP/1.1 that made it a such a sturdy pillar of the web for so long and then build a server in Go that implements them from scratch.

Improvements over HTTP/1.0

HTTP/1.1 wasn’t just a bump in the version number; it was a targeted response to the scaling bottlenecks of the 90s. It introduced specific features to cut down on connection overhead and handle dynamic data more efficiently than 1.0 ever could.

Persistent Connections (Keep-Alive)

This is probably the most important performance improvement in HTTP/1.1. In HTTP/1.0, every single request required a new, separate TCP connection. Setting up a TCP connection is a multi-step handshake process that introduces significant latency. For a typical website that requires dozens of resources (CSS, JavaScript, images), this overhead added up quickly. And this was before the web was encrypted, so wrapping requests in TLS for “secure pages” would add two more round trips too this.

HTTP/1.1 introduced persistent connections by default. This allows the browser to send multiple requests over a single TCP connection, eliminating the repeated connection setup cost.

A client or server can signal that they wish to close the connection after a request by sending the Connection: close header. Otherwise, the connection is assumed to be “kept alive”.

The Host Header: Enabling Virtual Hosting

In the early web, a server at a specific IP address hosted a single website. As the web grew, this became incredibly inefficient. The Host header, which became mandatory in HTTP/1.1, solved this. It specifies the domain name of the server the client is trying to reach. This allowed a single server (with a single IP address) to host hundreds or thousands of different websites, a practice known as virtual hosting. This was a critical innovation for the economic scalability of web hosting.

Chunked Transfer Encoding

Before HTTP/1.1, for a server to send a response, it had to know its exact size beforehand to set the Content-Length header. This was fine for static files but a major problem for dynamically generated content. What if you were streaming a large video or generating a big HTML page on the fly? You’d have to buffer the entire response in memory just to calculate its size.

Chunked Transfer Encoding elegantly solves this. The server can send the response body in a series of “chunks.” Each chunk is prefixed with its size in hexadecimal, followed by the chunk data itself. The stream is terminated by a final chunk of size 0.

Here’s what a chunked response looks like on the wire:

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

8\r\n
kmcd.dev\r\n
12\r\n
 is awesome!\r\n
0\r\n
\r\n

This allowed for much more efficient handling of dynamic content and laid the groundwork for streaming media.

Modern Caching with Cache-Control

HTTP/1.0 had basic caching headers, but HTTP/1.1 introduced the powerful Cache-Control header. This gave developers fine-grained control over how browsers and intermediate proxies cache resources. Directives like max-age, public, private, no-cache, and no-store allowed for sophisticated caching strategies, dramatically reducing bandwidth usage and improving load times.

A quick look at my own site’s headers shows this in action:

$ curl --http1.1 -I https://kmcd.dev
HTTP/1.1 200 OK
Date: Sat, 24 Jan 2026 10:12:55 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: max-age=31536000, public
Server: cloudflare

This tells any browser or CDN that it’s safe to cache this response for a full year, which is great for performance.

More Methods for RESTful APIs

HTTP/1.0 was primarily about GET, POST, and HEAD. HTTP/1.1 expanded the vocabulary of the web by adding new methods that were crucial for the development of RESTful APIs:

  • PUT: Create or replace a resource at a given URI.
  • PATCH: Partially modify a resource.
  • DELETE: Delete a resource at a given URI.
  • CONNECT: Used for creating tunnels, most notably for HTTPS through proxies.
  • OPTIONS: Describe the communication options for the target resource.
  • TRACE: Performs a message loop-back test along the path to the target resource.

100 Continue Status Code

When a client needs to send a large request body (like uploading a file), it can be inefficient to send the entire payload only for the server to reject it (e.g., due to authentication failure or size limits). The 100 Continue status code provides a solution.

The client can send the request headers with the Expect: 100-continue header and then wait.

PUT /images HTTP/1.1
Host: images.example.com
Content-Type: image/png
Content-Length: 500000
Expect: 100-continue

If the server is willing to accept the request, it responds with HTTP/1.1 100 Continue. The client then proceeds to send the request body. If the server is not going to accept it, it can immediately send a final error code like 413 Payload Too Large, and the client knows not to waste bandwidth sending the body.

Building a Simple HTTP/1.1 Server in Go

Now for the fun part. Let’s build a server that understands these new features. We’ll be pulling from the complete server code found here: main.go.

Handling Persistent Connections

To support keep-alive, our connection handler can’t just handle one request and then close the connection. It needs to loop, processing multiple requests on the same connection until the client or server decides to close it.

The structure of our server looks like this: ListenAndServe accepts new TCP connections and spins up a handleConnection goroutine for each one.

// ListenAndServe starts the server.
func (s *Server) ListenAndServe() error {
	// ... listener setup ...
	for {
		conn, err := l.Accept()
		if err != nil {
			return err
		}

		go func() {
			if err := s.handleConnection(conn); err != nil {
				slog.Error(fmt.Sprintf("http error: %s", err))
			}
		}()
	}
}

Now let’s look at the new handleConnection method. It contains an infinite loop that repeatedly calls handleRequest. Previously, this method didn’t exist because in HTTP/1.0 we only ever handled a single request per connection. The loop only exits if there’s a fatal error (timeout, protocol error, etc.) or if handleRequest signals that the connection should be closed like when the user sends a request with the header Connection: close.

func (s *Server) handleConnection(conn net.Conn) error {
	defer conn.Close()
	for {
		// handleRequest does the work of reading and responding
		shouldClose, err := s.handleRequest(conn)
		if err != nil {
			// io.EOF is a normal way for a persistent connection to end.
			if errors.Is(err, io.EOF) {
				return nil
			}
			return err
		}
		if shouldClose {
			return nil // Client requested a close, so we exit the loop.
		}
	}
}

Inside handleRequest, we determine if the connection should be closed by inspecting the Connection header.

// Default to keeping the connection alive for HTTP/1.1
req.Close = false 

// Check if the client or a previous handler wants to close the connection.
switch strings.ToLower(req.Header.Get("Connection")) {
case "close":
    req.Close = true
case "keep-alive":
	// This is the default, but we handle it explicitly.
    req.Close = false
}

Requiring the Host Header

This is a simple but crucial part of our server. After parsing the headers, we just check for the presence of the Host header. If it’s missing, we return an error and close the connection.

if _, ok := req.Header["Host"]; !ok {
    // We send an error response here in a real implementation.
    return true, errors.New("required 'Host' header not found")
}

Handling Chunked Bodies

This is the most complex part. We need to be able to both read a chunked request body from a client and send a chunked response.

Reading a Chunked Request

When we parse the request headers, if we see Transfer-Encoding: chunked, we know we can’t just use io.LimitReader with Content-Length. Instead, we need a special reader. Our chunkedBodyReader does just this.

type chunkedBodyReader struct {
	reader *bufio.Reader
	n      int64 // bytes left in current chunk
	err    error
}

func (r *chunkedBodyReader) Read(p []byte) (n int, err error) {
	if r.err != nil {
		return 0, r.err
	}
	// Do we need to read the size of the next chunk?
	if r.n == 0 {
		r.n, r.err = r.readChunkSize()
		if r.err != nil {
			return 0, r.err
		}
	}
	// If the next chunk size is 0, we're at the end.
	if r.n == 0 {
		return 0, io.EOF
	}
    // ... logic to read from the current chunk ...
}

The readChunkSize method is responsible for reading a line, parsing the hexadecimal chunk size, and preparing the reader to consume that many bytes.

Sending a Chunked Response

On the response side, things are even cooler. If our http.ResponseWriter implementation doesn’t have a Content-Length set when WriteHeader is called, we can automatically switch to using chunked encoding.

Our responseBodyWriter checks for this condition.

func (r *responseBodyWriter) writeHeader(conn io.Writer, proto string, headers http.Header, statusCode int) error {
	_, clSet := r.headers["Content-Length"]
	_, teSet := r.headers["Transfer-Encoding"]
	// If no length is set, we decide to use chunking.
	if !clSet && !teSet {
		r.chunkedEncoding = true
		r.headers.Set("Transfer-Encoding", "chunked")
	}
    // ... write headers ...
}

Then, the Write method, if chunkedEncoding is true, will write each chunk with the required formatting.

func (r *responseBodyWriter) Write(b []byte) (int, error) {
	// ... ensure headers are written ...

	if r.chunkedEncoding {
		// Write the chunk size in hex, followed by \r\n
		chunkSize := fmt.Sprintf("%x\r\n", len(b))
		if _, err := r.conn.Write([]byte(chunkSize)); err != nil {
			return 0, err
		}
	}

	// Write the actual chunk data
	n, err := r.conn.Write(b)
	if err != nil {
		return n, err
	}

	if r.chunkedEncoding {
		// Write the trailing \r\n for the chunk
		if _, err := r.conn.Write(nlcf); err != nil {
			return n, err
		}
	}

	return n, nil
}

Finally, after the last Write call, we send the terminal 0\r\n\r\n chunk to signal the end of the response.

Testing Our Server

Using command-line tools is the best way to see these features in action.

Testing Keep-Alive

We can use curl’s verbose mode (-v) to see connection reuse. We’ll make two requests to our server on the same command line.

$ curl --http1.1 -v http://127.0.0.1:9000/headers http://127.0.0.1:9000/status/204

*   Trying 127.0.0.1:9000...
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET /headers HTTP/1.1
> Host: 127.0.0.1:9000
> ...

< HTTP/1.1 200 OK
< Connection: keep-alive
< ...
* Connection #0 to host 127.0.0.1 left intact
* Found bundle for host 127.0.0.1: 0x1400084c0 [can pipeline]
* Re-using existing connection! (#0) with host 127.0.0.1
> GET /status/204 HTTP/1.1
> Host: 127.0.0.1:9000
> ...

< HTTP/1.1 204 No Content
< Connection: keep-alive
< ...
* Connection #0 to host 127.0.0.1 left intact

The key lines are Re-using existing connection! and Connection #0 to host 127.0.0.1 left intact, which confirm that the second request was sent over the same connection as the first.

Testing the Host Header

Using netcat (nc), we can manually craft an HTTP request. First, let’s try one without a Host header.

$ printf "GET / HTTP/1.1\r\n\r\n" | nc localhost 9000

The command returns nothing, and on the server side, we see our error log:

2026/01/24 19:34:39 ERROR http error: required 'Host' header not found

Success! Now let’s add the Host header.

$ printf "GET /headers HTTP/1.1\r\nHost: localhost\r\n\r\n" | nc localhost 9000
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json
Transfer-Encoding: chunked

61
{"Accept-Encoding":["gzip"],"Host":["localhost"],"User-Agent":["Go-http-client/1.1"]}
0

It works perfectly, and we even get a chunked response back!

Testing Chunked Encoding

Our server has an /echo/chunked endpoint that streams the request body right back to the response. We can use curl to send a chunked request to it.

# We send two chunks: "hello" and " world"
$ (
  printf "5\r\nhello\r\n";
  sleep 1;
  printf "6\r\n world\r\n";
  sleep 1;
  printf "0\r\n\r\n";
) | curl --http1.1 -X POST --header "Transfer-Encoding: chunked" -T - http://127.0.0.1:9000/echo/chunked

hello world

The command pipes a manually created chunked body into curl. curl sends it to our server, which reads it using chunkedBodyReader and writes it back using our chunked responseBodyWriter. The final output hello world confirms the whole process worked.

Conclusion

HTTP/1.1 was and still is an amazing protocol. It introduced connections re-use, virtual hosting, and streaming request and response bodies. The design choices made in HTTP/1.1 were so robust that they remain deeply embedded in the internet’s infrastructure today.

If HTTP/1.1 was so great, why was HTTP/2 created? And what’s the deal with HTTP/3? Stay tuned for the next post in this series where we start looking at HTTP/2.

See all of the code mentioned in this article here:

go/main.go (click to expand)View on GitHub 
package main

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"log"
	"log/slog"
	"math"
	"net"
	"net/http"
	"net/url"
	"strconv"
	"strings"
)

var nlcf = []byte{0x0d, 0x0a}

// Server is a simple HTTP/1.1 server.
type Server struct {
	Addr    string
	Handler http.Handler
}

// ListenAndServe starts the server.
func (s *Server) ListenAndServe() error {
	handler := s.Handler
	if handler == nil {
		handler = http.DefaultServeMux
	}
	l, err := net.Listen("tcp", s.Addr)
	if err != nil {
		return err
	}
	defer l.Close()

	for {
		conn, err := l.Accept()
		if err != nil {
			return err
		}

		go func() {
			if err := s.handleConnection(conn); err != nil {
				slog.Error(fmt.Sprintf("http error: %s", err))
			}
		}()
	}
}

func (s *Server) handleConnection(conn net.Conn) error {
	defer conn.Close()
	for {
		shouldClose, err := s.handleRequest(conn)
		if err != nil {
			if errors.Is(err, io.EOF) {
				return nil
			}
			return err
		}
		if shouldClose {
			return nil
		}
	}
}

func (s *Server) handleRequest(conn net.Conn) (bool, error) {
	// Limit headers to 1MB
	limitReader := io.LimitReader(conn, 1*1024*1024).(*io.LimitedReader)
	reader := bufio.NewReader(limitReader)

	reqLineBytes, _, err := reader.ReadLine()
	if err != nil {
		return true, fmt.Errorf("read request line error: %w", err)
	}
	reqLine := string(reqLineBytes)

	req := new(http.Request)
	var found bool

	req.Method, reqLine, found = strings.Cut(reqLine, " ")
	if !found {
		return true, errors.New("invalid method")
	}
	if !methodValid(req.Method) {
		return true, errors.New("invalid method")
	}

	req.RequestURI, reqLine, found = strings.Cut(reqLine, " ")
	if !found {
		return true, errors.New("invalid path")
	}
	if req.URL, err = url.ParseRequestURI(req.RequestURI); err != nil {
		return true, fmt.Errorf("invalid path: %w", err)
	}

	req.Proto = reqLine
	req.ProtoMajor, req.ProtoMinor, found = parseProtocol(req.Proto)
	if !found {
		return true, errors.New("invalid protocol")
	}

	req.Header = make(http.Header)
	for {
		line, _, err := reader.ReadLine()
		if err != nil && err != io.EOF {
			return true, err
		} else if err != nil {
			break
		}
		if len(line) == 0 {
			break
		}

		k, v, ok := bytes.Cut(line, []byte{':'})
		if !ok {
			return true, errors.New("invalid header")
		}
		req.Header.Add(strings.ToLower(string(k)), strings.TrimLeft(string(v), " "))
	}

	if _, ok := req.Header["Host"]; !ok {
		return true, errors.New("required 'Host' header not found")
	}

	switch strings.ToLower(req.Header.Get("Connection")) {
	case "keep-alive", "":
		req.Close = false
	case "close":
		req.Close = true
	}

	limitReader.N = math.MaxInt64

	ctx := context.Background()
	ctx = context.WithValue(ctx, http.LocalAddrContextKey, conn.LocalAddr())
	ctx, cancelCtx := context.WithCancel(ctx)
	defer cancelCtx()
	contentLength, err := parseContentLength(req.Header.Get("Content-Length"))
	if err != nil {
		return true, err
	}
	req.ContentLength = contentLength
	isChunked := req.Header.Get("Transfer-Encoding") == "chunked"
	if req.ContentLength == 0 && !isChunked {
		req.Body = noBody{}
	} else {
		if isChunked {
			req.Body = &chunkedBodyReader{
				reader: reader,
			}
		} else {
			req.Body = &bodyReader{
				reader: io.LimitReader(reader, req.ContentLength),
			}
		}
	}

	req.RemoteAddr = conn.RemoteAddr().String()

	w := &responseBodyWriter{
		req:     req,
		conn:    conn,
		headers: make(http.Header),
	}

	s.Handler.ServeHTTP(w, req.WithContext(ctx))
	if err := w.flush(); err != nil {
		return true, nil
	}
	return req.Close, nil
}

type noBody struct{}

func (noBody) Read([]byte) (int, error) { return 0, io.EOF }
func (noBody) Close() error             { return nil }

func parseContentLength(headerval string) (int64, error) {
	if headerval == "" {
		return 0, nil
	}
	return strconv.ParseInt(headerval, 10, 64)
}

func parseProtocol(proto string) (int, int, bool) {
	switch proto {
	case "HTTP/1.0":
		return 1, 0, true
	case "HTTP/1.1":
		return 1, 1, true
	}
	return 0, 0, false
}

func methodValid(method string) bool {
	switch method {
	case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete, http.MethodConnect, http.MethodOptions, http.MethodTrace:
		return true
	}
	return false
}

type bodyReader struct {
	reader io.Reader
}

func (r *bodyReader) Read(p []byte) (n int, err error) {
	return r.reader.Read(p)
}

func (r *bodyReader) Close() error {
	_, err := io.Copy(io.Discard, r.reader)
	return err
}

type chunkedBodyReader struct {
	reader *bufio.Reader
	n      int64 // bytes left in current chunk
	err    error
}

func (r *chunkedBodyReader) Read(p []byte) (n int, err error) {
	if r.err != nil {
		return 0, r.err
	}
	if r.n == 0 {
		r.n, r.err = r.readChunkSize()
		if r.err != nil {
			return 0, r.err
		}
	}
	if r.n == 0 {
		return 0, io.EOF
	}
	if int64(len(p)) > r.n {
		p = p[0:r.n]
	}
	n, err = r.reader.Read(p)
	r.n -= int64(n)
	if r.n == 0 && err == nil {
		// Read trailing \r\n
		b, err := r.reader.ReadByte()
		if err != nil {
			r.err = err
			return n, err
		}
		if b != '\r' {
			r.err = errors.New("missing \r after chunk")
			return n, r.err
		}
		b, err = r.reader.ReadByte()
		if err != nil {
			r.err = err
			return n, err
		}
		if b != '\n' {
			r.err = errors.New("missing \n after chunk")
			return n, r.err
		}
	}
	r.err = err
	return n, err
}

func (r *chunkedBodyReader) readChunkSize() (int64, error) {
	line, err := r.readLine()
	if err != nil {
		return 0, err
	}
	// chunkSize is hex
	n, err := strconv.ParseInt(strings.TrimSpace(string(line)), 16, 64)
	if err != nil {
		return 0, err
	}
	if n == 0 {
		// Read trailers
		for {
			line, err := r.readLine()
			if err != nil {
				return 0, err
			}
			if len(line) == 0 {
				break
			}
		}
	}
	return n, nil
}

func (r *chunkedBodyReader) readLine() (string, error) {
	var line []byte
	for {
		b, err := r.reader.ReadByte()
		if err != nil {
			return "", err
		}
		if b == '\n' {
			break
		}
		line = append(line, b)
	}
	return strings.TrimRight(string(line), "\r"), nil
}

func (r *chunkedBodyReader) Close() error {
	_, err := io.Copy(io.Discard, r)
	return err
}

type responseBodyWriter struct {
	req             *http.Request
	conn            net.Conn
	sentHeaders     bool
	headers         http.Header
	chunkedEncoding bool
	bodyBuffer      *bytes.Buffer
}

func (r *responseBodyWriter) Header() http.Header {
	return r.headers
}

func (r *responseBodyWriter) Write(b []byte) (int, error) {
	if !r.sentHeaders {
		if r.headers.Get("Content-Type") == "" {
			r.headers.Set("Content-Type", http.DetectContentType(b))
		}
		r.WriteHeader(http.StatusOK)
	}

	if r.chunkedEncoding {
		chunkSize := fmt.Sprintf("%x\r\n", len(b))
		if _, err := r.conn.Write([]byte(chunkSize)); err != nil {
			return 0, err
		}
	}

	n, err := r.conn.Write(b)
	if err != nil {
		return n, err
	}

	if r.chunkedEncoding {
		if _, err := r.conn.Write(nlcf); err != nil {
			return n, err
		}
	}

	return n, nil
}

func (r *responseBodyWriter) Flush() {
	if !r.sentHeaders {
		r.WriteHeader(http.StatusOK)
	}
	if flusher, ok := r.conn.(interface{ Flush() error }); ok {
		flusher.Flush()
	}
}

func (r *responseBodyWriter) flush() error {
	if r.chunkedEncoding {
		if _, err := r.conn.Write([]byte("0\r\n\r\n")); err != nil {
			return err
		}
	}

	r.writeBufferedBody()

	return nil
}

func (r *responseBodyWriter) WriteHeader(statusCode int) {
	if r.sentHeaders {
		slog.Warn(fmt.Sprintf("WriteHeader called twice, second time with: %d", statusCode))
		return
	}

	r.writeHeader(r.conn, r.req.Proto, r.headers, statusCode)
	r.sentHeaders = true
	r.writeBufferedBody()
}

func (r *responseBodyWriter) writeBufferedBody() {
	if r.bodyBuffer != nil {
		_, err := r.conn.Write(r.bodyBuffer.Bytes())
		if err != nil {
			slog.Error("Error writing buffered body", "err", err)
		}
		r.bodyBuffer = nil
	}
}

func (r *responseBodyWriter) writeHeader(conn io.Writer, proto string, headers http.Header, statusCode int) error {
	_, clSet := r.headers["Content-Length"]
	_, teSet := r.headers["Transfer-Encoding"]
	if !clSet && !teSet {
		r.chunkedEncoding = true
		r.headers.Set("Transfer-Encoding", "chunked")
	}

	if r.req.Close {
		r.headers.Set("Connection", "close")
	} else {
		r.headers.Set("Connection", "keep-alive")
	}

	if _, err := io.WriteString(conn, proto); err != nil {
		return err
	}
	if _, err := conn.Write([]byte{' '}); err != nil {
		return err
	}
	if _, err := io.WriteString(conn, strconv.FormatInt(int64(statusCode), 10)); err != nil {
		return err
	}
	if _, err := conn.Write([]byte{' '}); err != nil {
		return err
	}
	if _, err := io.WriteString(conn, http.StatusText(statusCode)); err != nil {
		return err
	}
	if _, err := conn.Write(nlcf); err != nil {
		return err
	}
	for k, vals := range headers {
		for _, val := range vals {
			if _, err := io.WriteString(conn, k); err != nil {
				return err
			}
			if _, err := conn.Write([]byte{':', ' '}); err != nil {
				return err
			}
			if _, err := io.WriteString(conn, val); err != nil {
				return err
			}
			if _, err := conn.Write(nlcf); err != nil {
				return err
			}
		}
	}
	if _, err := conn.Write(nlcf); err != nil {
		return err
	}
	return nil
}

func main() {
	addr := "127.0.0.1:9000"
	mux := http.NewServeMux()
	mux.Handle("/", http.FileServer(http.Dir(".")))
	mux.HandleFunc("/echo", func(w http.ResponseWriter, r *http.Request) {
		defer r.Body.Close()
		b, err := io.ReadAll(r.Body)
		if err != nil {
			w.WriteHeader(400)
			return
		}
		w.Write(b)
	})
	mux.HandleFunc("/echo/chunked", func(w http.ResponseWriter, r *http.Request) {
		defer r.Body.Close()
		io.Copy(w, r.Body)
	})
	mux.HandleFunc("/status/{status}", func(w http.ResponseWriter, r *http.Request) {
		status, err := strconv.ParseInt(r.PathValue("status"), 10, 64)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			io.WriteString(w, fmt.Sprintf("error: %s", err))
			return
		}
		w.WriteHeader(int(status))
	})
	mux.HandleFunc("/headers", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Add("content-type", "application/json")
		json.NewEncoder(w).Encode(r.Header)
	})
	mux.HandleFunc("/nothing", func(w http.ResponseWriter, r *http.Request) {})
	s := Server{
		Addr:    addr,
		Handler: mux,
	}
	log.Printf("Starting web server: http://%s", addr)
	if err := s.ListenAndServe(); err != nil {
		log.Fatal(err)
	}
}

]]>WHOIS is dead, long live RDAPhttps://kmcd.dev/posts/whois-from-scratch/Tue, 06 Jan 2026 10:00:00 +0000https://kmcd.dev/posts/whois-from-scratch/

WHOIS is dead. To memorialize this piece of internet history, let's build a tiny implementation from scratch. ]]>The whois protocol is dead. For decades, it was a fundamental tool for network reconnaissance, but its time has passed. The protocol was officially sunset for all generic top-level domains in early 2025, replaced by the more modern, web-based protocol, RDAP.

So why talk about WHOIS now? To pay our respects. Because the WHOIS protocol is so simple, it makes a perfect case study for basic network programming and a window into an earlier era of the internet. To help memorialize this piece of internet history, we will build a tiny implementation from scratch and understand why its death was necessary in the process.

What is WHOIS?

Think of WHOIS as the internet’s public directory. Its primary job is to resolve a domain name into a set of administrative details.

When you query example.com, you aren’t asking for the website content; you are asking for the paper trail. A standard response returns the Registrar (the vendor, such as Namecheap or GoDaddy), the Name Servers (which direct traffic), and key dates regarding the domain’s creation and expiration. In the early days of the web, this output also listed the owner’s full name, address, and phone number. Today, privacy regulations like GDPR have largely forced that personal information behind generic “Redacted for Privacy” placeholders.

Who provides this data?

WHOIS data is a requirement enforced by ICANN (Internet Corporation for Assigned Names and Numbers). ICANN sets the rules, Registries (like Verisign for .com) manage the master lists for their TLDs, and Registrars (like Google Domains or Namecheap) sell the names to you. Domain registrars are contractually obligated to maintain this registration data and make it available to the public.

Why do we need it?

While often used by developers to check if a cool side project name is taken, WHOIS is critical infrastructure for maintaining the internet’s health.

Despite these redactions, the protocol remains vital for security. ICANN mandates that every domain record must publicly display an abuse contact email and phone number. This provides a direct line for network operators to report domains hosting malware, phishing schemes, or spam.

Security researchers have also pivoted their tactics. instead of looking for a specific person, they look for digital fingerprints. If a cluster of 500 suspicious domains appears on the network, registered simultaneously via the same obscure Name Server and Registrar, it strongly suggests a coordinated botnet. You don’t need to know the name of the attacker to know the assets are connected.

Investigative journalists use historical WHOIS data to map state-sponsored disinformation campaigns. For modern investigations, RDAP introduces “tiered access,” theoretically allowing vetted professionals to request unredacted data for legitimate purposes, though this process is still maturing. Also, a new initiative called RDRS aims to standardize access to nonpublic registration data for legitimate purposes.

How WHOIS Works

The WHOIS protocol, defined in RFC 3912, is a simple exchange over a TCP connection.

  1. CONNECT: The client opens a TCP socket to a WHOIS server (on port 43).
  2. ASK: The client sends the query: a single line of text like example.com, terminated by a carriage return and line feed (<CR><LF>).
  3. RESPONSE: The server streams back the registration data as plain text.
  4. DISCONNECT: The server kills the connection.

There are no headers, no authentication, and no complex data formats (more on that later). It is quite literally one of the simplest protocols imaginable. This simplicity makes it a good candidate for a small project to demonstrate basic networking concepts.

Building a WHOIS Server

Let’s turn theory into code. Because the protocol is so trivial, we can implement a functional server in Go using a few lines of code and the Go standard library. We can then verify that it works using the tools already installed on your machine, like telnet or the whois command itself.

WHOIS Server Implementation

We’ll add a records map to hold fake domain data and implement a handleConnection function to process queries and send back the corresponding record.

whois-server/main.go (click to expand)View on GitHub 
package main

import (
	"bufio"
	"fmt"
	"log"
	"net"
	"strings"
)

// whoisData now holds a single, static WHOIS record for debugging purposes.
var whoisData = map[string]string{
	"google.com": `Domain Name: google.com
Registrar: My Go Server
Creation Date: 2025-12-15T00:00:00Z
`,
	"example.com": `Domain Name: example.com
Registrar: My Go Server
Creation Date: 2025-12-15T00:00:00Z
`,
}

func handleConnection(conn net.Conn) {
	log.Printf("new connection from %s", conn.RemoteAddr())
	defer log.Printf("connection to %s closed", conn.RemoteAddr())
	defer conn.Close()

	scanner := bufio.NewScanner(conn)
	var clientQuery string

	// Read the first line from the client.
	if scanner.Scan() {
		clientQuery = strings.TrimSpace(scanner.Text())
	}

	if err := scanner.Err(); err != nil {
		log.Printf("Error reading from client: %v", err)
		return
	}

	if clientQuery == "" {
		log.Printf("Client disconnected without sending a query or sent an empty query.")
		// Send a minimal response in case the client expects *something*
		_, err := fmt.Fprint(conn, "No query provided.\r\n")
		if err != nil {
			log.Printf("Error writing empty query response: %v", err)
		}
		return
	}

	log.Printf("Received query: %q", clientQuery)

	var response string
	if data, ok := whoisData[strings.ToLower(clientQuery)]; ok {
		response = strings.ReplaceAll(data, "\n", "\r\n")
	} else {
		response = fmt.Sprintf("No match for %s\r\n", clientQuery)
	}

	// Write the response and close the connection.
	_, err := fmt.Fprint(conn, response)
	if err != nil {
		log.Printf("Error writing response to client: %v", err)
	}
}

func main() {
	addr := ":43"
	listener, err := net.Listen("tcp", addr)
	if err != nil {
		log.Fatalf("Error listening: %v", err)
	}
	defer listener.Close()
	log.Printf("Static WHOIS server listening on %s", addr)

	for {
		conn, err := listener.Accept()
		if err != nil {
			log.Printf("Error accepting connection: %v", err)
			continue
		}
		go handleConnection(conn)
	}
}

To run the server, execute:

go run ./whois-server

With the server running, you can now test it with telnet and whois:

telnet localhost 43
# Trying ::1...
# Connected to localhost.
# Escape character is '^]'.
# example.com
# Domain Name: example.com
# Registrar: My Go Server
# Creation Date: 2025-12-15T00:00:00Z
# Connection closed by foreign host.

whois -h localhost google.com
# Domain Name: google.com
# Registrar: My Go Server
# Creation Date: 2025-12-15T00:00:00Z

Real-World Complications

Our server works for the domains stored in its local records map, but the real WHOIS system is a distributed, federated system of registries and registrars, not a single database.

This leads to concepts like “thin” and “thick” lookups. A “thick” registry (like .org) holds all the data, and one query is enough. A “thin” registry (like .com) only knows which registrar manages a domain (e.g., GoDaddy, Namecheap). A whois client querying a “thin” registry gets a referral and must make a second query to the correct registrar’s WHOIS server to get the full details.

This system is brittle, relying on parsing unstructured text to find the referral server. Classic whois clients, such as the rfc1036/whois, handle this by scanning each line of text for known referral markers using functions like find_referral_server_iana. This approach works, but it is fragile because every registry formats output differently. The brittleness of parsing free-form text was a key driver to replace it with a modern protocol that uses structured data like JSON.

RDAP: The Modern Successor

The push to replace it began back in 2013, when an ICANN Expert Working Group recommended that the WHOIS protocol should be tossed out. They proposed a system that would keep information secret from most users, disclosing data only for specific “permissible purposes” like legal actions or trademark enforcement. Notably, journalism was excluded from this list, despite WHOIS historically being a key tool for investigative reporting.

After years of debate and voting, the transition became official. On January 28, 2025, WHOIS was officially sunset for generic Top-Level Domains (gTLDs). Registries are no longer required to support it, and the industry has shifted its focus to RDAP (Registration Data Access Protocol).

RDAP performs the same function as WHOIS but it uses HTTPS and returns JSON instead of plain text.

Feature WHOIS RDAP
Transport TCP Port 43 HTTP/HTTPS (Port 80/443)
Format Unstructured Plain Text Structured JSON (machine-readable)
Security None Standard Web Security (TLS, Auth)
Discovery Brittle, text-based referrals Standardized discovery.

You can try it with curl:

curl -L https://rdap.verisign.com/com/v1/domain/google.com
Output (click to expand) 
{
  "objectClassName": "domain",
  "handle": "2138514_DOMAIN_COM-VRSN",
  "ldhName": "GOOGLE.COM",
  "links": [
    {
      "value": "https://rdap.verisign.com/com/v1/domain/GOOGLE.COM",
      "rel": "self",
      "href": "https://rdap.verisign.com/com/v1/domain/GOOGLE.COM",
      "type": "application/rdap+json"
    },
    {
      "value": "https://rdap.markmonitor.com/rdap/domain/GOOGLE.COM",
      "rel": "related",
      "href": "https://rdap.markmonitor.com/rdap/domain/GOOGLE.COM",
      "type": "application/rdap+json"
    }
  ],
  "status": [
    "client delete prohibited",
    "client transfer prohibited",
    "client update prohibited",
    "server delete prohibited",
    "server transfer prohibited",
    "server update prohibited"
  ],
  "entities": [
    {
      "objectClassName": "entity",
      "handle": "292",
      "roles": [
        "registrar"
      ],
      "links": [
        {
          "href": "http://www.markmonitor.com",
          "type": "text/html",
          "value": "https://rdap.markmonitor.com/rdap/",
          "rel": "about"
        }
      ],
      "publicIds": [
        {
          "type": "IANA Registrar ID",
          "identifier": "292"
        }
      ],
      "vcardArray": [
        "vcard",
        [
          [
            "version",
            {},
            "text",
            "4.0"
          ],
          [
            "fn",
            {},
            "text",
            "MarkMonitor Inc."
          ]
        ]
      ],
      "entities": [
        {
          "objectClassName": "entity",
          "roles": [
            "abuse"
          ],
          "vcardArray": [
            "vcard",
            [
              [
                "version",
                {},
                "text",
                "4.0"
              ],
              [
                "fn",
                {},
                "text",
                ""
              ],
              [
                "tel",
                {
                  "type": "voice"
                },
                "uri",
                "tel:+1.2086851750"
              ],
              [
                "email",
                {},
                "text",
                "abusecomplaints@markmonitor.com"
              ]
            ]
          ]
        }
      ]
    }
  ],
  "events": [
    {
      "eventAction": "registration",
      "eventDate": "1997-09-15T04:00:00Z"
    },
    {
      "eventAction": "expiration",
      "eventDate": "2028-09-14T04:00:00Z"
    },
    {
      "eventAction": "last changed",
      "eventDate": "2019-09-09T15:39:04Z"
    },
    {
      "eventAction": "last update of RDAP database",
      "eventDate": "2025-12-16T20:15:07Z"
    }
  ],
  "secureDNS": {
    "delegationSigned": false
  },
  "nameservers": [
    {
      "objectClassName": "nameserver",
      "ldhName": "NS1.GOOGLE.COM"
    },
    {
      "objectClassName": "nameserver",
      "ldhName": "NS2.GOOGLE.COM"
    },
    {
      "objectClassName": "nameserver",
      "ldhName": "NS3.GOOGLE.COM"
    },
    {
      "objectClassName": "nameserver",
      "ldhName": "NS4.GOOGLE.COM"
    }
  ],
  "rdapConformance": [
    "rdap_level_0",
    "icann_rdap_technical_implementation_guide_1",
    "icann_rdap_response_profile_1"
  ],
  "notices": [
    {
      "title": "Terms of Service",
      "description": [
        "Service subject to Terms of Use."
      ],
      "links": [
        {
          "href": "https://www.verisign.com/domain-names/registration-data-access-protocol/terms-service/index.xhtml",
          "type": "text/html",
          "value": "https://rdap.verisign.com/com/v1/domain/google.com",
          "rel": "terms-of-service"
        }
      ]
    },
    {
      "title": "Status Codes",
      "description": [
        "For more information on domain status codes, please visit https://icann.org/epp"
      ],
      "links": [
        {
          "href": "https://icann.org/epp",
          "type": "text/html"
        }
      ]
    },
    {
      "title": "RDDS Inaccuracy Complaint Form",
      "description": [
        "URL of the ICANN RDDS Inaccuracy Complaint Form: https://icann.org/wicf"
      ],
      "links": [
        {
          "href": "https://icann.org/wicf",
          "type": "text/html",
          "value": "https://rdap.verisign.com/com/v1/domain/google.com",
          "rel": "help"
        }
      ]
    }
  ]
}

The response is a structured JSON object that is far easier to parse than the free-form text of WHOIS.

Why .dev domains don’t work

If you try to run a legacy WHOIS lookup against a modern TLD like .dev, you will likely hit a dead end. Google, along with many newer registries, has effectively deprecated port 43. They are not required to support the old text-based protocol, so they don’t.

Instead, querying a .dev domain via the command line often returns a generic placeholder from IANA. It tells you who manages the .dev registry, but it won’t tell you anything about the specific domain you asked for (like kmcd.dev).

$ whois kmcd.dev
# % IANA WHOIS server
# % for more information on IANA, visit http://www.iana.org
# % This query returned 1 object

# domain:       DEV
# organisation: Charleston Road Registry Inc.
# ...
# remarks:      Registration information: https://www.registry.google
# source:       IANA

To get the actual data, you are supposed to use rdap. As shown below, the rdap command retrieves the full registration details you would expect:

rdap kmcd.dev
Output (click to expand) 
Domain:
  Domain Name: kmcd.dev
  Handle: E04E36511-DEV
  Status: client transfer prohibited
  Conformance: rdap_level_0
  Conformance: icann_rdap_response_profile_1
  Conformance: icann_rdap_technical_implementation_guide_1
  Notice:
    Title: RDAP Terms of Service
    Description: By querying our Domain Database as part of the RDAP pilot program (RDAP Domain Database), you are agreeing to comply with these terms and acknowledging that your information will be used in accordance with Charleston Road Registry's Privacy Policy (https://www.registry.google/about/privacy.html), so please read the terms and Privacy Policy carefully.
    Description: Any information provided is 'as is' without any guarantee of accuracy.
    Description: Please do not misuse the RDAP Domain Database. It is intended solely for query-based access on an experimental basis and should not be used for or relied upon for any other purpose.
    Description: Don't use the RDAP Domain Database to allow, enable, or otherwise support the transmission of mass unsolicited, commercial advertising or solicitations.
    Description: Don't access our RDAP Domain Database through the use of high volume, automated electronic processes that send queries or data to the systems of Charleston Road Registry or any ICANN-accredited registrar.
    Description: You may only use the information contained in the RDAP Domain Database for lawful purposes.
    Description: Do not compile, repackage, disseminate, or otherwise use the information contained in the RDAP Domain Database in its entirety, or in any substantial portion, without our prior written permission.
    Description: We may retain certain details about queries to our RDAP Domain Database for the purposes of detecting and preventing misuse.
    Description: We reserve the right to restrict or deny your access to the RDAP Domain Database if we suspect that you have failed to comply with these terms.
    Description: We reserve the right to modify or discontinue our participation in the RDAP pilot program and suspend or terminate access to the RDAP Domain Database at any time and for any reason in our sole discretion.
    Description: Reminder that underlying Registrant data may be requested via ICANN's RDRS service (https://rdrs.icann.org/).
    Description: We reserve the right to modify this agreement at any time.
    Link: https://pubapi.registry.google/rdap/help/tos
    Link: https://www.registry.google/policies/rdap-terms/
  Notice:
    Title: Status Codes
    Description: For more information on domain status codes, please visit https://icann.org/epp
    Link: https://icann.org/epp
  Notice:
    Title: RDDS Inaccuracy Complaint Form
    Description: URL of the ICANN RDDS Inaccuracy Complaint Form: https://icann.org/wicf
    Link: https://icann.org/wicf
  Link: https://pubapi.registry.google/rdap/domain/kmcd.dev
  Link: https://rdap.cloudflare.com/rdap/v1/domain/kmcd.dev
  Event:
    Action: registration
    Actor: cloudflare
    Date: 2024-05-18T19:33:01.182Z
  Event:
    Action: expiration
    Date: 2034-05-18T19:33:01.182Z
  Event:
    Action: last update of RDAP database
    Date: 2025-12-16T22:15:17.386Z
  Event:
    Action: last changed
    Date: 2025-10-08T17:07:04.569Z
  Secure DNS:
    Zone Signed: true
    Delegation Signed: true
    DSData:
      Key Tag: 2371
      Algorithm: 13
      Digest: 321F88BA26AB76AE885C06671798645ADF4D06448F52D67D627641FA28392AF7
      DigestType: 2
  Entity:
    Handle: 1910
    Public ID:
      Type: IANA Registrar ID
      Identifier: 1910
    Remark:
      Title: Incomplete Data
      Type: object truncated due to unexplainable reasons
      Description: Summary data only. For complete data, send a specific query for the object.
    Link: https://pubapi.registry.google/rdap/entity/1910
    Link: None
    Role: registrar
    vCard version: 4.0
    vCard fn: CloudFlare, Inc.
    Entity:
      Status: active
      Role: abuse
      vCard version: 4.0
      vCard fn: Abuse Team
      vCard tel: tel:+1.4153197517
      vCard email: registrar-abuse@cloudflare.com
  Nameserver:
    Nameserver: chuck.ns.cloudflare.com
    Handle: 13F5B5F1_HOW-GOOGLE
    Remark:
      Title: Incomplete Data
      Type: object truncated due to unexplainable reasons
      Description: Summary data only. For complete data, send a specific query for the object.
    Link: https://pubapi.registry.google/rdap/nameserver/chuck.ns.cloudflare.com
  Nameserver:
    Nameserver: oaklyn.ns.cloudflare.com
    Handle: 40ADE79A0-GOOGLE
    Remark:
      Title: Incomplete Data
      Type: object truncated due to unexplainable reasons
      Description: Summary data only. For complete data, send a specific query for the object.
    Link: https://pubapi.registry.google/rdap/nameserver/oaklyn.ns.cloudflare.com

Even though rdap works, it isn’t installed by default on most systems. Many people are probably going to forget to install rdap or will just default to whois out of habit. I figured that one way to get the old whois command working again is by making a proxy that speaks the WHOIS protocol to the client and will fetch the data using RDAP.

Building a WHOIS-to-RDAP Proxy

Now, I will walk you through a WHOIS server that acts as a proxy to other RDAP servers. It will listen for WHOIS queries on port 43 and when it receives a query, it will make an HTTPS request to the appropriate RDAP server, parse the JSON response, format the important details into a human-readable text format, and send that text back to the original WHOIS client. Simple, no?

D2 Diagram

This approach makes RDAP-only domains accessible to legacy tools that only speak the classic WHOIS protocol. Although, let’s be honest, you should probably just use the existing rdap command for anything serious. This is just a toy. But it was fun to make.

Here is the implementation of our new WHOIS->RDAP proxy server:

whois-server-proxy/main.go (click to expand)View on GitHub 
package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net"
	"net/http"
	"strings"
	"text/template"
	"time"

	_ "embed"
)

//go:embed rdap.template
var rdapTemplateContent string

// tldRdapServers provides a direct mapping for common TLDs to their RDAP servers.
var tldRdapServers = map[string]string{
	"com": "https://rdap.verisign.com/com/v1/domain/",
	"net": "https://rdap.verisign.com/net/v1/domain/",
	"org": "https://rdap.publicinterestregistry.org/rdap/domain/",
	"dev": "https://pubapi.registry.google/rdap/domain/",
}

// --- RDAP Data Structures ---

type RDAPLink struct {
	Rel  string `json:"rel"`
	Href string `json:"href"`
	Type string `json:"type"`
}

type RDAPEvent struct {
	Action string    `json:"eventAction"`
	Actor  string    `json:"eventActor"`
	Date   time.Time `json:"eventDate"`
}

type RDAPNameserver struct {
	LDHName string `json:"ldhName"`
	Handle  string `json:"handle"`
	Remarks []struct {
		Title       string   `json:"title"`
		Type        string   `json:"type"`
		Description []string `json:"description"`
	} `json:"remarks"`
	Links []RDAPLink `json:"links"`
}

type RDAPEntity struct {
	VCardArray VCard        `json:"vcardArray"`
	Roles      []string     `json:"roles"`
	Entities   []RDAPEntity `json:"entities"`
	Handle     string       `json:"handle"`
	PublicIDs  []struct {
		Type       string `json:"type"`
		Identifier string `json:"identifier"`
	} `json:"publicIds"`
	Remarks []struct {
		Title       string   `json:"title"`
		Type        string   `json:"type"`
		Description []string `json:"description"`
	} `json:"remarks"`
	Links  []RDAPLink `json:"links"`
	Status []string   `json:"status"`
}

type VCard []interface{}

func (vc VCard) GetField(key string) string {
	if len(vc) < 2 {
		return ""
	}
	properties, ok := vc[1].([]interface{})
	if !ok {
		return ""
	}
	for _, prop := range properties {
		propertyArray, ok := prop.([]interface{})
		if !ok || len(propertyArray) < 4 {
			continue
		}
		propKey, ok := propertyArray[0].(string)
		if !ok || propKey != key {
			continue
		}
		val, ok := propertyArray[3].(string)
		if ok {
			return val
		}
	}
	return ""
}

type SecureDNSData struct {
	Algorithm  int    `json:"algorithm"`
	Digest     string `json:"digest"`
	DigestType int    `json:"digestType"`
	KeyTag     int    `json:"keyTag"`
}

type RDAPResponse struct {
	LDHName     string           `json:"ldhName"`
	Handle      string           `json:"handle"`
	Nameservers []RDAPNameserver `json:"nameservers"`
	Events      []RDAPEvent      `json:"events"`
	Entities    []RDAPEntity     `json:"entities"`
	Links       []RDAPLink       `json:"links"`
	Status      []string         `json:"status"`
	Conformance []string         `json:"rdapConformance"`
	Notices     []struct {
		Title       string     `json:"title"`
		Description []string   `json:"description"`
		Links       []RDAPLink `json:"links"`
	} `json:"notices"`
	SecureDNS struct {
		ZoneSigned       bool            `json:"zoneSigned"`
		DelegationSigned bool            `json:"delegationSigned"`
		DSData           []SecureDNSData `json:"dsData"`
	} `json:"secureDNS"`
	Remarks []struct {
		Title       string   `json:"title"`
		Description []string `json:"description"`
	} `json:"remarks"`
}

func (r *RDAPResponse) getReferralURL() string {
	for _, link := range r.Links {
		if link.Rel == "related" && link.Type == "application/rdap+json" {
			return link.Href
		}
	}
	return ""
}

// Server holds the dependencies for the WHOIS server.
type Server struct {
	rdapTemplate *template.Template
}

// NewServer creates a new server and parses the RDAP template.
func NewServer() (*Server, error) {
	tmpl, err := template.New("rdap").Parse(rdapTemplateContent)
	if err != nil {
		return nil, fmt.Errorf("failed to parse template: %w", err)
	}
	return &Server{rdapTemplate: tmpl}, nil
}

// queryRDAP performs the RDAP lookup, following one level of referral if necessary.
func queryRDAP(domain string) (*RDAPResponse, error) {
	parts := strings.Split(domain, ".")
	var url string
	if len(parts) > 1 {
		tld := parts[len(parts)-1]
		if baseUrl, ok := tldRdapServers[tld]; ok {
			url = baseUrl + domain
			log.Printf("Found direct RDAP server for TLD .%s, using: %s", tld, url)
		}
	}

	if url == "" {
		url = "https://rdap.iana.org/domain/" + domain
		log.Printf("Using IANA bootstrap RDAP endpoint: %s", url)
	}

	// Perform the initial query
	resp, err := http.Get(url)
	if err != nil {
		return nil, fmt.Errorf("RDAP request failed: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		body, _ := io.ReadAll(resp.Body)
		return nil, fmt.Errorf("RDAP server returned status %d: %s", resp.StatusCode, string(body))
	}

	var initialResponse RDAPResponse
	if err := json.NewDecoder(resp.Body).Decode(&initialResponse); err != nil {
		return nil, fmt.Errorf("failed to decode initial RDAP JSON: %w", err)
	}
	resp.Body.Close() // Close the body of the first response now.

	// Check for a referral and follow it
	if referralURL := initialResponse.getReferralURL(); referralURL != "" {
		log.Printf("Following RDAP referral to: %s", referralURL)
		resp, err = http.Get(referralURL)
		if err != nil {
			return nil, fmt.Errorf("RDAP referral request failed: %w", err)
		}
		defer resp.Body.Close()

		if resp.StatusCode != http.StatusOK {
			body, _ := io.ReadAll(resp.Body)
			return nil, fmt.Errorf("RDAP referral server returned status %d: %s", resp.StatusCode, string(body))
		}

		var finalResponse RDAPResponse
		if err := json.NewDecoder(resp.Body).Decode(&finalResponse); err != nil {
			return nil, fmt.Errorf("failed to decode final RDAP JSON: %w", err)
		}
		return &finalResponse, nil
	}

	return &initialResponse, nil
}

func (s *Server) handleConnection(conn net.Conn) {
	log.Printf("new connection from %s", conn.RemoteAddr())
	defer log.Printf("connection to %s closed", conn.RemoteAddr())
	defer conn.Close()

	scanner := bufio.NewScanner(conn)
	var clientQuery string

	if scanner.Scan() {
		clientQuery = strings.TrimSpace(scanner.Text())
	}

	if err := scanner.Err(); err != nil {
		log.Printf("Error reading from client: %v", err)
		return
	}

	if clientQuery == "" {
		_, _ = fmt.Fprint(conn, "Please provide a domain name.\r\n")
		return
	}

	log.Printf("Received query for: %q", clientQuery)

	rdapData, err := queryRDAP(clientQuery)
	if err != nil {
		log.Printf("RDAP query for %q failed: %v", clientQuery, err)
		_, _ = fmt.Fprintf(conn, "Error performing RDAP lookup: %v\r\n", err)
		return
	}

	var responseBuilder strings.Builder
	if err := s.rdapTemplate.Execute(&responseBuilder, rdapData); err != nil {
		log.Printf("Internal error: failed to execute template: %v", err)
		_, _ = fmt.Fprint(conn, "Internal server error.\r\n")
		return
	}

	_, err = fmt.Fprint(conn, responseBuilder.String())
	if err != nil {
		log.Printf("Error writing response: %v", err)
	}
}

func main() {
	port := ":43"

	server, err := NewServer()
	if err != nil {
		log.Fatalf("Error creating server: %v", err)
	}

	listener, err := net.Listen("tcp", port)
	if err != nil {
		log.Fatalf("Error listening on port %s: %v", port, err)
	}
	defer listener.Close()
	log.Printf("RDAP Proxy WHOIS server listening on %s", port)

	for {
		conn, err := listener.Accept()
		if err != nil {
			log.Printf("Error accepting connection: %v", err)
			continue
		}
		go server.handleConnection(conn)
	}
}

The output is formatted using a Go template to create a classic WHOIS-style report from the RDAP JSON data:

whois-server-proxy/rdap.template (click to expand)View on GitHub 
Domain:
  Domain Name: {{.LDHName}}
  Handle: {{.Handle}}
  {{- range .Status}}
  Status: {{.}}
  {{- end}}
  {{- range .Conformance}}
  Conformance: {{.}}
  {{- end}}
  {{- range .Notices}}
  Notice:
    Title: {{.Title}}
    {{- range .Description}}
    Description: {{.}}
    {{- end}}
	{{- range .Links}}
    Link: {{.Href}}
    {{- end}}
  {{- end}}
  {{- range .Links}}
  Link: {{.Href}}
  {{- end}}
  {{- range .Events}}
  Event:
    Action: {{.Action}}
	{{- if .Actor}}
    Actor: {{.Actor}}
	{{- end}}
    Date: {{.Date.Format "2006-01-02T15:04:05.000Z"}}
  {{- end}}
  {{- with .SecureDNS}}
  Secure DNS:
    Zone Signed: {{.ZoneSigned}}
    Delegation Signed: {{.DelegationSigned}}
	{{- range .DSData}}
    DSData:
      Key Tag: {{.KeyTag}}
      Algorithm: {{.Algorithm}}
      Digest: {{.Digest}}
      DigestType: {{.DigestType}}
	{{- end}}
  {{- end}}
  {{- range .Entities}}
  Entity:
    Handle: {{.Handle}}
	{{- range .PublicIDs}}
    Public ID:
      Type: {{.Type}}
      Identifier: {{.Identifier}}
	{{- end}}
	{{- range .Remarks}}
    Remark:
      Title: {{.Title}}
      Type: {{.Type}}
	  {{- range .Description}}
      Description: {{.}}
	  {{- end}}
	{{- end}}
	{{- range .Links}}
    Link: {{.Href}}
	{{- end}}
    Role: {{range $i, $role := .Roles}}{{if $i}}, {{end}}{{$role}}{{end}}
	{{- if .VCardArray.GetField "fn"}}
    vCard version: 4.0
    vCard fn: {{.VCardArray.GetField "fn"}}
	{{- end}}
	{{- range .Entities}}
    Entity:
	  {{- range .Status}}
      Status: {{.}}
	  {{- end}}
      Role: {{range $i, $role := .Roles}}{{if $i}}, {{end}}{{$role}}{{end}}
      vCard version: 4.0
      vCard fn: {{.VCardArray.GetField "fn"}}
      vCard tel: {{.VCardArray.GetField "tel"}}
      vCard email: {{.VCardArray.GetField "email"}}
	{{- end}}
  {{- end}}
  {{- range .Nameservers}}
  Nameserver:
    Nameserver: {{.LDHName}}
    Handle: {{.Handle}}
	{{- range .Remarks}}
    Remark:
      Title: {{.Title}}
      Type: {{.Type}}
	  {{- range .Description}}
      Description: {{.}}
	  {{- end}}
	{{- end}}
	{{- range .Links}}
    Link: {{.Href}}
	{{- end}}
  {{- end}}

With the proxy running, we can query it for kmcd.dev and get a complete and useful response using a standard whois client:

# Run the proxy in one terminal
go run ./whois-server-proxy

# Query it from another
whois -h localhost kmcd.dev
Output (click to expand) 
Domain:
  Domain Name: kmcd.dev
  Handle: E04E36511-DEV
  Status: client transfer prohibited
  Conformance: rdap_level_0
  Conformance: icann_rdap_technical_implementation_guide_0
  Conformance: icann_rdap_response_profile_0
  Notice:
    Title: Cloudflare Registrar
    Description: Cloudflare provides more than 13 million domains with the tools to give their global users a faster, more secure, and more reliable internet experience.
    Description: Register your domain name at https://www.cloudflare.com/registrar/
    Link: https://www.cloudflare.com/registrar/
  Notice:
    Title: Terms of Use
    Description: Data in the Cloudflare Registrar WHOIS database is provided to you by Cloudflare under the terms and conditions at https://www.cloudflare.com/domain-registration-agreement/.
    Description: By submitting this query, you agree to abide by these terms.
    Link: https://www.cloudflare.com/domain-registration-agreement/
  Notice:
    Title: EPP Status Codes
    Description: For more information on domain status codes, please visit https://icann.org/epp.
    Link: https://icann.icann.org/epp
  Notice:
    Title: Whois Inaccuracy Complaint Form
    Description: URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf.
    Link: https://www.icann.org/wicf
  Event:
    Action: registration
    Date: 2024-05-18T19:33:01.000Z
  Event:
    Action: last changed
    Date: 2024-05-23T20:08:26.329Z
  Event:
    Action: expiration
    Date: 2034-05-18T19:33:01.000Z
  Event:
    Action: registrar expiration
    Date: 2034-05-18T19:33:01.000Z
  Secure DNS:
    Zone Signed: false
    Delegation Signed: true
    DSData:
      Key Tag: 2371
      Algorithm: 13
      Digest: 321F88BA26AB76AE885C06671798645ADF4D06448F52D67D627641FA28392AF7
      DigestType: 2
  Entity:
    Handle: 1910
    Public ID:
      Type: IANA Registrar ID
      Identifier: 1910
    Role: registrar
    vCard version: 4.0
    vCard fn: Cloudflare, Inc.
    Entity:
      Role: abuse
      vCard version: 4.0
      vCard fn: Cloudflare Registrar Abuse
      vCard tel: tel:+1.4153197517
      vCard email: registrar-abuse@cloudflare.com
  Entity:
    Handle:
    Remark:
      Title: DATA REDACTED
      Type: object redacted due to authorization
      Description: Some of the data in this object has been removed
    Role: registrant
    vCard version: 4.0
    vCard fn: DATA REDACTED
  Entity:
    Handle:
    Remark:
      Title: DATA REDACTED
      Type: object redacted due to authorization
      Description: Some of the data in this object has been removed
    Role: administrative
    vCard version: 4.0
    vCard fn: DATA REDACTED
  Entity:
    Handle:
    Remark:
      Title: DATA REDACTED
      Type: object redacted due to authorization
      Description: Some of the data in this object has been removed
    Role: technical
    vCard version: 4.0
    vCard fn: DATA REDACTED
  Entity:
    Handle:
    Remark:
      Title: DATA REDACTED
      Type: object redacted due to authorization
      Description: Some of the data in this object has been removed
    Role: billing
    vCard version: 4.0
    vCard fn: DATA REDACTED
  Nameserver:
    Nameserver: chuck.ns.cloudflare.com
    Handle:
  Nameserver:
    Nameserver: oaklyn.ns.cloudflare.com
    Handle:

Closing Thoughts

WHOIS is simple and approachable, but it belongs to a smaller and more trusting Internet. It relies on unstructured text, inconsistent formatting, informal conventions, and an unencrypted transport. It is out-of-place in the modern Internet.

RDAP is the natural evolution of WHOIS. It fixes the exact problems that made WHOIS brittle: structure, discovery and security.

By wrapping the new standard in the old interface, we bridged the gap between the past and present. With the WHOIS-to-RDAP proxy, we get the structured power of RDAP without losing the muscle-memory and intuitive naming of the whois command.

This was a toy project made to learn about both WHOIS and RDAP, but this acts as a useful lens on how internet protocols evolve and illustrates many features of modern web APIs that we take for granted today.

References

]]>Months Considered Harmfulhttps://kmcd.dev/posts/months-considered-harmful/Thu, 01 Jan 2026 10:00:00 +0000https://kmcd.dev/posts/months-considered-harmful/ <p> <img hspace="5" src="https://kmcd.dev/posts/months-considered-harmful/cover.svg" /> </p> Can months be fixed? The calendar we all live by is a shambling mess. It has arbitrary month lengths, misnamed months, and rules so absurd we rely on nursery rhymes just to remember them. For something that structures our lives, it is clunky and illogical. This post is my attempt to design something that actually makes sense.

What’s wrong with our calendar?

Our current calendar is far from optimal. Months have varying numbers of days, which isn’t just annoying; it makes planning harder. For example, budgeting is made harder because month lengths don’t just vary in days, but in the number of pay periods or expense cycles they contain, leading to unpredictable cash flow. This may seem trivial, but it’s a strange flaw in a system we use every day.

Built through centuries of compromises and quick fixes, what began as lunar tracking has decayed into something that does neither job well. While not directly relevant to most modern daily planning, the conceptual failure to maintain a consistent natural rhythm highlights the calendar’s fundamental breakage from its origins. This breakdown stems from prioritizing the solar year for stable seasons and holidays, an aim that directly conflicts with the irregular lunar cycle.

The naming convention is another glaring bug. Consider September, October, November, and December. Their Latin roots (septem (7), octo (8), novem (9), and decem (10)) clearly point to their original positions. Yet, today they are our 9th, 10th, 11th, and 12th months. This historical mess is the result of early Roman calendar reforms, where the start of the year was shifted, leaving several months with names that no longer match their position.

So what are we left with?

  • Months are inconsistent in length.
  • The naming of months is inconsistent with their meaning.
  • We have vestigial concepts that have lost their ties to their original purpose.

Part I: 13-Month Year

My first pass at a fix was all about logic. I designed a clean, predictable system based on a perfect grid.

The Spec: A Perfect Grid

The idea was a perpetual calendar:

  • 13 Months, each 28 days long.
  • 4 Perfect Weeks per month, starting on a Monday and ending on a Sunday. Always.
  • 1 “Year Day” at the end. 13 × 28 = 364. The 365th day is a special holiday outside the weekly cycle, with a second “Leap Day” when we need it.

To see the difference, compare a perfect 13th-month calendar month with our current January 2026.

“Perfect” Month

Mon Tue Wed Thu Fri Sat Sun
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28

January 2026

Mon Tue Wed Thu Fri Sat Sun
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

The appeal is its deterministic nature. The 10th of any month is always a Wednesday. Annoying date arithmetic for things like budgets, sprints, and payroll just becomes simple. It’s a clean API for time.

Where My Dream Falls Apart

Of course, for all its elegance on paper, this system slams into a wall of bitter reality.

The entire global economy runs on four quarters. 13 is a prime number, meaning you can’t divide it cleanly into equal quarters. While individuals don’t live their lives in quarters, corporations do. This would necessitate creating awkward 3-3-3-4 month quarters for financial reporting, which ultimately reintroduces the very inconsistency we were attempting to eliminate. Economic reality kills calendar idealism.

Implementing such a calendar change would be a monumental undertaking, akin to Y2K on a global scale but vastly more complex due to deeply embedded legal, civil, and international coordination challenges. Every piece of software, every legal contract, and every database on Earth would require extensive rewriting. The associated cost would be unfathomable and would ultimately outweigh any benefits that we’d gain.

Introducing a “timeless” day (or two on leap years) into the calendar disrupts the continuous seven-day weekly cycle. It is… awkward. How would you write the date for this day? This disruption would break critical real-world systems like payroll cycles, recurring shifts, and automated cron jobs that rely on an unbroken weekly sequence. Some people still have to work, so there would be a weird exception for planning for this day. Timekeeping software would need to specifically handle this.

After considering the downsides, I have decided that my dream will continue to stay a dream. Months cannot be fixed… at least not without creating more problems. Who really needs months anyway?

Part II: Deprecating the Month

Why do we use months at all? What purpose do they serve in our daily lives? They’re a mashup of bad Latin, Roman vanity, and arbitrary divisions that have no real connection to the 7-day cycle that actually governs our schedules. The bug isn’t the implementation of the month. The month itself is the bug.

So, let’s deprecate the abstraction.

Long live the week

The proposal is to use the week as the main unit of time beyond the day. A year is just 52 weeks. And get this: this isn’t some new idea I dreamed up. It’s already part of the ISO 8601 international standard.

The formats are clean and unambiguous:

  • Week: YYYY-Www. Example: 2025-W41 represents the 41st week of 2025.
  • Week with weekday: YYYY-Www-D. Example: 2025-W41-2 represents the 2nd day (Tuesday) of the 41st week of 2025. (Monday is 1, Sunday is 7).

The system is built and standardized; we just don’t use it as our default. Here’s how the first few weeks of 2026 look using the ISO 8601 week-date system, with the Gregorian date (day/month) for comparison.

ISO Week Mon Tue Wed Thu Fri Sat Sun
2026-W01 29/12 30/12 31/12 01/01 02/01 03/01 04/01
2026-W02 05/01 06/01 07/01 08/01 09/01 10/01 11/01
2026-W03 12/01 13/01 14/01 15/01 16/01 17/01 18/01
2026-W04 19/01 20/01 21/01 22/01 23/01 24/01 25/01
2026-W05 26/01 27/01 28/01 29/01 30/01 31/01 01/02

Note that week 1 of 2026 starts in December of 2025. This might seem confusing, but it’s a deliberate design choice. Rule of thumb: The first week of the year is the week that has the year’s first Thursday in it. This is the standard definition and explains why the week can straddle years—it ensures the majority of the week belongs to the new year. It’s a trade-off: the Gregorian calendar breaks weekday consistency at the year’s boundary, while the week-date system prioritizes the weekly cycle over the year boundary. In practice, you often stop caring which ‘year’ a Monday belongs to, because your planning horizon is almost always weeks ahead, not abstract year boundaries.

Addressing the Concerns

This idea has its own set of hurdles, but they feel more like matters of habit than hard blockers.

The fiscal quarter problem is solved. Remember how the 13-month calendar failed because 13 is a prime number? The week-based system fixes this beautifully. A 52-week year divides perfectly into four 13-week quarters. This is already a common practice in many industries (often called a 4-4-5 calendar) because it makes financial reporting and year-over-year comparisons much more consistent. Far from being a problem, this is a major advantage.

Months are poor seasonal anchors anyway. “December” is winter here in Copenhagen but high summer in Cape Town. Tying months to seasons is just a Northern Hemisphere bias. We can create new narratives. The argument that we need months for “chapter breaks” in our year assumes we’re incapable of making new patterns. We could easily create new milestones around 10-week blocks or the four 13-week quarters. The narrative doesn’t disappear; it just gets refactored.

As I’ve learned living in Denmark, using week numbers for planning can be completely normal. It coexists with the standard calendar. I’ve had people invite me to events in “week 45.” The first time, I had to look it up. The fifth time, I also had to look it up. But I assume that with enough usage I will build an intuition.

The Migration Path: An Evolution

The end result of this thought exercise is a new year years goal for myself: I’m going to lean into using week numbers more in my own life. This isn’t a radical proposition; it’s about consciously adopting a better system that already exists in parallel. It’s an evolutionary change, and here’s how you can start:

Add Week Numbers to Your Calendar. Most digital calendars support this as a display option. Making the week number visible is the first step to making it intuitive. Try it for three months.

Start Using Weeks in Planning. In your next meeting invite, add the week number. “Let’s sync up in Week 45.” Will people look at you weird when you ask that? Probably. But notice how often months stop mattering in your planning.

By pushing for a better standard that already exists, maybe you can start the slow process of deprecating using months in favor of weeks in your own life. Happy week 1, everyone!

References & Further Reading

For those interested in a deeper dive into the topics discussed, here are some of the resources used in researching this article:

]]>Encryption vs. Compressionhttps://kmcd.dev/posts/encryption-vs-compression/Mon, 22 Dec 2025 10:00:00 +0000https://kmcd.dev/posts/encryption-vs-compression/ <p> <img hspace="5" src="https://kmcd.dev/posts/encryption-vs-compression/cover.svg" /> </p> Is there a correct order when encrypting and compressing data? Compression shrinks data, encryption obfuscates it. Should you compress or encrypt first? If you get this wrong and you will waste CPU, storage, and bandwidth. Pick wisely.

Always compress first, then encrypt.

In backups, data pipelines, secure transport, or any place where performance matters, picking the wrong order will waste a lot of resources needlessly. I often ask this exact question during interviews. The question reveals more than just memorization of trivia; it shows whether they truly understand how compression and encryption work.

Why the order matters

The reason “encrypt then compress” is a terrible idea is due to the opposing goals of the two operations.

Compression tools are pattern-finders. A tool like gzip hunts for repeated sequences in your file, like a recurring log message or timestamp, and replaces those long strings with tiny references. It works best with order and predictability.

Encryption tools are pattern-destroyers. The goal of a strong encryption algorithm (like AES) is to scramble your orderly data into high-entropy, random-looking noise. A perfectly encrypted file should have no predictable patterns.

So, if you encrypt first, you’re feeding random noise to a compression tool that thrives on patterns. It finds none, gives up, and the file barely shrinks (or might even get slightly larger).

But if you compress first, the algorithm gets to work on the raw, patterned data. It shrinks it way down. Then you encrypt the much smaller result.

Let’s try it out

Instead of just talking about it, let’s run a small experiment to demonstrate what happens when you get the order wrong. We’ll run both pipelines side by side so you can see how the output sizes change. We can prove this in about thirty seconds with standard command-line tools you likely already have installed. We’ll use gzip for compression and gpg for encryption.

First, we need some data to play with. Log files are perfect candidates since they’re so repetitive. Instead of using a real system log, we’ll generate our own super-compressible file.

# Create a dummy log file that's roughly 6MB.
yes "this is a line in our log file" | head -n 200000 > log.txt
# (`yes` just prints the same line endlessly — we use `head` to cap it.)

# Check its size
ls -lh log.txt
# Output: -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 17:13 log.txt

With log.txt ready, it’s time to experiment.

The Wrong Way: Encrypt → Compress

Here, we’ll encrypt the file and then try to compress the result. But there’s a catch: modern gpg is smart and compresses data by default. We have to tell it not to with the --compress-algo none flag to truly see the wrong way in action.

What you run

# 1. Encrypt the log file (with GPG's internal compression DISABLED)
gpg \
  --batch --yes \
  --pinentry-mode loopback \
  --passphrase "test" \
  --compress-algo none \
  --symmetric \
  --output encrypted_first.gpg \
  log.txt

# 2. Now, try to compress the encrypted file
gzip -c encrypted_first.gpg > encrypted_then_compressed.gz

What you see

ls -lh log.txt encrypted_first.gpg encrypted_then_compressed.gz
# Output:
# -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 17:13 log.txt
# -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 18:14 encrypted_first.gpg
# -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 18:14 encrypted_then_compressed.gz

ls -l log.txt encrypted_first.gpg encrypted_then_compressed.gz
# Output:
# -rw-r--r--@ 1 kevin  staff  6200000 Dec 12 17:13 log.txt
# -rw-r--r--@ 1 kevin  staff  6200084 Dec 12 18:14 encrypted_first.gpg
# -rw-r--r--@ 1 kevin  staff  6202017 Dec 12 18:14 encrypted_then_compressed.gz

The file size didn’t reduce with this setup; it actually increased by 2 kilobytes.

Encryption produces high-entropy data, so compression has nothing to work with.

Why it matters The encrypted_first.gpg file is the same size as the original, and encrypted_then_compressed.gz is… also the same size. The compression tool looked at the encrypted gibberish, found no patterns, and gave up. A complete waste of effort

The Right Way: Compress → Encrypt

Now, let’s do it correctly. We’ll compress the file first, then encrypt the much smaller result.

What you run

# 1. Compress the raw log file
gzip -c log.txt > compressed_first.gz

# 2. Encrypt the compressed file
gpg \
  --batch --yes \
  --pinentry-mode loopback \
  --passphrase "test" \
  --symmetric \
  --output compressed_then_encrypted.gpg \
  compressed_first.gz

What you see

ls -lh log.txt compressed_first.gz compressed_then_encrypted.gpg
# Output:
# -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 17:13 log.txt
# -rw-r--r--@ 1 kevin  staff    15K Dec 12 18:15 compressed_first.gz
# -rw-r--r--@ 1 kevin  staff    15K Dec 12 18:15 compressed_then_encrypted.gpg

Why it matters Now that’s what we expected to see. The compressed_then_encrypted.gpg file is tiny (15 KB!). We have the same data, but it’s small and secret.

Compress raw data first to capture patterns, and encryption preserves the smaller size.

Final Results

Let’s put the two final files side-by-side for a dramatic comparison.

ls -lh encrypted_then_compressed.gz compressed_then_encrypted.gpg
# Output:
# -rw-r--r--@ 1 kevin  staff    15K Dec 12 18:15 compressed_then_encrypted.gpg
# -rw-r--r--@ 1 kevin  staff   5.9M Dec 12 18:14 encrypted_then_compressed.gz
D2 Diagram

Getting the order right saved us over 99% of the disk space.

A quick note on the results: The dummy log.txt file we generated is extremely repetitive, which makes for a dramatic demonstration. Real-world data won’t always compress this well, but the principle holds: you’ll always get better results by compressing the patterned data before you encrypt it.

Security Implications

Hopefully, you’re convinced that “Compress then Encrypt” is the rule to use in all cases, right? Almost. There’s one major exception where this exact sequence can create a subtle but powerful security hole.

When an attacker can influence data that gets compressed and then encrypted, they can create a “compression oracle” to slowly guess secret information. Compression leaks information because matching patterns in the data cause it to shrink more, so the final size becomes a side-channel that reveals details about the original content. This forms the basis of two infamous attacks: CRIME and BREACH.

CRIME: The Attack That Killed TLS Compression

The CRIME attack went after compression that was happening directly at the TLS layer, the ‘S’ in HTTPS.

  1. The Attack: CRIME allowed an attacker to steal secret authentication cookies. They’d trick a victim’s browser into sending requests containing their cookie, and they would inject their own guesses for the cookie’s value into the request. By watching the size of the final encrypted response from the server, they could see when their guess matched a character in the real cookie, because the compressed size would shrink just a tiny bit. They could repeat this to uncover the secret, one character at a time.

  2. The Impact: The consequences were immediate and severe. Everyone realized that letting the TLS protocol itself handle compression was a footgun waiting to go off. It was quickly disabled in all major browsers and is now forbidden in the TLS 1.3 specification. It’s gone for good.

BREACH: The Attack That Won’t Go Away

BREACH is CRIME’s younger, more persistent sibling. It targets compression at the HTTP level (the gzip or brotli your web server uses), not the TLS layer.

  1. The Attack: The idea is the same. An attacker tricks a browser into sending repeated requests, but this time they are trying to guess a secret embedded in the HTML of the response body, like a hidden CSRF token. If the response contains both the secret and some input from the attacker (like a reflected search term), they can once again watch the compressed size to build a compression oracle.

    BREACH only works when the response includes both attacker-controlled input and a secret in the same compressed payload. If your application never mixes these, the attack isn’t possible.

  2. The Impact: We can’t just turn off HTTP compression, as the performance hit to the web would be catastrophic. So, stopping BREACH falls on application developers, with a few key strategies:

    Using SameSite=Strict on your session cookies is a strong defense as it prevents the browser from sending them on any cross-site requests, blocking the main attack vector. While many applications must use SameSite=Lax for practical navigation, it also significantly reduces the risk by withholding cookies on cross-origin subrequests, preventing the most common BREACH scenarios.

    The best defense is to never mix secrets and user-controllable data in the same compressed response.

    Obscure the true length by adding a random number of bytes to the response, making the compression ratio useless to an attacker.

    Use rate-limiting to slow down attackers so the thousands of requests needed for the attack become impractical.

    At this point, you might be thinking: “Wait, HTTP/2 and HTTP/3 use header compression (HPACK and QPACK). If my secrets are in headers, doesn’t that bring back the BREACH risk?” That is a concern here. Headers like Authorization, Cookie, Set-Cookie, or X-Auth-Token often contain high-value secrets and are prime targets for a compression oracle. To handle this, HTTP/2 and HTTP/3 protocols include a specific “sensitive” flag. When a header is marked this way, the compressor is told to treat the value as a literal rather than adding it to the dynamic compression table. This prevents the value from being used as a reference point for future matches and prevents the side-channel without forcing us to disable compression for the entire connection.

Final thoughts

What started as an interesting little interview question lead us to discussing real security concerns with mixing compression and encryption.

In most cases, the rule is unequivocal: compress first, then encrypt. Whether you are archiving logs or moving backups, compressing patterned data before it becomes random noise saves significant time and money.

The only real exception is when you are handling interactive traffic. If an attacker can control part of a message that contains a secret, the compressed size actually leaks information. For static archives, this isn’t an issue. But for live web protocols, you either have to use modern safeguards like sensitive header flags or disable compression entirely for those specific payloads.

Further Reading

The Specifications

The Security Vulnerabilities

The Theory

]]>On Creating My Own Cover Arthttps://kmcd.dev/posts/cover-art/Mon, 15 Dec 2025 10:00:00 +0000https://kmcd.dev/posts/cover-art/ <p> <img hspace="5" src="https://kmcd.dev/posts/cover-art/cover.svg" /> </p> Why I felt guilty using AI art, and chose to embrace a more personal, code-driven creative process. I felt guilty using AI-generated art for my blog.

When AI tools first appeared, they felt like magic. A simple text prompt could conjure a unique, often beautiful image to accompany my writing. Sometimes it would even have the correct number of fingers. Amazing! It felt like a powerful new way to communicate, especially for me, who is not artistically inclined. Over time, however, the process felt hollow. Beyond the feeling of it being a shortcut, I found that AI art often fell short of capturing the vision I had in mind for my cover art. It felt like the cover art was the result of me complaining at a machine until it produced something that was kind-of what I wanted and had the fewest obvious flaws.

I was doing this as a shortcut but it began to feel like a chore. So starting with my previous post, I’m changing how the covers are generated for my posts.

I’ve always loved generative art: art created from code, algorithms, and a touch of randomness. It just felt right to spend time generating my own cover art with code, rather than asking an AI. For a blog about making things with code, it just makes sense that code would generate the cover art as well.

So, I built a small Go program that generates a raster image, and then feeds it into a Go tool/library called primitive which reinterprets it into a stylized vector piece.

The Go script generates random shapes of different sizes, pulling from a few color palettes that I created. It also randomly draws lines connecting the shapes together, because I like the “graph diagram” feel that it gives. For example, here’s the little chunk of code that draws a star. For drawing the initial shapes, I decided to use a library called Go Graphics (github.com/fogleman/gg):

func drawStar(dc *gg.Context, points, centerX, centerY, outerRadius float64) {
	innerRadius := outerRadius * 0.4
	dc.NewSubPath()
	for i := 0.0; i < points*2; i++ {
		r := outerRadius
		if int(i)%2 != 0 {
			r = innerRadius
		}
		angle := (math.Pi*2/(points*2))*i - math.Pi/2
		x := centerX + r*math.Cos(angle)
		y := centerY + r*math.Sin(angle)
		dc.LineTo(x, y)
	}
	dc.ClosePath()
}

The output is a chaotic but structured smattering of shapes and lines.

To transform this into something with more visual interest, I turned to primitive. This library attempts to recreate the input image using a limited number of geometric shapes. It smooths out the chaos, turning the harsh lines into a stylized, abstract vector piece.

Before:

Before

After:

After
Before and after applying the primitive transformation.

It’s also worth noting that the resulting images from primitive are SVGs. Because they’re vector-based, they are often much smaller than their raster counterparts (PNG, JPG) and scale perfectly to any size. This means they look crisp on everything from a mobile phone to a 4K monitor without increasing the file size.

This new process feels right. I’m pretty happy with how this has turned out. It’s more personal, and there’s a fun element of surprise in seeing what the code comes up with each time. If I ever get bored of the results, I have the ability to change the code or some of the settings.

Here’s a gallery of the art this process has helped create. Note that not all of these images look amazing. When creating new articles, I plan on generating images with this script until I find one that I like well enough to use for the cover.

Click here for the full source code.

]]>Traceroute Tool from Scratch in Gohttps://kmcd.dev/posts/traceroute/Tue, 09 Dec 2025 10:00:00 +0000https://kmcd.dev/posts/traceroute/ <p> <img hspace="5" src="https://kmcd.dev/posts/traceroute/cover.svg" /> </p> Map your route through the Internet Traceroute is a network diagnostic utility that maps the path that packets take across an IP network. It provides a list of the intermediate routers a packet traverses on its way to a final destination, along with the time taken for each “hop.” This information is crucial for diagnosing network latency and identifying points of failure. Personally, I think it is super cool that there’s a way to figure out the route that your packets are taking.

In this article, I’ll dig into how traceroute works and show you how to build a simple version from scratch using Go.

ICMP: The Internet’s Control Protocol

Before diving into traceroute, we need to understand the protocol that powers it: ICMP (Internet Control Message Protocol). ICMP is a network-layer protocol used by network devices to send error messages and operational information. It’s not for exchanging data, but for diagnostics, control, and error reporting. In fact, many routers and devices will heavily throttle ICMP traffic to protect their CPU usage, since generating these error messages is computationally expensive compared to simply forwarding packets.

The Classic Use Case: ping

The most common and well-known use of ICMP is the ping utility. ping is used to test the reachability of a host on an IP network. It works by sending an ICMP Echo Request to the target host. If the host is reachable, it responds with an ICMP Echo Reply.

Here is what a typical ping to google.com looks like:

$ ping google.com
PING google.com (192.0.0.88): 56 data bytes
64 bytes from 192.0.0.88: icmp_seq=0 ttl=64 time=88.471 ms
64 bytes from 192.0.0.88: icmp_seq=1 ttl=64 time=88.708 ms
64 bytes from 192.0.0.88: icmp_seq=2 ttl=64 time=88.535 ms
64 bytes from 192.0.0.88: icmp_seq=3 ttl=64 time=88.579 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 88.471/88.573/88.708/0.087 ms

This tells us that google.com is reachable and provides the round-trip time for each packet.

From ping to traceroute

While ping tells you if you can reach a destination, traceroute tells you how you get there. It builds on the same ICMP foundation but uses a different ICMP message in a clever way. Instead of just checking for the final “Echo Reply,” it intentionally triggers ICMP Time Exceeded messages from intermediate routers to map out the path. This is all done by manipulating the Time-To-Live (TTL) field in the IP packet header.

IP and ICMP Packet Formats

To fully understand how traceroute works, we should understand the basic structure of IP and ICMP packets.

IPv4 Packet Format

All network traffic on the internet is encapsulated within IP packets. The IP header contains crucial information for routing, including source and destination IP addresses, and, most importantly for traceroute, the Time-To-Live (TTL) field.

packet
0-3: "Version"
4-7: "IHL"
8-15: "Type of Service (TOS)"
16-31: "Total Length"
32-47: "Identification"
48-50: "Flags"
51-63: "Fragment Offset"
64-71: "TTL"
72-79: "Protocol"
80-95: "Header Checksum"
96-127: "Source IP Address"
128-159: "Destination IP Address"
160-191: "Options & Padding"
192-223: "Data (Variable Length)"

Mostly, I want you to note the Time-To-Live (TTL) field, which is central to how traceroute operates. We’ll touch more on TTLs a little bit later. But first, let’s look at the structure of ICMP packets.

ICMP Packet Format

ICMP messages, including the Echo Request/Reply used by ping and the Time Exceeded messages used by traceroute, are themselves encapsulated within an IP packet’s payload. The ICMP header specifies the message type (e.g., Echo Request, Echo Reply, Time Exceeded) and a code, along with a checksum for integrity. For Echo messages, it also includes an identifier and sequence number.

packet
0-7: "Type (e.g., 8=Request, 0=Reply)"
8-15: "Code (e.g., 0)"
16-31: "Checksum"
32-47: "Identifier (BE) / (Varies by Type)"
48-63: "Sequence Number (BE) / (Varies by Type)"
64-95: "Data Payload (variable length)"

Crucially, since ICMP messages are encapsulated within IP packets, it is the enclosing IP packet that carries the TTL field and other IP header information, not the ICMP message itself. The ICMP header defines the specific control message, while the IP header handles its transport.

How does traceroute work?

Unlike a direct message, data sent over the Internet hops through a chain of routers to reach its destination. Traceroute reveals this path by exploiting a fail-safe mechanism called Time-To-Live (TTL).

Unlike what the name suggests, TTL does not represent actual time. It represents the number of hops before routers are instructed to give up on the packet. It is kind-of like a self-destruct counter. Every time a packet passes through a router (a “hop”), this counter decreases by 1. When it hits 0, the router discards the packet and sends an error message back to you.

Traceroute uses this mechanism to map the network:

  1. Hop 1: It sends a packet with a TTL of 1. The very first router receives it, decreases the TTL to 0, drops the packet, and replies with “Time Exceeded.” We have now identified the first router.
  2. Hop 2: It sends a new packet with a TTL of 2. The first router passes it along (TTL becomes 1). The second router receives it, decreases TTL to 0, drops it, and replies “Time Exceeded.” We have identified the second router.
  3. Repeat: It continues increasing the TTL by 1 until the packet finally reaches the destination server and receives a standard reply.

Probe Methods: ICMP, UDP, and TCP

While our examples use ICMP Echo Requests, it’s important to understand that the TTL expiration mechanism is protocol-agnostic. The TTL field is in the IP header, which encapsulates TCP, UDP, and ICMP packets alike. Because of this, traceroute utilities can use different protocols for their “probes.”

  • UDP Probes: This is the traditional method used by Linux and macOS. traceroute sends UDP datagrams to an invalid, high-numbered port at the destination. For intermediate hops, the process is identical: routers send ICMP Time Exceeded messages when the TTL expires. When the UDP packet finally reaches the destination, the host’s OS sees that no service is listening on that port and sends back an ICMP Port Unreachable error. This “unreachable” message ironically signals a successful arrival at the destination.

  • TCP Probes: Some traceroute versions can send TCP SYN packets, similar to how a TCP connection is initiated. Intermediate routers still respond with ICMP Time Exceeded. If the packet reaches the destination, the host will respond with either a TCP SYN/ACK (if the port is open) or a TCP RST (if the port is closed). Either response confirms that the destination has been reached.

  • ICMP Probes: This method leverages ICMP Echo Requests, similar to the ubiquitous ping command found on Linux (and other OSes). While Linux traceroute traditionally uses UDP, Windows’ tracert utility uses ICMP Echo Requests by default, as does our Go example. It listens for ICMP Time Exceeded from routers and a final ICMP Echo Reply from the destination.

Regardless of the protocol used for the probe, the fundamental mechanic remains the same: an ICMP Time Exceeded message is always generated by routers when the TTL reaches zero.

What layer does traceroute operate on?

Traceroute operates at Layer 3 (the Network Layer) of the OSI model. This is because its core components are all Layer 3 protocols and mechanisms:

  • IP (Internet Protocol): Traceroute is fundamentally about sending and routing IP packets.
  • ICMP (Internet Control Message Protocol): The diagnostic messages, such as Time Exceeded and Echo Reply, are ICMP messages. ICMP is a network-layer protocol that operates alongside IP.
  • TTL (Time To Live): The TTL field is part of the IP header itself. Routers, which are Layer 3 devices, are responsible for decrementing this value.

Here’s a diagram of the OSI model, showing where traceroute fits in:

Layer. Note
7: Application
6: Presentation
5: Session
4: Transport
3: Network (IP) ← You are here.
2: Data Link
1: Physical

While the core TTL mechanism and ICMP Time Exceeded messages clearly place traceroute’s fundamental operation at Layer 3, it’s important to note that the probe packets themselves can originate from Layer 4 protocols. As discussed in the “Probe Methods” section, traceroute often utilizes UDP or TCP packets. In these cases, the probe originates at Layer 4 (Transport Layer) but relies on the Layer 3 IP header’s TTL for its diagnostic function. This makes traceroute a tool that crosses over a couple OSI layer boundaries, sending probes with Layer 3 or Layer 4 and utilizing ICMP’s time exceeded errors from Layer 3.

TTL Exhaustion

When a packet’s TTL is set to 1, its journey is cut short at the first router. The router discards the packet and sends an ICMP Time Exceeded message back. This is the fundamental mechanism for discovering a hop.

D2 Diagram

Symmetric Packet Flow to Destination

In an ideal scenario, a packet reaches the final server, and the ICMP Echo Reply travels back to the client along the exact same path.

D2 Diagram

Asymmetric Routing (The Reality of the Internet)

It is important to note that the return path is not guaranteed to be the same as the request path. The Internet is a mesh of dynamic networks, and routing decisions are made independently in each direction. The diagram below shows a more realistic scenario where the return path uses a different set of routers.

D2 Diagram

For dealing with the complexities of inconsistent and asymmetric routes, tools like mtr (My Traceroute) continuously repeat traceroutes, providing real-time statistics and a clearer picture of network performance over time.

Building a Traceroute in Go

We can build a functional traceroute tool in Go using the golang.org/x/net/icmp package, which provides access to the low-level networking primitives required. The process involves setting up a “listener” to receive ICMP messages, then looping to send probes with an incrementally higher TTL.

Step 1: Crafting and Sending the ICMP Probe

First, we need to construct an ICMP Echo Request. The ID field should be unique to our process to distinguish replies intended for us from other network traffic, and we use the Seq (sequence) number to track which hop this probe corresponds to. After crafting the message, we set the TTL for this specific packet and send it on its way.

// Set the TTL for our outgoing packet
if err := c.IPv4PacketConn().SetTTL(ttl); err != nil {
    log.Fatalf("SetTTL failed: %s", err)
}

// Create an ICMP Echo Message
m := icmp.Message{
    Type: ipv4.ICMPTypeEcho, Code: 0,
    Body: &icmp.Echo{
        ID:   os.Getpid() & 0xffff, // Use process ID to uniquely identify this traceroute
        Seq:  ttl,                  // Use the TTL as the sequence number
        Data: []byte("HELLO-TRACEROUTE"),
    },
}
b, err := m.Marshal(nil)
if err != nil {
    log.Fatalf("Marshal failed: %s", err)
}

// Send the ICMP packet to the destination
if _, err := c.WriteTo(b, dstAddr); err != nil {
    log.Fatalf("WriteTo failed: %s", err)
}

Step 2: Receiving and Validating the Reply

After sending the probe, we must wait for a response. We set a read deadline on our connection to act as a timeout. If we receive a packet, we parse it to see what kind of ICMP message it is.

A crucial step is to validate that the reply is for a packet we sent. Raw sockets receive all ICMP traffic on the machine, so we could accidentally process a reply meant for another program (like a ping running in another terminal). We do this by checking the ID in the ICMP message, which we set to our process ID (PID). For an Echo Reply, this is straightforward. For a Time Exceeded message, the original packet’s ID is nested inside the message body, requiring a bit of parsing to extract.

The following snippet shows the conceptual logic for reading the reply. The full validation is in the complete implementation below.

// Wait for a reply
reply := make([]byte, 1500) // 1500 is the standard MTU (Maximum Transmission Unit) for Ethernet
// ... set read deadline ...
n, peer, err := c.ReadFrom(reply)
if err != nil {
    // A timeout means we didn't hear back, continue to next TTL
    fmt.Printf("%d\t*\n", ttl)
    continue
}

// Parse the reply message
rm, err := icmp.ParseMessage(1, reply[:n]) // 1 for ICMPv4
// ... handle parse error ...

Step 3: The Main Loop and Interpreting Results

Finally, we wrap the sending and receiving logic in a for loop that increments the TTL from 1 to a maximum value. Before sending a probe, we record the current time, and after receiving a reply, we calculate the duration. This gives us the round-trip time (RTT), which is the total time it takes for the packet to travel to the intermediate router and for the ICMP error message to travel back.

Inside the loop, after receiving and validating a reply, a switch statement checks the type of the ICMP message to determine if we’ve found an intermediate router (Time Exceeded) or reached the final destination (Echo Reply).

// Loop from TTL 1 up to a max number of hops (e.g., 64)
for ttl := 1; ttl <= 64; ttl++ {
    // ... (Code from Step 1: Craft and Send Probe) ...

    // ... (Code from Step 2: Receive and Validate Reply) ...
    
    elapsed := time.Since(start)

    // Check the type of the received ICMP message
    switch rm.Type {
    case ipv4.ICMPTypeEchoReply:
        // We've reached the final destination
        fmt.Printf("%d\t%v\t%v\n", ttl, peer, elapsed)
        return // We are done
    case ipv4.ICMPTypeTimeExceeded:
        // This is a reply from an intermediate router
        fmt.Printf("%d\t%v\t%v\n", ttl, peer, elapsed)
    default:
        // Other ICMP type
        fmt.Printf("%d\t%v\t%v (type %d)\n", ttl, peer, elapsed, rm.Type)
    }
}

Full Go Implementation

The following code combines all the steps into a complete, runnable traceroute program, including the crucial validation logic to ensure we only process replies to the probes we sent.

package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"time"

	"golang.org/x/net/icmp"
	"golang.org/x/net/ipv4"
)

func main() {
	if len(os.Args) < 2 {
		fmt.Println("Usage: go run traceroute.go <destination>")
		os.Exit(1)
	}
	destination := os.Args[1]
	dstAddr, err := net.ResolveIPAddr("ip4", destination)
	if err != nil {
		log.Fatalf("Could not resolve destination: %s", err)
	}

	// Listen for ICMP packets
	c, err := icmp.ListenPacket("ip4:icmp", "0.0.0.0")
	if err != nil {
		log.Fatalf("ListenPacket failed: %s", err)
	}
	defer c.Close()

	fmt.Printf("Traceroute to %s (%s)\n", destination, dstAddr)

	for ttl := 1; ttl <= 64; ttl++ {
		start := time.Now()

		// Set TTL
		if err := c.IPv4PacketConn().SetTTL(ttl); err != nil {
			log.Fatalf("SetTTL failed: %s", err)
		}

		// Create ICMP Echo Message
		m := icmp.Message{
			Type: ipv4.ICMPTypeEcho, Code: 0,
			Body: &icmp.Echo{
				ID:   os.Getpid() & 0xffff,
				Seq:  ttl,
				Data: []byte("HELLO-TRACEROUTE"),
			},
		}
		b, err := m.Marshal(nil)
		if err != nil {
			log.Fatalf("Marshal failed: %s", err)
		}

		// Send
		if _, err := c.WriteTo(b, dstAddr); err != nil {
			log.Fatalf("WriteTo failed: %s", err)
		}

		// Wait for reply
		reply := make([]byte, 1500) // 1500 is the standard MTU (Maximum Transmission Unit) for Ethernet
		if err := c.SetReadDeadline(time.Now().Add(3 * time.Second)); err != nil {
			log.Fatalf("SetReadDeadline failed: %s", err)
		}
		n, peer, err := c.ReadFrom(reply)
		if err != nil {
			fmt.Printf("%d\t*\t*\t*\n", ttl) // Timeout
			continue
		}
		elapsed := time.Since(start)

		// Parse the reply message
		rm, err := icmp.ParseMessage(1, reply[:n]) // 1 for ICMPv4
		if err != nil {
			log.Printf("Error parsing ICMP message: %s", err)
			continue
		}

		// Check if the reply is for our process and probe
		switch rm.Type {
		case ipv4.ICMPTypeEchoReply:
			if rm.Body.(*icmp.Echo).ID != os.Getpid()&0xffff {
				continue // Not our packet
			}
			fmt.Printf("%d\t%v\t%v\n", ttl, peer, elapsed)
			fmt.Println("Destination reached.")
			return // We are done
		case ipv4.ICMPTypeTimeExceeded:
			// For simplicity, we assume any TimeExceeded message is for our probe.
			// A robust implementation would parse the body of the message
			// to verify the ID of the original packet.
			fmt.Printf("%d\t%v\t%v\n", ttl, peer, elapsed)
		default:
			// This could be Destination Unreachable or other types. We'll ignore them for this simple tool.
			fmt.Printf("%d\t%v\t%v (type %d)\n", ttl, peer, elapsed, rm.Type)
		}
	}
}

This script combines the previous steps into a fully functioning (although not fully featured) traceroute utility. Now it’s time to use it.

Running the Code

Save the code as traceroute.go and execute it with a destination as the argument. Since it requires a raw socket, it must be run with sudo.

sudo go run traceroute.go google.com

Here are some of my runs (from a VPN):

$ sudo go run traceroute.go kmcd.dev
Traceroute to kmcd.dev (172.64.80.1)
1       10.5.0.1        88.094958ms
2       87.249.138.252  88.137959ms
3       79.127.195.58   88.360125ms
4       45.134.215.17   89.163958ms
5       162.158.61.119  90.93775ms
6       172.64.80.1     88.631208ms
Destination reached.

This is CloudFlare’s DNS service.

$ sudo go run traceroute.go 1.1.1.1
Traceroute to 1.1.1.1 (1.1.1.1)
1       10.5.0.1        109.916792ms
2       5.104.76.1      109.879916ms
3       78.152.53.114   110.688917ms
4       207.162.204.138 110.696958ms
5       1.1.1.1 109.922875ms
Destination reached.

Microsoft.com seems to block ICMP traffic, so we see a lot of timeouts after a certain point:

$ sudo go run traceroute.go microsoft.com
Traceroute to microsoft.com (13.107.246.35)
1       10.5.0.1        110.333709ms
2       5.104.76.1      114.67925ms
3       78.152.53.114   110.453333ms
4       141.136.111.141 121.163833ms
5       62.115.44.160   123.153875ms
6       *       *       *
7       62.115.140.169  125.062291ms
8       *       *       *
9       51.10.27.124    121.206958ms
10      104.44.231.105  121.929917ms
11      *       *       *
12      *       *       *
13      *       *       *
14      *       *       *
15      *       *       *
...

You will often see rows of asterisks like this (* * *). This usually doesn’t mean the router is down; it means the router is configured to drop ICMP packets with an expired TTL without sending a response. This is often done for security reasons or to de-prioritize ICMP traffic to protect the router’s CPU.

A Note on sudo and Raw Sockets

You might ask if sudo is strictly necessary. This is a common point of confusion for developers new to network programming in Go, as tools like the standard traceroute on macOS and Linux can often run without sudo by sending UDP packets. While sending UDP packets is unprivileged, listening for the returning ICMP Time Exceeded errors is a privileged operation that often requires elevated permissions or specific system configurations (like modifying net.ipv4.ping_group_range).

Our code simplifies this by using icmp.ListenPacket("ip4:icmp", ...), which creates a powerful raw ICMP socket. This approach requires sudo because listening directly to the entire ICMP protocol is a privileged operation, but it saves us from writing more complex, OS-specific code. This feels more appropriate for this tutorial.

Conclusion

Traceroute is a powerful diagnostic tool that reveals the path our data takes across the internet. By cleverly using the Time-To-Live (TTL) field in IP packets, it turns an error-reporting mechanism into a tool for discovery. We’ve walked through how this works, from sending probes with an ever-increasing TTL to interpreting the ICMP Time Exceeded messages that let us map the network hop-by-hop.

Using this knowledge, we built a functional traceroute tool from scratch in Go. Our implementation uses ICMP echo requests, just like the ping utility, and listens for replies from intermediate routers as well as the final destination. While we focused on ICMP, we also touched on alternative probing methods using UDP and TCP.

Building a tool like this from the ground up really demystifies the magic behind everyday network diagnostics and gives a deeper appreciation for the protocols that run the internet. The journey of a single packet is a fascinating one, and with a little bit of Go, we’ve built a window to watch it.

Next Steps

While this implementation demonstrates the core logic of traceroute, a production-grade tool would include several enhancements:

  • Reverse DNS Lookup: The tool currently only shows IP addresses. A reverse-DNS lookup could be added to resolve these IPs into more human-readable hostnames.
  • Support for UDP and TCP Probes: Extend the tool to allow different probe methods, such as UDP and TCP, for increased flexibility and compatibility with various network environments and firewalls.
  • ASN Lookup: By querying the Autonomous System Number (ASN) of each IP, the tool could identify the specific ISP or organization that owns a router. This enables the visualization of the AS-Path to show how traffic hands off between different entities, for example: Comcast → Tata Communications → Google. Visualizing these organizational jumps is often more insightful than viewing a raw list of IP addresses.
  • Geo-location: Beyond simple IP and ASN lookups, more advanced tools can be used to geo-locate routers. By examining router hostnames (which often contain location codes), querying WHOIS databases for IP ranges, and consulting resources like PeeringDB or Internet BGP tables, it’s possible to infer the physical location and network ownership of each hop, providing richer diagnostic information.
  • Concurrency: To speed up the process, multiple probes could be sent concurrently using goroutines rather than sequentially.
  • Multiple Probes: Production tools often send multiple probes per hop and display statistics like average latency and packet loss. Multiple probes would reveal that there may be multiple paths that you are taking, as internet routing can be very dynamic.

These features are excellent next steps for expanding this simple tool into a more powerful network diagnostic utility, but this is left as an exercise for the audience. And until then, just use mtr, which has most of these features!

References

]]>My Favorite Interview Questionhttps://kmcd.dev/posts/favorite-interview-question/Mon, 22 Sep 2025 10:00:00 +0000https://kmcd.dev/posts/favorite-interview-question/ <p> <img hspace="5" src="https://kmcd.dev/posts/favorite-interview-question/cover_hu_db4c0ea7c272c8cc.webp" /> </p> Why I love asking candidates to explain how the internet works. Interviews are a strange dance. As an interviewer, you’re trying to get a signal on a candidate’s skills, experience, and personality in a very short amount of time. As a candidate, you’re trying to showcase your best self while under pressure. It’s a tough situation for everyone involved.

Over the years, I’ve asked a lot of interview questions. Some have been great, some have been duds. But I always come back to my absolute favorite question:

“How does the internet work?”

I know what you’re thinking. That’s a huge, open-ended question! And you’re right, it is. That’s exactly why I love it. You also might be thinking that this is a super common question. I agree! But unlike most questions, this one is still extremely useful despite its popularity.

Why this question is so powerful

This question is a fantastic tool for a few reasons:

  • It’s a blank canvas. The candidate can start anywhere they want. Do they start with the physical layer? The application layer? The philosophy of interconnectedness? It gives them a chance to show me what they think is important.
  • It reveals their depth of knowledge. The question can be answered at many different levels of abstraction. A junior engineer might give a high-level overview, while a senior engineer might be able to dive into the nitty-gritty details of TCP handshakes and BGP routing.
  • It tests communication skills. Can the candidate take a complex topic and explain it in a clear, concise way? This is a crucial skill for any engineer, especially in a team environment.
  • It’s a conversation starter. The question often leads to a natural back-and-forth conversation. I can ask follow-up questions to probe deeper into areas where the candidate seems knowledgeable or to help them along if they’re struggling.

Sometimes, the sheer open-endedness of the question can be a bit overwhelming for a candidate. If they’re struggling to start or getting too philosophical, I’ll give them a more concrete prompt:

“What happens when you type google.com into your browser and press Enter?”

This usually gets the ball rolling. Here’s a breakdown of the kind of answer I’m looking for, with varying levels of detail depending on the candidate’s seniority.

What I expect from most candidates

For any software engineering role that involves web development, I expect the candidate to be able to walk me through the following steps:

  • DNS Lookup: The browser needs to translate the human-readable domain name google.com into a machine-readable IP address. This involves checking the browser’s cache, the OS’s cache, and then querying a DNS resolver, which has its own upstream sources and cache.
  • TCP Connection: Once the browser has the IP address, it establishes a TCP connection with the server. This involves establishing a connection using TCP’s three-way handshake (SYN, SYN-ACK, ACK).
  • HTTP Request/Response: The browser sends an HTTP GET request to the server. The server processes the request and sends back an HTTP response, which includes the HTML, CSS, and JavaScript files that make up the Google homepage.
  • Rendering: The browser parses the HTML and renders the page. It also executes the JavaScript, which might make additional requests to the server to fetch more data.

What I hope for from senior candidates

For a more senior role, especially at a company that deals with networking or infrastructure, I’m hoping for a deeper dive into some of the following topics:

  • ARP (Address Resolution Protocol): Before the browser can even send a packet to the router, it needs to know the router’s MAC address. This is where ARP comes in.
  • DHCP (Dynamic Host Configuration Protocol): - How did our computer get an IP address in the first place? A senior candidate might explain that the machine likely requested an IP address, a subnet mask, the default gateway’s IP, and the DNS server’s IP from a DHCP server on the local network.
  • NAT (Network Address Translation): - To conserve the limited global supply of IPv4 addresses, most home and office networks use a single public IP address for many devices. NAT is the mechanism that allows a router to translate between private, internal IP addresses and that single public one.
  • BGP (Border Gateway Protocol): How does my request find its way from my local network to Google’s servers, potentially halfway across the world? Mentioning BGP shows an understanding that the internet is a ’network of networks’ and that routing between these large autonomous systems is a complex, solved problem.
  • HTTP/2 and HTTP/3 (QUIC): Is the connection really just a single TCP connection? A senior candidate might discuss the limitations of TCP, like head-of-line blocking, and how newer protocols solve them. Pointing out that since the destination is Google, the connection is likely using HTTP/3 (which runs on QUIC over UDP) is a huge plus.
  • TCP Keep-Alives: How does the connection stay open for subsequent requests? What are the trade-offs of keep-alives?
  • Caching: Where does caching happen in this whole process? DNS caching, browser caching, CDN caching… there are many layers.
  • Pre-flight Requests (CORS): If the page makes requests to other domains, the browser might need to make a pre-flight OPTIONS request to check if the cross-origin request is allowed.
  • Packet Breakdown: What does an actual packet look like? What are the different headers (Ethernet, IP, TCP, HTTP)?

It’s not about getting it “right”

I want to be clear: I don’t expect any candidate to know everything about the internet. That’s impossible. What I’m looking for is a candidate who has a solid foundation, can reason about complex systems, and is curious to learn more. How does the candidate behave when I press into an area that they don’t know a lot about? Are they curious? Are they willing to say they don’t know? Do they make things up? All of that is very useful for evaluating skill and knowledge level.

Sometimes a candidate’s answer goes in a completely unexpected direction. I once had a candidate describe the hardware interrupts and OS context switches that happen just from typing ‘google.com’. Even though it was a tangent, it was a valuable signal that showed me how they reason about systems from the hardware up.

This question is a fantastic way to gauge all of those things. It’s a journey we go on together, and I often learn something new from the candidate’s answer.

What’s your go-to interview question? I’d love to hear it.

]]>From JSON to Protobufhttps://kmcd.dev/posts/from-json-to-protobuf/Tue, 02 Sep 2025 10:00:00 +0000https://kmcd.dev/posts/from-json-to-protobuf/ <p> <img hspace="5" src="https://kmcd.dev/posts/from-json-to-protobuf/cover_hu_40e13ae2e18fb743.webp" /> </p> A Tool to Ease Your Schema Journey When I first approached the possibility of using protobuf or gRPC, I was intimidated by a few things. First, it was the terrible tooling: Protoc was weird, the plugin system it used was odd, the source of binaries was from wildly different places depending on the platform, and the code it generated was insane. However, the buf CLI fixes most of these issues for me. The next challenge I distinctly remember was getting started with my own Protobuf definitions. The tutorials were perfectly fine, but once you are set off on your own it’s kind of hard to know what to do next. I ended up reading the spec, since it’s not all that long. But I do feel like others don’t learn well from that method. Others will learn from examples. And the best examples are those that build on a foundation of knowledge they already have.

This is the problem I built a tool to solve. JSON has become an inescapable reality for developers of all types. It is very hard to become a junior developer without seeing and using a decent amount of JSON. Therefore, I figured that JSON is a good starting point for explaining protobufs. They are similar in many ways; in fact, there is now a standard way to convert protobuf types into JSON. That said, you may know that protobuf has some differences. First, it has an encoding that is more efficient than JSON, in terms of space and often in terms of processing time. Second, it is a schema-driven format. Protobuf files can be used to generate types for code in most programming languages and can be used to automatically generate documentation and other kinds of artifacts. This is a powerful concept that enables the sharing of types to many different languages. This approach reduces errors, reduces the amount of duplicate code that you have to write and maintain for common types and it enables some more efficient storage and processing of data.

The tool I built is meant to be a bridge for those stuck in the world of schema-less JSON, guiding them into a world of type safety and all the other benefits that Protobufs provide. I introduce, json-to-proto.kmcd.dev. This website/tool will take example JSON documents and produces a protobuf file that can consume and emit the same JSON. The goal here is mostly for education; to translate something junior developers know (how to read JSON) to something that they might not know (how to make the equivalent protobuf file). I’m hoping that it will provide enough material to get developers started.

json-to-proto.kmcd.dev

json-to-proto.kmcd.dev

Give it a try at json-to-proto.kmcd.dev!
Use this as an educational tool. Understand your protobufs before you ship.

How it works

The magic of this tool lies in its straightforward approach to translating the flexible world of JSON into the structured world of Protobuf. It does this by applying a series of simple rules and heuristics to the input data. The core of this process is type mapping, where the tool identifies each JSON data type (a string, number, boolean, or object) and converts it to a suitable Protobuf counterpart.

JSON String → Protobuf string

All JSON strings are converted to the Protobuf string type.

{
  "username": "joe"
}

syntax = "proto3";

package my_package;

message UserProfile {
  string username = 1;
}

This is a straightforward mapping. The tool recognizes the string value "joe" and translates the field to the corresponding string type in Protobuf, a simple and direct conversion that works for text of all kinds.

JSON Number → Protobuf int64 or double

The converter maps integer numbers to int64 and floating-point numbers to double. The use of int64 and double is a safe choice to ensure both large numbers and decimal values are handled correctly.

{
  "userId": 12345,
  "gpa": 3.4
}

syntax = "proto3";

package my_package;

message UserProfile {
  int64 userId = 1;
  double gpa = 2;
}

JSON Boolean → Protobuf bool

JSON boolean values (true and false) are converted to the Protobuf bool type.

{
  "graduated": false,
  "enrolled": true
}

syntax = "proto3";

package my_package;

message UserProfile {
  bool graduated = 1;
  bool enrolled = 2;
}

Like strings, booleans have a direct Protobuf equivalent. The tool sees the true and false values and maps them directly to a bool type, making this a simple and lossless conversion.

JSON Object → Protobuf message

Any JSON object is mapped to a new, nested Protobuf message type. The name of the new message is derived from the object’s field name. I’ve been using objects for all of my examples since they are essentially required at the top level. However, there’s something interesting to note about the attribute names. If the JSON attributes don’t match in terms of capitalization and underscore usage that protojson uses, json-to-proto.kmcd.dev will add a json_name annotation to make it match up correctly.

{
  "user_id": 12345,
  "UserName": "joe",
  "ENROLLED": true
}

syntax = "proto3";

package my_package;

message UserProfile {
  int64 user_id = 1 [json_name = "user_id"];
  string user_name = 2 [json_name = "UserName"];
  bool enrolled = 3 [json_name = "ENROLLED"];
}

This is where the tool gets clever with its handling of naming conventions. Notice how UserName is converted to user_name in the Protobuf file but keeps its original name in the JSON with a json_name option. This ensures the generated Protobuf still works seamlessly with your existing JSON data, a powerful feature for maintaining compatibility.

JSON Array → Protobuf repeated

A JSON array is converted into a repeated field in Protobuf. The type of the elements within the repeated field is inferred from the data contained in the array.

{
  "courses": [
    {
      "courseId": "CS101",
      "courseName": "Intro to CS"
    },
    {
      "courseId": "MA203",
      "courseName": "Linear Algebra",
      "credits": null
    }
  ],
  "login_timestamps": [
    1679400000,
    1679486400
  ],
  "mixed_data": [
    1,
    "test",
    true,
    {
      "key": "value"
    },
    null
  ]
}

syntax = "proto3";

import "google/protobuf/struct.proto";

package my_package;

message Course {
  repeated string course_id = 1;
  repeated string course_name = 2;
  repeated google.protobuf.Value credits = 3;
}

message UserProfile {
  repeated Course courses = 1;
  repeated int64 login_timestamps = 2 [json_name = "login_timestamps"];
  repeated google.protobuf.Value mixed_data = 3 [json_name = "mixed_data"];
}

This example shows three of these cases:

  • courses has a repeated field of Course messages.
  • login_timestamps as a repeated field with int64 values.
  • mixed_data as a repeated field of google.protobuf.Value messages, because the JSON array has several different types.

Another edge-case is hit here as well. The Course message has a credits field of type google.protobuf.Value. Why? Because the input JSON only has that field as being null. So the tool doesn’t have enough information to know what type "credits" should be.

Here, the tool’s logic for handling arrays shines. A repeated field is Protobuf’s way of representing a list. The tool infers the types within each array: a custom Course message, a list of int64 values, and a catch-all google.protobuf.Value for the mixed_data array, which contains different data types. To get the most accurate protobuf schema, it’s best to provide a representative JSON example that includes all possible data types and structures.

When you shouldn’t use this tool

It’s crucial to acknowledge that a tool like this is not a perfect replacement for human judgment. It is a powerful starting point, but a developer must always review and refine the output to ensure it perfectly matches the application’s needs.

Type Inference

While the tool makes an intelligent guess about types, it can’t read your mind. For example, a JSON number might be inferred as an int64, but your application may only ever need an int32. A developer should always make the final decision based on their knowledge of the data.

map vs. message

The tool defaults to a message for JSON objects. A developer might need to manually change this to a map if the object’s keys are truly arbitrary (e.g., a dictionary of feature flags).

A Starting Point, Not a Destination

In a world where JSON’s flexibility often leads to schema-less chaos, Protobuf offers a path to efficiency, speed, and type safety. My tool is designed to serve as a powerful bridge, providing a smooth on-ramp for developers who are intimidated by the initial hurdle of defining a .proto schema from scratch.

I encourage you to give it a try. The tool is a powerful assistant that eliminates the tedious first steps, freeing you up to focus on what matters most: building robust, scalable applications with a solid schema as their foundation.

]]>Breaking gRPChttps://kmcd.dev/posts/breaking-grpc/Tue, 05 Aug 2025 10:00:00 +0000https://kmcd.dev/posts/breaking-grpc/ <p> <img hspace="5" src="https://kmcd.dev/posts/breaking-grpc/cover_hu_72d86e0dbbafcc2.webp" /> </p> How to avoid breaking gRPC clients. When we use gRPC, we often praise its efficiency and strong contracts defined by Protocol Buffers (.proto files). We know that gRPC uses protobuf’s binary format for fast, compact, and forward/backward-compatible communication. But what happens when you expose your gRPC service to clients who speak JSON, like a web frontend?

The encoding you use (binary protobuf or transcoded JSON) dramatically changes the rules of what constitutes a “safe” or “breaking” change to your API. A change that is perfectly harmless for a protobuf client can completely break a JSON client. Let’s dig into this more.

How Encodings Work

First, a quick refresher on how each format represents data. Consider this simple protobuf message:

syntax = "proto3";

package my_service.v1;

message User {
  // A unique identifier for the user.
  int64 user_id = 1;
  // The user's full name.
  string name = 2;
}

Protobuf: It’s All About the Numbers

On the wire, the binary protobuf encoding doesn’t care about the field names (user_id, name). It only cares about the field numbers (1, 2) and their wire types. A simplified view of the encoded data is a series of key-value pairs where the key is the field number. I dig into this further in my gRPC from Scratch series, where I discuss the binary protobuf encoding.

Because of this, you can rename a field in your .proto file, and as long as the field number and type remain the same, it’s a non-breaking change for protobuf clients.

JSON: It’s All About the Names

When a gRPC gateway or library transcodes this message to JSON, it produces a standard JSON object. JSON is also a perfectly valid encoding to use with gRPC. By default, it uses the protobuf field names (converted to lowerCamelCase) as the JSON keys:

{
  "userId": 12345,
  "name": "Alex"
}

Since JSON clients are coupled to these names, changing them will inevitably break the integration. JSON clients are coupled to field names, not field numbers. This fundamental difference is the source of many potential compatibility issues.

Analyzing API Changes: Breaking vs. Non-Breaking

Let’s look at common changes you might make to a .proto file and see their impact on each encoding.

Change Protobuf Impact JSON Impact Explanation
Renaming a field (name to full_name) ✅ Non-breaking 💥 Breaking Protobuf clients only see the field number (2), which hasn’t changed. JSON clients expect the key "name" but will now see "fullName".
Changing a field number (= 2 to = 3) 💥 Breaking ✅ Non-breaking This is a cardinal sin in the protobuf world. A client expecting field 2 will no longer find it. JSON clients, however, still see the key "name" and are unaffected.
Adding a new field (email = 3) ✅ Non-breaking ✅ Non-breaking Well-behaved clients in both formats are designed to ignore unknown fields, making this a safe operation.
Removing or deprecating a field ✅ Non-breaking ✅ Non-breaking Similar to adding a field, clients should handle missing fields gracefully. It’s best practice to deprecate a field before removing it.
Changing a compatible type (int32 to int64) ✅ Non-breaking ✅ Non-breaking These types have compatible wire formats in protobuf. For JSON, both are simply numbers, so there’s no issue.
Changing an incompatible type (int64 to string) 💥 Breaking 💥 Breaking The wire format for a number and a string are different, breaking protobuf clients. The data type in JSON also changes (e.g., 123 vs. "123"), which will break any client expecting a number.

The Solution: Decouple Names with json_name

So, how do you refactor your .proto field names without breaking your JSON clients? The protobuf specification provides a simple and elegant solution: the json_name field option.

This option lets you explicitly set the JSON key for a field, decoupling it from the .proto field name.

Let’s revise our User message. Suppose we want to rename name to full_name for clarity in our Go or Python code, but we can’t break existing JSON clients that rely on the "name" key.

syntax = "proto3";

package my_service.v1;

message User {
  int64 user_id = 1 [json_name = "userId"];

  // The field is now 'full_name' in code, but will still be 'name' in JSON.
  string full_name = 2 [json_name = "name"];
}

With json_name = "name", we’ve instructed the transcoder to do the following:

  1. For Protobuf: Continue using field number 2. The field name full_name is used by the code generator.
  2. For JSON: Always use the key "name" during serialization, regardless of what the .proto field is called.

Now, you are free to change the full_name field to something else (e.g., user_display_name) in the future, and your JSON contract remains stable.

Automating Your Safety Net with buf breaking

Remembering all these nuanced rules across different encodings is difficult and error-prone. This is where automated tooling becomes essential. The popular Buf toolchain includes a powerful command, buf breaking, designed specifically for this problem.

The buf breaking command compares your current .proto files against a previous state (like your main git branch) and reports any changes that would break your API consumers. Crucially, it understands that “breaking” means different things to different clients. You can configure it to check against multiple compatibility strategies.

In your buf.yaml configuration file, you can specify which rule sets to check against:

  • FILE: Checks for backward-incompatible changes at the .proto file level, like deleting a field or changing a field number. This protects your protobuf-based clients.
  • WIRE_JSON: Checks for backward-incompatible changes for the JSON wire format. This catches things like renaming a field without using json_name. This protects your JSON-based clients.
  • PACKAGE: Checks for source-code-level breaking changes in the generated stubs for languages like Go and Java. This protects the developers using your generated code.

A typical configuration for a service with both gRPC and JSON clients might look like this:

# buf.yaml
version: v2
breaking:
  use:
    - FILE
    - WIRE_JSON

By integrating buf breaking into your CI/CD pipeline, you can automatically prevent developers from merging changes that would break any of your consumers, whether they speak protobuf or JSON.

Conclusion

Evolving an API for both protobuf and JSON clients is a recipe for a very specific kind of headache, the kind that pages you at 3 AM. You’ve got protobuf, which only cares about numbers, and JSON, which only cares about names. A “safe” refactor for one is a production-breaking slap in the face for the other. This is where a schema-first approach, backed by powerful schema-aware tooling, isn’t just a good idea; it’s the only thing keeping you from questioning all your life choices.

Protobuf’s semantics, like the json_name option, give you a powerful escape hatch. It makes certain refactors, like renaming a field for internal clarity, trivial if you have the right tooling in place. You can change your code without your JSON clients ever knowing you touched a thing. This decoupling is a superpower, but only if you use it correctly.

And that’s the catch: don’t rely on developers’ goldfish sized memory or manual code reviews to enforce these complex, conflicting rules. That’s how you break production at 3 AM. Instead, let the robots do the heavy lifting. Integrating a tool like buf breaking into your CI pipeline is like having an unblinking, unforgiving guardian for your API. It understands the different breaking change rules for both protobuf and JSON and will stop a bad change before it ever gets merged. This is the real strength of a schema-first workflow: it makes complex refactors not just possible, but safe. You can merge with confidence and keep all your clients (binary or JSON) happy.

]]>Morse Codehttps://kmcd.dev/posts/morse/Tue, 22 Jul 2025 10:00:00 +0000https://kmcd.dev/posts/morse/ <p> <img hspace="5" src="https://kmcd.dev/posts/morse/cover_hu_724955e1dc30dcfb.webp" /> </p> Explore the history of Morse code, from its dits and dahs to the first transatlantic cable. Discover how this elegant system was the original binary and test your own skills with an interactive speed typer. I’m continuing my journey into discovering more about older technology by making silly games. This time, I made a website that allows you to test your speed skills in the most useful skill ever! Morse code! Now you can test your skills in writing and interpreting morse code! And it even gives you your (likely abysmal) words per minute (wpm). (Plus, it’s great practice for Keep Talking and Nobody Explodes)

Head over to https://morse.kmcd.dev and try the Morse Speed Typer today! And if you need to brush up on your skills, check out the learning page.

Screenshot of morse.kmcd.dev

What is Morse Code?

Morse code is a character encoding system used in telecommunication that represents letters, numbers, and punctuation marks as sequences of short and long signals. These signals, often called dots (or “dits”) and dashes (or “dahs”), can be transmitted in various ways, such as through sound, light flashes, or electrical pulses. A simple and universally recognized example is the distress signal SOS, which is represented as ... --- ....

Developed in the 1830s by Samuel Morse and Alfred Vail, it revolutionized long-distance communication. Before we had fiber optics and satellites, we had the simple, elegant language of dits and dahs clicking away across telegraph wires.

The Rhythm of the Code: Timing ⏱️

Morse code isn’t just about the dots and dashes; it’s about the silence in between. The timing is crucial for distinguishing letters and words. The entire system is based on the length of a single dot.

  • A dot (dit) is the basic time unit: 1 unit long.
  • A dash (dah) is three times longer than a dot: 3 units long.
  • The space between parts of the same letter (e.g., the gap between the . and - in ‘A’) is 1 unit long.
  • The space between letters in a word is 3 units long.
  • The space between words is 7 units long.

This precise rhythm is what allows a trained operator to “read” the code by ear.

The Morse Alphabet

Just like in modern data compression, Morse code was designed with efficiency in mind. The most frequently used letters in the English language (like E and T) are assigned the shortest Morse code sequences, while less common letters have longer, more complex patterns. This clever design allowed for faster and more efficient communication.

Character Morse Code
A.-
B-...
C-.-.
D-..
E.
F..-.
G--.
H....
I..
J.---
K-.-
L.-..
M--
Character Morse Code
N-.
O---
P.--.
Q--.-
R.-.
S...
T-
U..-
V...-
W.--
X-..-
Y-.--
Z--..
Number Morse Code
0-----
1.----
2..---
3...--
4....-
5.....
6-....
7--...
8---..
9----.

The Original Binary Code

Illustration of telegraph operators

Long before computers used 1s and 0s, Morse code was encoding information using just two states: a short signal and a long signal (or, more fundamentally, “signal on” and “signal off”). In this sense, Morse code is one of the earliest forms of a binary code.

Just as modern computers use a standardized system like ASCII or Unicode to map binary digits to characters, Morse code provides a map from dits and dahs to the letters of the alphabet. It’s a brilliant, early example of how complex information like human language can be broken down into simple, transmittable units.

Cool Facts from the World of Morse 🌍

Morse code has a rich history filled with interesting trivia and conventions.

  • SOS is not an acronym. The famous distress signal, ...---..., was chosen because its pattern is simple and unmistakable. The continuous sequence of three dots, three dashes, and three dots is easy to recognize even through heavy static or interference. It doesn’t stand for “Save Our Ship” or “Save Our Souls,” though those are memorable mnemonics!

    SOS

  • K (Invitation to Transmit). In Morse conversations, sending a single letter ‘K’ (-.-) is an invitation for the other person to start transmitting. It’s the equivalent of saying “over” or “your turn.”

    K

  • 73 and 88. The world of ham radio, which heavily uses Morse code, developed its own numeric shorthand. 73 means “Best regards,” and 88 means “Love and kisses,” typically used when signing off with a close friend or partner.

    73
    88

  • What hath God wrought! On May 24, 1844, this was the first official message sent by Samuel Morse on the telegraph line between Washington, D.C., and Baltimore. The phrase, from the Book of Numbers, was suggested by Annie Ellsworth, the daughter of a friend. Here’s the message in Morse:

    WHAT
    HATH
    GOD
    WROUGHT

Conquering the Ocean: The First Global Connection

It’s easy to take for granted that we can send a message across the world in an instant. But before the 1860s, the fastest way to get information across the Atlantic was on a ship. A message sent from London to New York would take at least 10 days to arrive, and a reply would take just as long. A simple business transaction or a piece of breaking news could take nearly a month to cross the ocean and return. The world was vast, and the oceans were a barrier to communication.

The transatlantic telegraph cable changed everything. After a failed attempt in 1858, the first lasting connection was established in 1866. Laying a single, insulated copper wire across more than 3,000 kilometers of the treacherous North Atlantic seabed was one of the greatest engineering feats of the 19th century.

Suddenly, the 10-day journey of a message became a matter of minutes. The world shrank in a way that was previously unimaginable. For the first time, continents could converse in near real-time. News, financial data, and personal messages that once traveled at the speed of a steamship now traveled at the speed of electricity. It was the birth of our global network.

That single wire, painstakingly laid over 150 years ago, was the ancestor of the incredible web of cables that powers our internet today. Instead of one copper wire carrying a few words per minute in Morse code, we now have hundreds of undersea fiber-optic cables carrying terabits of data every second. They are the backbone of our modern world, transmitting everything from video calls to this very article.

To see the stunning evolution from that first transatlantic wire to the dense global network of today, explore this interactive map of modern undersea fiber-optic cables. It’s the direct legacy of the dits and dahs that first conquered the ocean. If you’ve been following my side projects, you might have seen this link coming.

Map of the Internet

The Enduring Echo of Dits and Dahs

From revolutionizing global communication on telegraph wires to its enduring legacy in maritime and amateur radio, Morse code is far more than a historical artifact. It stands as a powerful testament to human ingenuity. It embodies the fundamental principle that any complex idea, from a simple greeting to a national headline, can be distilled into a simple, universal language.

So while it may no longer be the backbone of our global network, learning Morse code offers a unique connection to the history of technology. It’s a chance to appreciate the rhythm and precision that started it all.

Ready to see how you measure up? Tap into history and test your skills on the Morse Speed Typer. You might just find a new appreciation for the simple ... --- ... that connected the world.

]]>Can You Hack a Phone with Your Voice?https://kmcd.dev/posts/can-you-hack-a-phone-with-your-voice/Tue, 15 Jul 2025 10:00:00 +0000https://kmcd.dev/posts/can-you-hack-a-phone-with-your-voice/ <p> <img hspace="5" src="https://kmcd.dev/posts/can-you-hack-a-phone-with-your-voice/cover_hu_f5daa6597fe6bacd.webp" /> </p> Or How I Got Thousands of Mastodon Users to Whistle at Their Screens Thousands of people on Mastodon recently started whistling at their computer screens. I’m the one who asked them to do it. No, it wasn’t a strange new CAPTCHA, but an experiment in hacking history.

You see, long before the internet became our digital playground, a different breed of tech enthusiast was busy exploring the phone network. These “phone phreaks” were the original hackers, and one of their most legendary feats was using a simple 2600Hz tone to gain control of phone lines in the US.

Now, let’s be clear: most phreakers couldn’t just whistle this with perfect pitch. That’s the stuff of phreaking legend. In reality, they were a resourceful bunch, using whatever they could get their hands on—instruments, tone generators, and even a toy whistle that famously came in a box of Cap’n Crunch cereal. That little plastic toy just happened to produce a perfect 2600Hz tone, making it an unlikely key to the entire phone system. This discovery helped kick off a whole subculture of people building “blue boxes” and other devices to explore the network’s hidden depths. The community’s name is immortalized in the legendary 2600: The Hacker Quarterly magazine and its podcast, Off The Hook.

As a developer with a soft spot for retro-tech, I had to bring this piece of history to life. So, I built the “Phone Phreak Emulator.” It’s a simple web app that listens to your whistle and tells you how close you get to that magic frequency. No free calls, sorry, but you do get bragging rights.

Here’s what it looks like:

26000Hz Phreaker emulator

I tossed a link up on Mastodon, and things got a little wild. The post exploded. Thousands of people started whistling at their screens. It turns out, hitting that perfect tone is a lot harder than it sounds. I, for one, can barely squeak out 1500Hz. My wife, a woodwind player? She nails it. Consistently.

But beyond the fun of watching the stats climb, something even more incredible happened.

How Mastodon Gave Me a History Lesson

The most incredible outcome of this little project was the history lesson I received from the community. I put out a simple app, and what I got back was a masterclass in telecommunications history. The replies were flooded with fascinating details from people who were there, engineers, and fellow enthusiasts. The thread felt like old friends sharing stories and personal anecdotes around a campfire.

I learned about the intricacies of in-band signaling, the difference between the US “Ma Bell” system and the UK’s GPO network, and so much more. This flood of knowledge was too good to ignore. Thanks to the community’s wonderful, detailed replies, I discovered their phone network used 2280Hz. As a direct result of that fantastic, crowdsourced history lesson, I updated the app. The Phone Phreak Emulator now includes a “UK Mode,” allowing you to test your skills against the 2280Hz tone as well.

It’s been a beautiful, chaotic, and wonderfully nerdy experience. It was a whole community coming together to collectively remember a piece of hacking history, all while looking slightly ridiculous as they whistled at their monitors and phones.

So, if you’re feeling adventurous and want to see if you have the vocal chops of a legendary phone phreak, give it a shot. Head over to phreak.kmcd.dev and let me know how you do. Just don’t blame me if your coworkers start giving you funny looks.

]]>Visualizing the Internet (2025)https://kmcd.dev/posts/internet-map-2025/Mon, 16 Jun 2025 00:00:00 +0000https://kmcd.dev/posts/internet-map-2025/ <p> <img hspace="5" src="https://kmcd.dev/posts/internet-map-2025/cover_hu_1b09e4230b456bc4.webp" /> </p> An all-new interactive map of the Internet, showing the evolution of undersea cables and internet exchanges with year-by-year animation and detailed statistics. For the past couple of years, I’ve been creating visualizations of the internet’s physical infrastructure. This project pieces together data from a few sources, and for me, seeing this data visualized together is compelling. These maps show the undersea fiber optic cables that form the backbone of global connectivity and the Internet Exchange Points (IXPs) where networks meet. This year, I’m thrilled to announce a major evolution of the project. Instead of static images and pre-rendered videos, the Internet Map is now a fully interactive, animated map that you can see online.

You can explore the new map live at map.kmcd.dev.

The new version lets you take control. You can pan, zoom, and step through time from the earliest days of the subsea cable network to the latest deployments in 2025. A new statistics panel provides a detailed snapshot for any given year, offering a richer understanding of how our connected world has grown.

Map of the Internet

What You’re Looking At

The map visualizes two critical components of the internet’s physical layer.

A Note on What You’re Seeing (and Not Seeing)

It’s important to note that this map visualizes publicly available data, which doesn’t capture the full picture of global connectivity. The city peering information, for instance, is sourced from PeeringDB, which tracks publicly advertised connections at IXPs. A vast amount of internet traffic also flows through private peering arrangements and paid transit links that are not publicly documented and therefore do not appear here.

Similarly, the map focuses on the intercontinental backbone of submarine cables. It does not show the incredibly dense web of terrestrial fiber optic cables that run under our streets and alongside major roads. While that data would be fascinating, visualizing it would be overwhelming, and acquiring a complete dataset is nearly impossible as network providers rarely share this proprietary information.

Submarine Cables

The lines snaking across the ocean floors are submarine communications cables. These bundles of fiber optic strands are the high-speed data arteries that connect continents. Laying and maintaining them is a modern marvel of engineering, involving everything from specialized cable-laying ships to underwater robots for repairs. As you explore the map, you can see how the web of these cables has become denser over time, enabling the global, real-time communication we now take for granted. By the start of 2025, the network has grown to 599 cables, spanning a staggering 1,602,092 kilometers.

Map of the Internet

A Physical Target: Vulnerabilities and Sabotage

While these cables are heavily armored, especially in shallower coastal waters where most damage occurs, their isolation on the seabed makes them vulnerable. For decades, the most common threat has been accidental damage from fishing trawlers and dragged anchors. However, in recent years, a more alarming trend has emerged: intentional sabotage. The increasing frequency of suspicious cable cuts suggests that these vital arteries of communication are becoming targets in geopolitical conflicts, a reality that may have brought many new visitors to this map.

Here are just a few of the many recent incidents:

  • The Balticconnector Pipeline and Cable Damage (October 2023): The Balticconnector gas pipeline and two telecom cables between Finland and Estonia were damaged by a dragged anchor from the Hong Kong-flagged ship Newnew Polar Bear. China later admitted its vessel was responsible but claimed it was an accident.

  • The C-Lion1 and BCS East-West Interlink Cuts (November 2024): Two key telecom cables in the Baltic Sea, C-Lion1 (Finland-Germany) and BCS East-West Interlink (Lithuania-Sweden), were severed. The Chinese-owned cargo ship Yi Peng 3 was the primary suspect after it was observed making anomalous movements in the area.

  • The Christmas Day Baltic Cable Cuts (December 2024): On Christmas Day 2024, the Estlink 2 power cable and other telecom cables between Finland and Estonia were damaged by a dragged anchor. Finnish authorities seized the suspected vessel, the oil tanker Eagle S, which was identified as part of Russia’s “shadow fleet”.

  • The Matsu Islands Blackout (February 2023): Two undersea cables connecting Taiwan to its outlying Matsu Islands were severed by Chinese vessels, leaving the 14,000 residents with severely disrupted internet for over 50 days. The incident highlighted the societal impact of such disruptions and was seen as part of a broader pressure campaign by China.

  • The Trans-Pacific Express Cable Cut (January 2025): The major Trans-Pacific Express international cable was cut near Taiwan, with suspicion falling on the Chinese-owned cargo ship Shunxin 39. The vessel had a history of using multiple identities to evade tracking and sailed erratically over the cable’s location before the incident.

  • The Red Sea Cable Disruption (February 2024): Three critical cables in the Red Sea were severed by the anchor of the sinking cargo ship Rubymar, which had been struck by a Houthi missile. The incident disrupted a significant portion of Europe-Asia data traffic and highlighted the vulnerability of infrastructure in contested maritime chokepoints.

  • The Gulf of St. Lawrence Sabotage (December 2023 & 2024): A subsea cable connecting Nova Scotia and Newfoundland was deliberately cut in December 2023 and again in December 2024. Evidence showed an “angle grinder cut” through the steel-wrapped cable, confirming sabotage, though the perpetrator and motive remain unknown.

These examples represent only a fraction of such incidents, which have escalated in frequency and impact in recent years.

Internet Exchange Points (IXPs)

The circles on the map represent cities with Internet Exchange Points. If submarine cables are the interstate highways of the internet, then IXPs are the bustling, hyper-connected metropolitan areas where all the traffic is headed.

An IXP is a physical data center, or a set of connected data centers, where many different networks can physically plug into each other to exchange traffic directly. This process is called “peering.”

So why do hundreds of networks choose to gather in the same buildings in cities like Frankfurt or Amsterdam? The answer is a powerful network effect that you could call digital gravity. The value of an IXP is determined by the networks present there. Once a major network joins, it becomes exponentially more attractive for others to join as well.

The most powerful sources of this gravity are large content providers like Google, Meta, Apple, and Netflix. These companies have an “open peering policy.” In essence, they are saying to any Internet Service Provider (ISP): “Connect with us directly here at the IXP, and we will give your customers a faster, better path to our services.”

This creates a powerful win-win scenario:

  • The ISP wins because their customers get lightning-fast, low-latency access to YouTube, Google Drive, or Apple’s App Store. This makes the ISP’s own service more valuable and competitive.
  • Google and Apple win because they get to deliver their content without paying high fees to third-party backbone carriers. Every byte of data they serve over a direct peering connection is money saved.

This economic incentive is the engine that drives the growth you see on the map. ISPs flock to the IXPs where the big content providers are, which in turn attracts more content providers, creating a feedback loop of ever-increasing capacity and value. This is why a handful of cities have become global hubs with staggering traffic volumes, while others remain smaller, regional nodes.

D2 Diagram

The World in 2025: A Snapshot

The animation and data now extend to 2025, revealing significant ongoing investment. In this year alone, 31 new cables were added (or are promised very soon), stretching over 144,320 kilometers, enough to circle the earth three and a half times.

Some of the longest and most impactful new cables of 2025 include:

  • Bifrost (19,888 km): A monumental project directly connecting Singapore to North America via Indonesia, the Philippines, and Guam. It’s a joint effort by Meta, Keppel, and Telin to bolster connectivity across the Asia-Pacific region.
  • Echo (17,184 km): Another critical trans-Pacific cable, built by Google and Meta, that forges a new, resilient path from the U.S. to Singapore, also landing in Guam and Indonesia. This cable deliberately avoids the crowded northern routes to increase network diversity.
  • Firmina (14,517 km): A Google-led cable enhancing the North-South America connection. It runs from the U.S. East Coast to Argentina, with landings in Brazil and Uruguay, dramatically improving access to Google services in South America.
  • TPU (13,470 km): A Google-owned cable system connecting the U.S. with Taiwan and the Philippines, bolstering trans-Pacific capacity.
  • JUNO (11,710 km): A cable system by Seren Juno Network connecting Japan to the U.S., utilizing advanced technology to offer a high number of fiber pairs and enhance communication resiliency.

Regional Peering Powerhouses

Looking at the total peering capacity reveals a clear global hierarchy. Europe remains the undisputed leader, with an incredible 1.5 Pbit/s of capacity. Asia and North America follow with robust networks of their own, while South America shows impressive growth.

pie title Peering Capacity by Region (2025)
    "Europe": 1500
    "Asia": 430
    "North America": 403
    "South America": 313
    "Africa": 67.9
    "Oceania": 67.2

This massive regional capacity is concentrated in a few key metropolitan hubs. The list of top peering cities shows just how vital they are to the global network:

  • Amsterdam, NL: 200 Tbit (+12.7 Tbit)
  • Frankfurt, DE: 166 Tbit (+8.19 Tbit)
  • São Paulo, BR: 157 Tbit (+3.16 Tbit)
  • London, GB: 113 Tbit (+3.21 Tbit)
  • Tokyo, JP: 90.2 Tbit (+2.18 Tbit)

The growth in a city like São Paulo is remarkable and shows the increasing investment in internet infrastructure in South America, directly supported by new cables like Firmina.

How It’s Made: The New Tech Stack

The transition to an interactive map required a complete overhaul of the technology stack. The previous versions, which relied on generating static SVG images, faced several challenges. It was difficult to dynamically size the lines representing cables and find the right balance of detail for country borders; too much detail slowed the map down, while too little looked simplistic when zoomed in.

The solution was to adopt a tile-based methodology, where map tiles at different levels of detail are fetched dynamically as a user zooms—the same concept used by Google Maps. I was faced with a choice: implement this highly complex tiling logic myself or use a well-supported library. Since the project’s focus was on data visualization, I opted for the more direct path by using Leaflet, a powerful library for creating dynamic and interactive maps.

The back-end Go scripts that gather and process the data from sources like TeleGeography and PeeringDB were largely unchanged, only needing a few new fields in the JSON output to power the new front end.

Map of the Internet

Beyond the core technology, I also wanted to incorporate a few small details to improve the user experience and make the map feel more intuitive and personal:

  • Personalized View: The map automatically geolocates your region and centers the initial view there. My hope is that the map feels familiar and relevant the moment you open it, no matter where you are in the world.
  • Persistent ‘About’ Section: The ‘About this map’ panel remembers its state. If you close it, it stays closed on subsequent visits and refreshes until you decide to open it again.
  • Relative City Sizing: The size of each city circle is relative to its total peering bandwidth, giving an immediate visual sense of where the major hubs of connectivity are concentrated.
  • Interactive Highlighting: Hovering over or clicking on any cable or city highlights it and brings it to the forefront. This small detail makes it much easier to focus on and explore individual parts of the network.

Closing Thoughts

This project continues to be a fascinating exploration of the physical reality of our digital world. By making the map interactive, I hope to provide a more powerful tool for anyone curious about the immense and intricate infrastructure that underpins our daily lives. As global data demand soars, the growth of these subsea cables and peering exchanges will only become more critical. Explore the map, watch the internet grow, and see for yourself how the world gets connected.

This project was a significant undertaking, and I’ve been thrilled to see it shared in various places online. If you choose to share it, I only ask that you please provide attribution by linking back to the project, just as I give credit to my own data sources, TeleGeography and PeeringDB.

Thank you for reading, and please feel free to explore and share the map with others! map.kmcd.dev

]]>HTTP QUERY and Gohttps://kmcd.dev/posts/http-query/Wed, 04 Jun 2025 10:00:00 +0000https://kmcd.dev/posts/http-query/

We need another HTTP verb. I'll explain why. ]]>You’re likely familiar with the HTTP methods, GET and POST, the workhorses of HTTP. These have both worked surprisingly well and have provided well-defined caching behavior for over a quarter of a century. However, neither of these solves the problem of complex request parameters without completely throwing out the caching semantics of GET. This is where a concept like a QUERY method comes in, and it’s not just a thought experiment; it’s an area actively being explored by the IETF HTTP Working Group.

But, why?

The standard HTTP methods serve us well, except the ones that don’t… but that’s a topic for another day. GET is great for simple data retrieval, but its reliance on URL parameters makes it cumbersome for complex queries or large sets of input parameters. URLs have practical length limits, and embedding deeply nested structures is awkward.

This often leads developers to use POST for what are semantically read-only query operations. Many widely-used protocols effectively do this:

  • GraphQL typically uses POST requests with JSON bodies to send queries.
  • gRPC and gRPC-Web typically rely exclusively on POST.
  • Older protocols like SOAP and XML-RPC almost exclusively use POST to encapsulate their operations, including data retrieval. This has been an issue for a long time!

While using POST works, it’s a compromise. POST traditionally implies an action that might change state on the server, which means intermediaries (read: caches) and even client-side logic might treat these “query-via-POST” requests with undue caution, forgoing caching or automatic retries that would be safe for a truly read-only operation.

This is precisely the problem that a dedicated safe method with a request body aims to solve. The IETF HTTP Working Group is discussing such a method in a draft titled “A Safe HTTP Method with a Request Content”. As the draft states:

The QUERY method provides a solution that spans the gap between the use of GET and POST. As with POST, the input to the query operation is passed along within the content of the request rather than as part of the request URI. Unlike POST, however, the method is explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.

While the draft uses “QUERY” as a candidate name (among others), the core idea is what we’re exploring: a method for safe, idempotent data retrieval that can carry a payload. For the rest of this article, we’ll continue to use “QUERY” to represent this concept and show how you can implement such a custom method in Go today.

It’s crucial to remember that until such a method is formally standardized and widely adopted, using custom HTTP methods can impact interoperability. However, for internal APIs, tightly controlled systems, or as a forward-looking experiment, they can be very useful.

Server-Side with Go

Go 1.22 introduced enhancements to http.ServeMux that allow you to register handlers for specific HTTP methods and paths more directly. Let’s build a server that handles our custom QUERY method:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
)

func queryHandler(w http.ResponseWriter, r *http.Request) {
	// The new ServeMux handles method checking based on registration.

	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "Error reading request body", http.StatusInternalServerError)
		return
	}
	defer r.Body.Close()

	// In a real application, you'd parse the query from the body
	// (e.g., JSON) and fetch data accordingly.
	fmt.Fprintf(w, "Received your QUERY request with body: %s\n", string(body))
	log.Printf("Handled QUERY request with body: %s", string(body))
}

func main() {
	mux := http.NewServeMux()
	// Register handler specifically for QUERY method on /data path
	mux.HandleFunc("QUERY /data", queryHandler)

	log.Println("Server starting on port 8080, handling custom QUERY method...")
	if err := http.ListenAndServe(":8080", mux); err != nil {
		log.Fatal(err)
	}
}

In this server:

  1. We use mux.HandleFunc("QUERY /data", queryHandler) to directly associate the queryHandler with our custom QUERY HTTP method for the /data path.
  2. The queryHandler reads the request body, where the complex query parameters would reside.

Congratulations, we’ve made an API that uses the QUERY method. Now let’s write a client to pair with this server.

Client-Side, also with Go

Here’s how a Go client can send a QUERY request:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
)

func main() {
	queryPayload := `{"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}`
	client := &http.Client{}

	// Create a new request with the custom "QUERY" method
	req, err := http.NewRequest("QUERY", "http://localhost:8080/data", strings.NewReader(queryPayload))
	if err != nil {
		log.Fatalf("Error creating request: %v", err)
	}
	req.Header.Set("Content-Type", "application/json") // Important for body processing

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Error sending request: %v", err)
	}
	defer resp.Body.Close()

	fmt.Println("Response Status:", resp.Status)
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Error reading response body: %v", err)
	}
	fmt.Println("Response Body:", string(body))
}

The client uses http.NewRequest("QUERY", ...) to specify the custom method and sends the queryPayload in the request body.

Client-Side with cURL

Although web browsers won’t randomly make QUERY calls, many other tools can use QUERY. Many also support arbitrary HTTP verbs, although this isn’t typically leveraged due to filters from load balancers, firewalls, etc.

Anyway, here’s what that request looks like with cURL:

$ curl -X QUERY -d '{"filters": {"status": "active", "category": "electronics"}, "
fields": ["name", "price"]}' http://localhost:8080/data -v
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> QUERY /data HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 89
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 89 bytes
< HTTP/1.1 200 OK
< Date: Sat, 31 May 2025 15:47:45 GMT
< Content-Length: 129
< Content-Type: text/plain; charset=utf-8
< 
Received your QUERY request with body: {"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}
* Connection #0 to host localhost left intact

QUERY vs. GET and POST - A Clearer Separation

Let’s summarize the distinctions with our (potentially future-standard) QUERY method in mind:

  • GET: Used for retrieving data. Parameters are typically sent in the URL. GET requests must be safe and idempotent. They generally don’t have a body.
  • POST: Used for submitting data to be processed, often resulting in a change of state or side effects on the server (e.g., creating a new resource). Parameters are sent in the request body. Not necessarily safe or idempotent.
  • QUERY (custom/proposed): Used to request data (like GET) but with the ability to send a complex query in the request body (like POST). Crucially, it is defined as safe and idempotent (like GET). This explicitly tells intermediaries and clients that the request has no side effects and can be cached or retried automatically.

Caching Behavior: GET, POST, and QUERY

HTTP caching is vital for performance. A method’s characteristics (especially safety and idempotency) directly influence its cacheability.

1. GET

  • Behavior: GET requests are inherently cacheable. Caches readily store and serve responses to GET requests if caching headers (like Cache-Control, Expires, ETag) allow. This is a foundational aspect of HTTP performance.

2. POST

  • Behavior: POST requests are generally not cacheable by default. Since POST can have side effects, caching responses could lead to unintended consequences or stale data if the action isn’t repeated.

3. QUERY

The IETF draft emphasizes that a method like QUERY is “explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.” This is key.

  • Behavior (Current Custom Method): Because our QUERY method is non-standard today, caches will not cache it by default. They typically only automatically consider standard safe methods like GET. To make responses to a custom QUERY request cacheable:
    • The server must send explicit caching headers (e.g., Cache-Control: public, max-age=3600). These headers signal the cacheability of the response.
    • Intermediary caches (CDNs, reverse proxies) might need specific configuration to recognize and cache responses for this custom method.
    • The Vary HTTP header is important if the response depends on the request body content.
  • Behavior (Future Standardized Method): If a method like QUERY becomes a recognized HTTP standard, caches would likely treat it similarly to GET for caching purposes, provided it’s implemented according to its safe and idempotent semantics. This would be a major advantage, allowing complex queries with bodies to be cached as effectively as GET requests.

Key Takeaway for QUERY Caching

The intent of a QUERY method is to be cacheable. If using a custom method today, you must provide explicit caching directives. The ongoing standardization effort aims to make this caching behavior more automatic and universally understood, unlocking performance benefits for complex, body-inclusive queries that are currently often forced into less cache-friendly POST requests. How long will this process take? Who knows! These kinds of changes can take decades.

What’s Next?

The discussion around a safe HTTP method with a request body is an exciting development. By understanding its purpose and experimenting with custom methods like QUERY in Go, we can better appreciate the nuances of HTTP and prepare for potential future standards that will make our APIs more robust and performant.

Some projects have already taken to adding support for the QUERY method. Take a look at NodeJS: Support for ‘QUERY’ method, which added support last year. As demonstrated earlier in this article, using QUERY is already possible in Go. This general support is often true for other languages and libraries as well, since custom HTTP methods are a feature of the HTTP specification.

The other aspect of moving this forward is a bit more nebulous: bureaucracy. There is still work and review to be done to graduate the draft into an official RFC from the IETF. But fear not. There’s actually steady changes being made to the draft document. You can tell this from the git history on the httpwg’s repo. At the time of writing May 19th was when the last change was made, so I’m certain that this hasn’t been forgotten about.

]]>I made a daily word gamehttps://kmcd.dev/posts/wordseq/Wed, 14 May 2025 10:00:00 +0000https://kmcd.dev/posts/wordseq/ <p> <img hspace="5" src="https://kmcd.dev/posts/wordseq/cover_hu_431a8b94add23f27.webp" /> </p> Taking a peek under the covers of making a daily puzzle game Ever stare at a word so long it stops looking like a word? Like naming a variable data for the 8th time and suddenly wondering what data even means?

That effect is called semantic satiation. I made a game that dives headfirst into the theme.

In wordseq, you swap letters to form words, but the deeper you go, the more the grid feels like a linguistic fever dream. One moment you’re proud to find “plop”, the next you’re doubting if “plop” was ever a real word or just a sound effect from a comic book. You win by dragging yourself back to meaning. Enjoy the victory while you can, because there’s a new puzzle tomorrow.

I’m excited to share a look behind the scenes of my new daily word puzzle game, wordseq. If you haven’t tried it yet, you can play the latest puzzle here!. wordseq is a game where you swap adjacent letters to form new words, aiming to find the longest possible sequence. Here’s what it looks like to play the game (by a super unrealistically fast player).

wordseq gameplay

One of the most challenging and rewarding aspects of building wordseq was developing the system that generates the daily puzzles. My goal was to create levels that are not just solvable, but also consistently engaging, fair, and offer that satisfying “aha!” moment where the answer you’ve been looking for hits you in the face. Conversely, I want to avoid the “Huh, is this actually the solution?” moments.

In this post, we’ll explore the intricate process of generating these daily puzzles. We’ll cover everything from initial grid creation and word validation to ensuring puzzles are solvable and use interesting vocabulary, leveraging techniques like concurrency in Go and smart dictionary management.

wordseq logo

Core Gameplay Mechanics

Before we dive into generation, here’s a quick overview of how to play wordseq:

  • The Core Mechanic: Players swap any two letters that are directly next to each other (horizontally or vertically).
  • Word Formation: Each swap must result in at least one new word spanning the length of a row (this length can vary by difficulty). This new word is formed by the entire row or column where the swap occurred.
  • The Goal: The objective is to find the longest possible sequence of valid words by making these swaps. This is what we call the “optimal path.”
  • Difficulty Levels: wordseq offers ’normal’, ‘hard’, and ‘impossible’ modes, which influence parameters like grid size, word length, and the complexity of the solution paths the generator aims for.

To better understand how these swaps lead to word formation, let’s look at the visual feedback provided to the player during gameplay.

Movement

Here is what the movement looks like.

Bad move, results in no new words. Don’t you like the little wiggle? It’s the small things that really shape the experience:

letters wiggling and turning red

Non-optimal move, a move that sets you up to fail. You may think that you made a clever move and discovered a word just slightly askew, but there are many times where there’s another word that you need to find first to best set up the longest sequence. You will be greeted by a not-as-scary orange color. Usually you will want to undo your move and look for another:

letters swapping and a newly completed word turning orange

And, of course, the optimal move, the move that finds a new word but also sets you up for the most followup words.

letters swapping and a newly completed word turning green

The challenge with this game isn’t only finding/completing words, but also setting yourself up for the next one. Don’t worry, there’s no downside for finding the ‘wrong’ paths.

The Tech Stack

The level generation is a computationally intensive process, so I opted to build the generation script in Go, primarily for its excellent performance and built-in support for concurrency, but also it’s the language that I know best right now. This script outputs JSON data for each level. There will be more on the level format later.

The frontend, where you play wordseq, is built with:

The JSON level data from the Go script is fetched by the React frontend to power each day’s puzzle.

Generating Engaging Levels

Crafting a “good” wordseq puzzle involves several steps and considerations. We want puzzles that aren’t too easy, don’t rely on super obscure words, and have a satisfying solution depth. Here’s how the generator tackles this:

Step 1: Initial Grid Generation

  • It starts with Randomness: The process starts by generating a grid (e.g., 5x5) filled with random letters. However, just picking letters completely at random can lead to grids full of very rare letters (like Q, Z, X, J) which makes forming words difficult or leads to trivial “write-offs” where those letters are ignored. To combat this, the generator uses approximate letter frequency (based on English language letter usage) to populate the grid. This creates a more balanced and natural distribution of letters, similar to what you might find in a Scrabble bag.

  • Initial Validation: Before any moves are possible, the generator verifies that the starting grid contains no valid words matching the puzzle’s target length. This ensures the puzzle begins only with the player’s first action and avoids potential confusion where the initial state might seem like part of the solution, or where swapping back to the original layout could be misinterpreted as finding a word.

This check involves looking for rows in each row and column, like this:

D2 Diagram

It is important to note that words spent going upwards and to the left aren’t seen as words. The game would be far too complex if that were the case.

Step 2: Building the Exploration Tree - Finding All Possibilities

This is where the magic (and a lot of computation) happens. For a given initial grid that passes validation, the generator explores all possible valid game sequences:

  • Iteration: From the current grid state, it identifies every possible adjacent letter swap.
D2 Diagram
  • Validating Swaps (The Big Dictionary): For each potential swap, it temporarily performs the swap and then checks if the new grid configuration forms at least one new word that spans the length of the grid (horizontally or vertically). This check uses a large, comprehensive dictionary. The reason for this large dictionary is critical for player experience: if a player sees a word on the grid and forms it, the game must recognize it, even if it’s a bit uncommon. It’s incredibly frustrating for a game to not accept a word you know is valid!
D2 Diagram
  • Recursive Exploration: If a swap is valid (i.e., forms at least one new word), this new grid state and the move that led to it become a new “node” in an exploration tree. The process then recursively explores all valid swaps from this new state, and so on. This continues until no more valid moves can be made from a state, or a predefined maximum search depth is reached.
D2 Diagram

Step 3: Analyzing the Tree

Once an exploration tree (or a significant portion of it) is built for an initial grid:

  • Calculating maxDepthReached: For every node in the tree (each representing a game state reached by a valid move), the generator calculates the maxDepthReached. This value signifies the length of the longest possible sequence of further valid moves that can be made starting from that particular node.
  • Determining Overall Puzzle Depth: The overall “solution length” for the puzzle (which the frontend knows as gameData.maxDepthReached) is the highest maxDepthReached value found among the initial possible moves from the starting grid. This represents the longest chain of words the player can achieve.
  • Filtering Puzzles by Length: We use RequiredMinTurns and RequiredMaxTurns (configurable parameters for the generator) to filter these potential puzzles. Grids that lead to solutions that are too short (not challenging enough) or excessively long (potentially too tedious) are discarded.

Step 4: Word Dictionaries

This is where the two-dictionary approach comes into play:

  • The Large Dictionary (for Gameplay Logic): As mentioned, this is used during the tree exploration step to ensure any valid word a player might form is recognized. At the time of writing, the large dictionary has 129,493 words in it.
  • The Smaller, Curated Dictionary (for Puzzle Quality): After a potential puzzle is generated, the generator collects all unique words that appear in any valid path within that entire tree. These words are then checked against this smaller, more “usable” or common dictionary. If the given puzzle uses words that are too obscure or too obscene then the puzzle is discarded entirely. Many potential puzzles are discarded from this check. This is a key step to ensure the final puzzles feel fair and use vocabulary that players are likely to know or find satisfying to discover. At the time of writing, this dictionary has 6,956 words - an absolutely tiny amount compared to the large one.
  • Controlling Word Variety: Another filter is MaxUniqueWords. If a potential puzzle contains too many distinct words across all its possible solution paths, it might be too chaotic. This parameter helps keep the word set focused.

Step 5: Concurrency

Generating a single grid, building its exploration tree, and then validating it against all these criteria can take time. To find enough high-quality puzzles for daily release, I needed to process many initial random grids. Doing this sequentially was quite slow, so I ended up splitting up the work by using go routines. This greatly improved the throughput. It’s honestly so nice to have a highly parallelizable, CPU-bound problem to work with. It’s a nice break from the world of the web where I/O is typically the bottleneck.

Step 6: Outputting the Puzzle Data

Once a grid and its exploration tree pass all the filters, it’s deemed a “good” puzzle. The generator then outputs JSON with the details about a puzzle. To illustrate this, consider a simplified example with a single possible move for illustration purposes:

{
  "initialGrid": [
    ["i","s","w","y"],
    ["e","v","a","p"],
    ["r","a","b","a"],
    ["h","l","l","a"]
  ],
  "wordLength": 4,
  "maxDepthReached": 1,
  "explorationTree": [
    {
      "move": {
        "from": [1, 0],
        "to": [1, 1]
      },
      "wordsFormed": ["seal"],
      "maxDepthReached": 0,
      "nextMoves": []
    }
  ]
}
  • initialGrid: The starting letter configuration.
  • wordLength: The target word length for this puzzle.
  • maxDepthReached: The length of the optimal solution path(s).
  • explorationTree: The full tree structure, containing all valid moves, the words they form, and the maxDepthReached from each node.

This JSON file is then what the React frontend loads each day to power the game you play. This is what an extremely simple grid looks like, with only one valid move from the initial state.

Lessons Learned

  • Dictionaries are Key: The quality and scope of your dictionaries profoundly impact both gameplay fairness and puzzle quality. The two-dictionary system was vital.
  • Iterative Refinement: Level generation isn’t something you get right on the first try. It requires constant tweaking of parameters, testing, and playing the generated levels yourself.
  • Concurrency is Your Friend: For computationally heavy tasks like this, leveraging concurrency (like Go offers) is almost essential for practical generation times.
  • Define “Good” as early as you can: Having clear criteria for what constitutes a good puzzle (solvable, right length, good words) helps guide the entire generation logic. I was lucky to have my wife, who’s a daily word game solver as a test user. Her feedback was (and is) invaluable. Making a game is an iterative process. A lot of times you legitimately don’t know if the game you’re making is fun. Who knows, maybe I made a game only my wife loves. And honestly? If she’s the only one who loves it, that still feels like a win. (But I do hope you’ll enjoy it too.)
  • Tools to help testing are worth it: I didn’t mention this yet, but I have some tooling now to make testing a lot easier. Not only do I get clear display of the solutions (cheap mode!) but I can push a button and have the game played for me. This has helped a lot when I needed to get to certain game states quickly.
  • Data structures and Game Design: I really love how this game relies heavily on computer science fundamentals and data structures. At the end of the day this is just a non-obvious tree traversal with a UI on top of it.

What’s Next for wordseq’s Puzzles?

I think there’s a lot of additional things I can use to judge the difficulty of a given grid. I could judge how often completed words switch rows for the next word, or (if possible), how often it alternates from rows to columns. Often, the puzzle will be easier when it’s only one row that is constantly updating. Changing one letter of an existing word is a lot easier for a mind to handle than re-evaluating the whole board every turn.

There’s always more refinement that can be done with the dictionary. The small dictionary’s pretty solid now, though an occasional spicy word still sneaks through. Oops.

On the frontend side, I feel like I’m getting close to a decent interface. I want to build an “infinite mode” where you can play random levels (identified by an ID so you can link them and go back to those puzzles later). I actually think this wouldn’t be too crazy with the way the components are laid out now.

I’m also contemplating how hard it might be to implement a Danish version as I am learning Danish.

Conclusion

Building the level generator for wordseq has been a fascinating journey into algorithms, data structures, and the subtle art of puzzle design. It’s a complex system, but seeing it produce fun and challenging puzzles each day is incredibly rewarding.

I hope this peek behind the curtain was interesting!

  • Play wordseq daily: wordseq.com
  • I’d love to hear your feedback on the puzzles or any thoughts you have on level generation. Drop a comment below or find me on Blue Sky or Mastodon!

Thanks for reading!

]]>Protovalidate: Can Input Validation Be This Easy?https://kmcd.dev/posts/protovalidate/Tue, 11 Feb 2025 10:00:00 +0000https://kmcd.dev/posts/protovalidate/ <p> <img hspace="5" src="https://kmcd.dev/posts/protovalidate/cover_hu_7516b8e4ea8158ed.webp" /> </p> Effortless input validation for Protobuf! Protovalidate lets you define rules directly in your .proto files.

Since writing this article, ECMAScript support for protovalidate was created and has had its first release with a new library called protovalidate-es! Finally, an easy way to share validation rules across backend, frontend and to all kinds of other clients and message-based systems is here.

User input can be absolute garbage. I can’t be trusted to type my name correctly half of the time, so it’s obvious that we need to validate input to catch the most obvious mistakes. Zip codes don’t include letters, number of pets can’t be negative and Hitachi Rail Italy Driverless Metro is not a model of car.

Input validation is a consistent problem in the web services industry. Usually there are constraints that are defined at multiple levels:

  • Databases often have constraints. For example, the size of a single field is usually constrained in some way and there are usually types that offer some guarantees.
  • Statically typed programming languages have some amount of safety for input. You know an integer type isn’t going to magically become a string based on user input.
  • There are many libraries that assist with this, but backend programmers will have to add constraints on input from users. Usernames can’t be 1000 characters long, someone can’t be aged negative 10, etc. This validation is often the most important for data integrity because it’s the last part of the architecture where developers have 100% control.
  • On the frontend, many of the constraints and rules from the backend are often replicated. This is usually done for UX (User eXperience) reasons. You don’t want to wait until you submit a large form with many inputs before realizing you messed up. It’s often way better to highlight issues before users try to submit. Often, these constraints are communicated by hand-written documentation that is often not kept up-to-date or not at all, forcing frontend developers to duplicate backend validation logic.
D2 Diagram

This is where protovalidate comes in. Protovalidate allows you to specify constraints beside your protobuf-defined API and type definitions. I talked before about API contracts, but protovalidate goes a step above what protobuf offers by default. In addition to the cross-language type safety that protobuf offers, protovalidate allows you to define additional constraints for each field. So now the answer to many input validation questions can be answered by directly looking at this file or by using a protovalidate library written for several languages. Now both the frontend and backend logic can be powered by the same declarative schema.

D2 Diagram

Why protovalidate?

The traditional approach to input validation is often a fragmented and inconsistent mess. Frontend and backend teams may implement their own validation rules, leading to duplicated code, discrepancies, and the constant struggle to keep everything in sync. Imagine the headache of trying to track down and fix a validation bug that exists in three different places! Protovalidate offers a much better way. It provides a centralized, declarative approach, allowing you to define your validation rules directly in your .proto files – the same place you define your data structures. For example, specifying that a username must be between 3 and 50 characters is as simple as adding a few annotations to your Protobuf definition: string name = 1 [(buf.validate.field).string.min_len = 3, (buf.validate.field).string.max_len = 50];. This not only significantly reduces boilerplate code but also ensures consistency across your entire application. Because the rules are defined alongside your data structures, they become a living part of the API contract, improving communication between teams and reducing the risk of misinterpretations.

Protovalidate supports multiple languages, including Go, Java, Python, and C++, with community support for others like .NET. Built on top of the Common Expression Language (CEL), Protovalidate offers a powerful and extensible way to define even the most complex validation rules, ensuring your application’s data remains clean and reliable.

Showing it off

I can tell you how cool protovalidate is, but I figured it would be better to show you. Let’s look at some of the constraints.

Built-in constraints

Protovalidate comes with a set of built-in constraints for common data types. Let’s explore some of these, starting with strings.

Strings

Let’s start simple. Here’s some constraints that you might see for a username. It has a minimum length of 3 characters and a max length of 50. And this is a required field:

string name = 1 [
  (buf.validate.field).string = {
    min_len: 3
    max_len: 50
  },
  (buf.validate.field).required = true
];

Email addresses can also be validated:

string email = 3 [(buf.validate.field).string.email = true];

You can use regex to have precisely describe what a valid value looks like:

// City of residence, must only contain letters, spaces, and hyphens.
string city = 5 [(buf.validate.field).string.pattern = "^[a-zA-Z]+(?:[\\s-][a-zA-Z]+)*$"];

Integers

int32 age = 2 [(buf.validate.field).int32.gte = 0, (buf.validate.field).int32.lte = 150];

Enums

One potential issue with Protobuf enums is that they don’t inherently enforce validation against defined values. For example for an enum defined like this:

enum Status { UNKNOWN = 0; ACTIVE = 1; }

Without explicit validation, a server could receive an undefined enum value (e.g., 420) and potentially misinterpret it. To fix this glaring issue, protovalidate has a constraint to limit enums to known and defined values only. Let’s look at a field that has this constraint:

Status status = 4 [(buf.validate.field).enum.defined_only = true];

With a single annotation you just removed a whole class of issues and simplified input validation.

Repeated Fields

There are also constraints for repeated fields. You can constrain the number of items in a list. You can force the items to be unique. You can even add constraints for each item in the list as well.

repeated string tags = 7 [
  (buf.validate.field).repeated.min_items = 1,
  (buf.validate.field).repeated.max_items = 2,
  (buf.validate.field).repeated.unique = true,
  (buf.validate.field).repeated.items.string.min_len = 4
];

CEL-powered custom constraints

One thing that sets protovalidate apart from other validation libraries is the extendability. Protovalidate is built on top of CEL, which is an embeddable expression engine. What this means for protovalidate is that you can write your own constraints. All of the examples above constrain you on a single field, but it can be incredibly powerful to use multiple fields. This can be useful for many situations but in particular it is useful when you have a defined start and end range where the start strictly needs to come before the end. So let’s take a look at what that looks like:

message Event {
  int64 start_time = 1;
  int64 end_time = 2;
  option (buf.validate.message).cel = {
    id: "event.start_time_before_end_time",
    message: "Start time must be before end time",
    expression: "this.start_time < this.end_time",
  };
}

This example demonstrates a constraint involving two fields, start_time and end_time. A reasonable constraint is that start time must go before end time, so that’s what this custom constraint is doing. The expression field contains the CEL expression that enforces this rule. CEL’s simplicity makes it easy to define custom constraints. message allows you to customize the message.

And now are you ready for the big reveal? Every single standard constraint in protovalidate works the same way as this. It’s all powered by CEL expressions. This not only provides you with a good reference to make your own custom constraints but it also makes it much easier to add language support for protovalidate. All you need is a good protobuf and CEL library, and you have most of the work already done for you.

Complete example

See the full prototype from this post here.

See a full example protobuf file here. Reference protovalidate’s validate.proto file to see every option available.

Now, let’s see Protovalidate in action. We’ll use buf curl, a command-line tool for interacting with gRPC and Connect services, to send a request to our example service and observe how Protovalidate enforces the defined validation rules.

buf curl \
    --protocol=grpcweb \
    --http2-prior-knowledge \
    -d '{"profile": {"user": {}}}' \
    http://localhost:6660/example.ExampleService/CreateProfile
{
   "code": "invalid_argument",
   "message": "validation error:\n - profile.user.name: value is required [required]\n - profile.user.email: value is empty, which is not a valid email address [string.email_empty]\n - profile.user.city: value does not match regex pattern `^[a-zA-Z]+(?:[\\s-][a-zA-Z]+)*$` [string.pattern]\n - profile.user.account_id: value must be greater than 0 [int64.gt]\n - profile.user.tags: value must contain at least 1 item(s) [repeated.min_items]\n - profile.user.created_at: value is required [required]",
   "details": [
      {
         "type": "buf.validate.Violations",
         "value": "CjAKEXByb2ZpbGUudXNlci5uYW1lEghyZXF1aXJlZBoRdmFsdWUgaXMgcmVxdWlyZWQKXAoScHJvZmlsZS51c2VyLmVtYWlsEhJzdHJpbmcuZW1haWxfZW1wdHkaMnZhbHVlIGlzIGVtcHR5LCB3aGljaCBpcyBub3QgYSB2YWxpZCBlbWFpbCBhZGRyZXNzCmgKEXByb2ZpbGUudXNlci5jaXR5Eg5zdHJpbmcucGF0dGVybhpDdmFsdWUgZG9lcyBub3QgbWF0Y2ggcmVnZXggcGF0dGVybiBgXlthLXpBLVpdKyg/Oltccy1dW2EtekEtWl0rKSokYApBChdwcm9maWxlLnVzZXIuYWNjb3VudF9pZBIIaW50NjQuZ3QaHHZhbHVlIG11c3QgYmUgZ3JlYXRlciB0aGFuIDAKTgoRcHJvZmlsZS51c2VyLnRhZ3MSEnJlcGVhdGVkLm1pbl9pdGVtcxoldmFsdWUgbXVzdCBjb250YWluIGF0IGxlYXN0IDEgaXRlbShzKQo2Chdwcm9maWxlLnVzZXIuY3JlYXRlZF9hdBIIcmVxdWlyZWQaEXZhbHVlIGlzIHJlcXVpcmVk",
         "debug": {
            "violations": [
               {
                  "fieldPath": "profile.user.name",
                  "constraintId": "required",
                  "message": "value is required"
               },
               {
                  "fieldPath": "profile.user.email",
                  "constraintId": "string.email_empty",
                  "message": "value is empty, which is not a valid email address"
               },
               {
                  "fieldPath": "profile.user.city",
                  "constraintId": "string.pattern",
                  "message": "value does not match regex pattern `^[a-zA-Z]+(?:[\\s-][a-zA-Z]+)*$`"
               },
               {
                  "fieldPath": "profile.user.account_id",
                  "constraintId": "int64.gt",
                  "message": "value must be greater than 0"
               },
               {
                  "fieldPath": "profile.user.tags",
                  "constraintId": "repeated.min_items",
                  "message": "value must contain at least 1 item(s)"
               },
               {
                  "fieldPath": "profile.user.created_at",
                  "constraintId": "required",
                  "message": "value is required"
               }
            ]
         }
      }
   ]
}

Here’s the response from the server. Notice the code: “invalid_argument” and the message field, which clearly explains the validation errors. Protovalidate caught six distinct issues: the name field was missing, the email was empty (and therefore invalid), the city didn’t match the required regex, the account_id was zero (not greater than zero), the tags list was empty (requiring at least one item), and the created_at field was missing.

Now, let’s correct the errors by providing valid data. We’ll include a name, a valid email, and so on. When we fix all of these issues, suddenly it works!

$ buf curl \
    --protocol=grpcweb \
    --http2-prior-knowledge \
    -d '{
        "profile": {
            "address": {
                "city": "New York",
                "postal_code": "10001",
                "street": "25th Street West"
            },
            "user": {
                "name": "Bob",
                "email": "bob@example.com",
                "account_id": 10,
                "tags": ["new_user"],
                "created_at": "2023-01-01T00:00:00Z"}}}' \
    http://localhost:6660/example.ExampleService/CreateProfile
{}

As you can see, this validation ensures the data conforms to our defined rules.

My side projects that leverage protovalidate

You’ve probably realized by now that I’m a big fan of protovalidate. So it’s no surprise for you to know that I’ve used it in both of my larger personal projects recently:

protoc-gen-connect-openapi

protoc-gen-connect-openapi generates OpenAPI specifications for ConnectRPC servers. This uses protovalidate rules to further annotate the OpenAPI spec.

FauxRPC

FauxRPC is my project that is a self-mocking protobuf driven server that supports gRPC/gRPC-Web/ConnectRPC/REST. Not only will it enforce protovalidate annotations on requests made to it but it also uses these annotations to generate more realistic mock data automatically. I feel like this really shows off the power of contract-driven APIs. I actually generated the examples above using FauxRPC!

$ buf build ssh://git@github.com/sudorandom/kmcd.dev.git#branch=main,subdir=content/posts/2025/protovalidate -o protovalidate-example.binpb
$ fauxrpc run --schema protovalidate-example.binpb

Run in the web

Currently, protovalidate doesn’t run on a web frontend, at least not yet Update: Now you can!. I feel like this is the last piece that can finally unite input validation in the frontend and backend. I mentioned at the start of this article about how the frontend developers have to duplicate many of the rules that already exist in the backend. However, once a typescript version of protovalidate exists suddenly all of this work is completely handled simply by adding some protobuf annotations.

The potential of this library is incredible. I feel like we can have a world where types are defined in one place and used in any number of languages.

Rough Edges

Protovalidate does have a few rough edges. Overusing validation rules can sometimes make it hard to re-use types for multiple purposes. If you have a type that you use for a user and define a bunch of rules about required fields for that user type, how would you then update only a few of the fields for a user in a different call? I think there are ways around it, but required fields may in fact be the issue yet again. There’s a past debate within Google about required fields and the result was the removal of the feature in core protobuf from proto2 to proto3. I feel like this may be the same issues manifesting again.

The recommended way of dealing with this situation is to avoid re-using types. Buf, the creators of protovalidate heavily dogfood their own validation library in their own APIs. You can see that they separate types meant for creating and types meant for updating. This isn’t super ideal because you likely want to keep most of the rules synchronized between both of these types but based on hard-earned experience, you often do have less problems if you simply create types for each use-case.

So instead of re-using a User type for ‘create’ and ‘update’ operations you could have another type for updating specific fields, like this:

message User {
    string name = 1;
    string email = 2;
    //... other fields
}

message UserProfileUpdate {
    optional string email = 1;
    optional string about_me = 2;
    //... other updatable fields
}

What to take away

Protovalidate is a powerful and versatile tool that simplifies input validation for Protobuf-based applications. By defining validation rules directly in your .proto files, you can ensure data integrity, reduce boilerplate code, and streamline your development process. With its support for custom constraints using CEL expressions, Protovalidate offers flexibility and extensibility for various use cases. While it currently lacks direct integration for web frontends, the potential for a unified validation approach across frontend and backend is super exciting.

So, why wait? Dive into protovalidate today and discover how it can revolutionize your protobuf development workflow. If you’re already leveraging ConnectRPC or gRPC in one of the supported languages, this library is a no-brainer to try.

]]>Behold! The Barcode Scannerhttps://kmcd.dev/posts/barcode-scanner/Mon, 27 Jan 2025 10:00:00 +0000https://kmcd.dev/posts/barcode-scanner/ <p> <img hspace="5" src="https://kmcd.dev/posts/barcode-scanner/cover_hu_a0105d126c57d428.webp" /> </p> I have a confession. I’ve become that person. I recently got my hands on the 5100 Laser 1D Wireless Barcode Scanner, and I’m officially obsessed. My wife and I have started increasing our book collection, so keeping track of books that we have is starting to get a bit tedious. Spreadsheets, handwritten lists, trying to type in those ridiculously long ISBNs… it was annoying.

Enter the barcode scanner.

My barcode scanner

You may have a pre-conceived notion that you need tons of super custom software to operate a barcode scanner but this barcode scanner acts just like a keyboard. You plug in the little USB receiver, scan a barcode, and BAM, the numbers appear whenever your text cursor is. Absolutely no required vendor software or drivers needed.

Why I love it

  • Fast: No more painstakingly typing out ISBNs.
  • Accurate: Typos are a thing of the past. The scanner captures the information perfectly every time as long as you hear it beep.
  • Spreadsheets: This thing makes updating a spreadsheet of ISBNs an absolute breeze.
  • Organization: I can finally keep track of which books I’ve loaned out, which ones I want to buy, and which ones are hiding in that stack in the corner.

Personally, I’ve been using librarything.com to track my budding book collection, but there are a few other alternatives like libib.com, CLZ and GoodReads. And I’m considering using this for more use-cases, like board games, electronics, etc.

What do you think? Was this a waste of money or do you think it’s a good idea?

]]>Mixing CEL and Protobuf for Funhttps://kmcd.dev/posts/mixing-cel-and-protobuf-for-fun/Tue, 17 Dec 2024 10:00:00 +0000https://kmcd.dev/posts/mixing-cel-and-protobuf-for-fun/ <p> <img hspace="5" src="https://kmcd.dev/posts/mixing-cel-and-protobuf-for-fun/cover_hu_9ce5b419b9b65439.webp" /> </p> Protobufs offer a structured approach to data definition, but testing gRPC services built with them can be a hurdle. By leveraging FauxRPC and CEL, you can accelerate development cycles, enhance test coverage, and ensure the reliability of your microservices.

Scaling Protobuf Testing

gRPC has emerged as a powerful framework for building efficient and scalable microservices, enabling seamless communication across diverse technologies. Its language-agnostic nature, powered by Protobuf, allows services written in Go, Java, Python, or any other supported language to interact effortlessly.

However, while gRPC excels at interoperability, generating test data that conforms to Protobuf specifications can still be a hurdle and most tools focus on testing specific languages. Imagine this: you’re building a gRPC service in Go, but your clients might be written in Python, Java, or even C++. Ensuring your tests cover a wide range of scenarios with valid Protobuf messages for all these languages can become quite cumbersome. Now consider that each language has its own tooling for generating and using stub data. While this probably works for most language, I feel like this fragmentation of the ecosystem goes against the mission of protobuf and gRPC. Instead, I think a better approach is a language agnostic approach.

This is where FauxRPC steps in, offering a streamlined approach to gRPC testing. Instead of crafting individual Protobuf responses for each test case, FauxRPC empowers you to define gRPC services that automatically generate realistic fake data for every response. This means you can effortlessly create a wide array of scenarios, simulate various data conditions, and thoroughly exercise your gRPC clients without the overhead of manually constructing complex Protobuf messages. However, this does fall short when you want to test specific test scenarios. That’s where FauxRPC stubs come in.

The Pet Store Example

For the example, we are using an example protobuf file from vanguard-go for this demonstration.

Creating a basic stub

First let’s create a stub file at get-pets.yaml to stub the response to the GetPetByID method.

---
stubs:
- id: get-pets-by-id-id-1
  target: io.swagger.petstore.v2.PetService/GetPetByID
  content:
    id: 1
    category:
      id: 1
      name: cat
    name: Whiskers
    photo_urls:
    - https://cataas.com/cat
    tags:
    - id: 1
      name: cute
    - id: 2
      name: kid-friendly
    status: available

This is a simple static stub. And will be returned any time GetPetByID is called.

Now start the FauxRPC server:

$ buf build ssh://git@github.com/connectrpc/vanguard-go.git -o petstore.binpb --path internal/examples/pets/internal/proto/io/swagger/petstore/v2
$ fauxrpc run --schema=petstore.binpb --only-stubs --stubs=get-pets.yaml

Notice that I actually ran two commands: the first one uses buf build to build a buf image using protobuf schema defined in the vanguard-go repo. The other actually starts the server. Let’s break down the options used real quick:

  • --schema=petstore.binpb: tells FauxRPC about the schema. We use the buf image from the previous command for this.
  • --only-stubs: tells FauxRPC to only use pre-defined stubs and to avoid generating random data when there are no relevant stubs defined.
  • --stubs=get-pets.yaml: tells FauxRPC to load stubs from the get-pets.yaml file from above. This option can be specified multiple times and if you give it a directory it will recursively look for stub files to use.

Now, when I hit the GetPetByID method, I now get Whiskers back.

$ buf curl --http2-prior-knowledge -d '{"pet_id": "1"}' http://127.0.0.1:6660/io.swagger.petstore.v2.PetService/GetPetByID
{
  "id": "1",
  "category": {
    "id": "1",
    "name": "cat"
  },
  "name": "Whiskers",
  "photoUrls": [
    "https://cataas.com/cat"
  ],
  "tags": [
    {
      "id": "1",
      "name": "cute"
    },
    {
      "id": "2",
      "name": "kid-friendly"
    }
  ],
  "status": "available"
}

If I were to add more stub entries, the response will be random amongst the applicable set of stubs. This is fine for some situations, but this also falls flat when you want to set up more complex test scenarios.

Let’s Improve This

Enter CEL (Common Expression Language), a powerful and versatile expression language. With the latest release, FauxRPC now supports using CEL to define Protobuf messages. This unlocks a whole new level of conciseness, flexibility, and readability for your gRPC tests. Imagine creating dynamic messages, generating test data on the fly, and expressing complex scenarios with ease – all within your familiar Go environment.

So what does this mean? Well, there’s three new attributes for FauxRPC stubs: active_if priority and cel_content. Let’s improve upon the stubs that we defined above:

---
stubs:
- id: get-pets-by-id-id-1
  target: io.swagger.petstore.v2.PetService/GetPetByID
  active_if: req.pet_id == 1
  priority: 100
  content:
    id: 1
    category:
      id: 1
      name: cat
    name: Whiskers
    photo_urls:
    - https://cataas.com/cat
    tags:
    - id: 1
      name: cute
    - id: 2
      name: kid-friendly
    status: available
- id: get-pets-by-id-default
  target: io.swagger.petstore.v2.PetService/GetPetByID
  cel_content: |
    {
        'id': req.pet_id,
        'category': {'id': gen, 'name': 'gen'},
        'name': gen,
        'photo_urls': [gen, gen],
        'tags': [{'id': gen, 'name': gen}],
        'status': gen
    }

Well, the first entry now has two new lines: active_if: req.pet_id == 1 and priority: 100. The active_if property lets us define an expression to decide if this stub should be used or not. In this case, if the request has pet_id == 1 then we will use this stub. The second option sets this stub to the highest priority so it will be considered first.

Next, we added a new stub that uses cel_contents. This new field defines a CEL expression to generate a response for us. In this case we use req.pet_id from the request. This demonstrates You can reference the request message in order to craft more believable response messages. For every other field we generate random responses using the special gen value. Using gen will use the field descriptor to decide what kinds of data should be generated.

Here’s what it looks like to call this metheod with a pet_id set to something other than 2:

$ buf curl --http2-prior-knowledge -d '{"pet_id": "2"}' http://127.0.0.1:6660/io.swagger.petstore.v2.PetService/GetPetByID
{
  "id": "2",
  "category": {
    "id": "6775158014153383345",
    "name": gen
  },
  "name": "Mohammad",
  "photoUrls": [
    "https://picsum.photos/400",
    "https://picsum.photos/400"
  ],
  "tags": [
    {
      "id": "3433292194552134769",
      "name": "Chad"
    }
  ],
  "status": "deleted"
}

As you can see, this makes stubs much more dynamic. The output can be based on the input so at the very least the id field can match. Let’s delve deeper into how CEL can be used to create more sophisticated and dynamic test scenarios.

Conditional Logic

Imagine a scenario where you want to return different responses based on specific conditions in the request. You could use CEL to define these conditions:

- id: get-pets-by-id-conditional
  target: io.swagger.petstore.v2.PetService/GetPetByID
  cel_content: |
    {
        'id': req.pet_id,
        'category': {'id': 1, 'name': 'cat'},
        'name': 'Whiskers',
        'photoUrls': ['https://cataas.com/cat'],
        'tags': [{'id': 1, 'name': 'cute'}, {'id': 2, 'name': 'kid-friendly'}],
        'status': 'available'
    }å
    if req.pet_id == 1 else {
        'id': req.pet_id,
        'category': {'id': 2, 'name': 'dog'},
        'name': 'Buddy',
        'photoUrls': ['https://dog.ceo/api/breeds/image/random'],
        'tags': [{'id': 3, 'name': 'friendly'}, {'id': 4, 'name': 'playful'}],
        'status': 'pending'
    }

In this example, if the pet_id is 1, the first response is returned. Otherwise, the second response is returned.

Dynamic Data Generation

CEL can be used to generate dynamic data based on various factors, such as integers, usernames, sentences, random numbers, or values from the request:

- id: get-pets-by-id-dynamic
  target: io.swagger.petstore.v2.PetService/GetPetByID
  cel_content: |
    {
        'id': req.pet_id,
        'category': [{'id': 1, 'name': 'cat'}, {'id': 2, 'name': 'dog'}][fake_int() % 2],
        'name': ['Mr', 'Ms'][fake_int() % 2] + ' ' + fake_first_name(),
        'photoUrls': ['https://picsum.photos/200'],
        'tags': [{'id': gen, 'name': gen}],
        'status': ['available', 'pending', 'sold'][fake_int() % 3]
    }
  • id: This represents the unique identifier of a pet. It takes its value directly from req.pet_id, which is pulled the pet ID from the incoming request.
  • category: This defines the pet’s category. fake_int() % 2 is used to randomly select one of the categories since the result of the modulo operation will be either 0 or 1.
  • name: This generates a random pet name. It combines ‘Mr’ or ‘Ms’ (chosen randomly using fake_int() % 2) with a fake first name generated by the fake_first_name() function.
  • photoUrls: This field is meant to hold an array of URLs pointing to pet photos. In this case, it’s a single-element array with a placeholder URL from https://picsum.photos/200 (which generates a random image).
  • tags: This defines an array of tags associated with the pet. gen is used to generate a random (but type appropriate) value for a single tag’s ID and name fields.
  • status: This indicates the pet’s current status. It randomly selects one of three possible values (‘available’, ‘pending’, or ‘sold’) using fake_int() % 3.

All of the functions starting with fake_ are provided by a new FauxRPC package, celfakeit. This library exposes many functions from gofakeit as CEL functions.

Recap

Since I’ve been making a lot of changes and improvements, it seems like a good idea to recap how FauxRPC works and how it’s different from other similar tools.

Conclusion

In conclusion, the combination of Protobuf and CEL offers a powerful and flexible approach to testing gRPC services. By leveraging CEL’s expressive power, developers can define dynamic and realistic test data, significantly enhancing the efficiency and effectiveness of their testing efforts.

FauxRPC, with the integration of CEL, simplifies the process of generating complex Protobuf messages, allowing developers to focus on writing robust and reliable gRPC services. By embracing this innovative approach, teams can accelerate their development cycles and deliver high-quality gRPC applications with confidence.

… and don’t forget to star the repo on GitHub. It helps more than you know!

]]>FauxRPC and Protovalidatehttps://kmcd.dev/posts/fauxrpc-protovalidate/Tue, 12 Nov 2024 10:00:00 +0000https://kmcd.dev/posts/fauxrpc-protovalidate/ <p> <img hspace="5" src="https://kmcd.dev/posts/fauxrpc-protovalidate/cover_hu_cc063547c579daa5.webp" /> </p> FauxRPC, a tool for generating fake gRPC servers, now integrates with protovalidate, which lets you define validation rules in your Protobuf definitions. Now every request processed by FauxRPC will be automatically validated against your protovalidate rules. Not only will you get high quality data validation in your application, but now you can have the same validation before you even write your application logic! Let’s walk through how this new feature works.

How it works

First, define your validation rules using protovalidate’s constraint annotations within your Protobuf definitions. For guidance, you should reference the protovalidate documentation. Fauxrpc will take care of the rest, automatically validating each request against these rules. If a request fails validation, a detailed error message will be returned, guiding you toward quickly fixing the issue.

For this example, we’re going to use a simple service where you can call with a name and it will return a greeting. Here’s what that would look like in protobuf:

greet.proto

syntax = "proto3";

package greet.v1;

import "buf/validate/validate.proto";

message GreetRequest {
  string name = 1 [(buf.validate.field).string = {min_len: 3, max_len: 20}];
}

message GreetResponse {
  string greeting = 1 [(buf.validate.field).string.example = "Hello, user!"];
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

The buf.validate.field annotations are from protovalidate. The section (buf.validate.field).string = {min_len: 3, max_len: 20} ensures that the name field is between 3 and 20 characters. (buf.validate.field).string.example = "Hello, user!" is declaring what an example response might look like. This mostly serves as documentation but it also influences other tools, including protoc-gen-connect-openapi and, now, FauxRPC! We’ll see how FauxRPC uses both of these constraints in a second. First, let’s start up the FauxRPC server.

To use protovalidate with FauxRPC, we need to leverage another tool that’s extremely helpful for working with protobuf definitions. This is because we used protovalidate, a dependency. Usually dependencies are hard to deal with in protobuf because the default protobuf tooling just doesn’t manage dependencies at all. However, the Buf CLI and the BSR can help us here.

First, create a new buf.yaml file in the same directory as greet.proto with the contents:

version: v2
deps:
  - buf.build/bufbuild/protovalidate

Now we’re just a few commands away from having a mock service running:

# Get Buf CLI to pull our new dependency
$ buf dep update
# Build our protobuf file (and dependencies) into a protobuf "image": https://buf.build/docs/build/overview/
$ buf build . -o greet.binpb
# Start FauxRPC with this image
$ fauxrpc run --schema=greet.binpb

Now let’s try some requests against this new service:

$ buf curl --http2-prior-knowledge -d '{}' http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
   "code": "invalid_argument",
   "message": "validation error:\n - name: value length must be at least 3 characters [string.min_len]",
   "details": [
      {
         "type": "buf.validate.Violations",
         "value": "CkIKBG5hbWUSDnN0cmluZy5taW5fbGVuGip2YWx1ZSBsZW5ndGggbXVzdCBiZSBhdCBsZWFzdCAzIGNoYXJhY3RlcnM",
         "debug": {
            "violations": [
               {
                  "fieldPath": "name",
                  "constraintId": "string.min_len",
                  "message": "value length must be at least 3 characters"
               }
            ]
         }
      }
   ]
}

Oh, duh! We hit the length constraint we added for the name field, so our empty object {} isn’t good enough anymore. We need a name and it needs to have between 3 and 20 characters. Let’s try once more:

$ buf curl --http2-prior-knowledge -d '{"name": "Bob"}' http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
  "greeting": "Hello, user!"
}

Great, we received a response! And the response is populated using our (buf.validate.field).string.example annotation. This will allow us to stand up a fake service with a few simple commands that has request validation built in and will use the constraints to make realistic fake data.

This shows how protovalidate uses protovalidate constraints both for input validation and fake data generation.

On a related note, I am personally excited to eventually have Typescript Support for protovalidate. That would allow us to use protovalidate for both clean frontend UX and robust server side validation. In my eyes, this would completely solve the problem of having duplicate and inconsistent validation rules between the frontend and backend. Backend needs these rules to ensure system reliability and consistency. Frontend needs these rules for better UX.

Benefits for you

FauxRPC and protobuf synergizes well with the model-driven API design with protobuf. From a single API definition you have strongly typed definitions that has support for many programming languages, powerful validation constraints, examples of what each field looks like. And FauxRPC lets you experiment with all of that before writing a single line of application code. This means:

  • Reduced development time: Spend less time debugging data issues and more time building amazing features.
  • Increased confidence: Trust that your RPCs are handling data correctly, leading to more stable and reliable applications.
  • Improved collaboration: Make it easier for teams to work together by ensuring everyone adheres to the same data standards.

Ready to give it a try?

Ready to experience the power of FauxRPC and protovalidate?

For reference, all of the code in the article is available here.

]]>The Call of the Monolithic Codebasehttps://kmcd.dev/posts/call-of-the-monolith-codebase/Tue, 22 Oct 2024 10:00:00 +0000https://kmcd.dev/posts/call-of-the-monolith-codebase/ <p> <img hspace="5" src="https://kmcd.dev/posts/call-of-the-monolith-codebase/cover_hu_db19af5e196aee10.webp" /> </p> Are monoliths cool again? As the microservices landscape continues to evolve, a sobering reality is setting in for many organizations: managing many microservice repositories has become a significant hurdle. In the pursuit of scalability and flexibility offered by microservices, we may have inadvertently sacrificed maintainability, but now the pendulum is swinging back towards simpler code management while retaining the ability to deploy at scale. For organizations already entrenched in the microservices world, it’s time to reconsider an old friend: the monolithic codebase.

Devs at these organizations might have noticed that as the number of services grow, so does the complexity of managing multiple codebases, each with its own repository, build pipelines, etc. This explosion of the number of repos introduces significant overhead, including:

  • Increased Operational Complexity: More repositories mean more CI/CD pipelines, dependency management, and security audits.
  • Higher Cognitive Load for Developers: Context switching between numerous repositories can decrease productivity. “Oh, this repo uses the old docker-compose scheme for testing, we should really remember update that.”
  • Unified Change Management Challenges: Implementing organization-wide changes, such as dependency updates or coding standards, becomes a monumental task.

Scenario: Updating log4j across 20 Microservices Repositories

To further illustrate the challenges of managing multiple repositories in a microservices architecture, let’s delve into the process of updating a commonly used library across all microservice repos.

  • Discovery:
    • Identify the need to update log4j due to a security vulnerability. Maybe you hear about this from dependabot spamming you, maybe another security scanner picked up that your repo is using a log4j, maybe your devs see it at the top of hacker news or maybe you just have a grumpy security operations team reviewing every CVE and deciding if anything the company uses is vulnerable. Hopfully it’s all of the above if the vulnerability is really severe.
    • Determine the versions currently in use across all 20 microservices.
  • Planning:
    • Coordinate with multiple team leads to schedule the update. Depending on how “mono” the “monolith” is, you may have several teams sharing the same codebase.
    • Ensure compatibility with each microservice’s specific dependencies.
  • Execution:
    1. Update Dependency:
      • Modify the pom.xml (Maven) or build.gradle (Gradle) in each of the 20 repositories.
      • Commit changes with a consistent, informative message (e.g., “Security: Update log4j to v2.17.1”).
    2. Verify and Test:
      • Run automated tests for each microservice to catch any breaking changes.
      • Perform manual testing where automated coverage is insufficient.
    3. Deployment:
      • Trigger CI/CD pipelines for each microservice to deploy the updated library.
  • Verification and Monitoring:
    • Confirm the update has been successfully deployed across all microservices.
    • Monitor application logs for any issues related to the log4j update.

Challenges Encountered

Now this is the process if everything goes well. There’s several different points where this process can increase the complexity of our simple task:

  • Version Inconsistencies: Different microservices were using various versions of log4j, requiring tailored updates. Maybe some repos were on a previous major version and you’ll need to update a lot of code in order to upgrade to the latest version. Or you may consider backporting the fix and maintaining patches or a fork of this repository.
  • Dependency Conflicts: Updates in some microservices revealed hidden dependencies on outdated libraries. Maybe the latest version of log4j pulls in a new version of apache commons. Maybe another dependency, for some reason, doesn’t work with the latest version. This kind of thing does happen and can be a real pain to resolve.

With all of this tedious work you should remember that this type of update is the simplest kind of change to make. Now just imagine trying to be proactive and keep all of your services up-to-date and to not let the least loved ones get left behind.

Monolithic Codebase Alternative

Because of the overhead in this process, some developers have leaned towards consolidating most of the code that powers microservices into a single repo. We refer to this type of code repository as a “monorepo”. In our log4j scenario, it would help with several of these steps.

  • Single Update Point: Modify the log4j dependency in one place, within the unified monorepo.
  • Simplified Testing and Deployment: Leverage the monorepo’s integrated testing and deployment mechanisms. If this is a commonly used library, more automated tests will need to be ran, but these are tests that would have ran anyway, but just spread out amongst many different repositories.
  • Reduced Complexity and Resource Usage: Minimize the logistical overhead and focus on verifying the update’s success.

The Allure of the Monolithic Codebase

As demonstrated by the log4j update scenario, managing numerous repositories can be a complex and time-consuming process. A monolithic codebase offers a compelling alternative by simplifying many aspects of development and maintenance. Let’s explore some of its key advantages:

  • Simplified Repository Management: Instead of juggling 20 separate repositories, a monorepo consolidates everything into a single location. This significantly reduces the operational overhead associated with managing multiple codebases, build pipelines, and deployment processes. Imagine the time saved by not having to update log4j in 20 different places!
  • Unified Codebase Governance: Implementing organization-wide changes, like security updates or coding standard enhancements, becomes significantly easier with a monorepo. A single update propagates across all services, ensuring consistency and reducing the risk of inconsistencies or oversights that can occur when managing multiple repositories.
  • Improved Developer Experience: Developers can navigate the entire codebase more easily, reducing the cognitive load associated with context switching between different repositories. No more remembering which repo uses a specific Docker Compose scheme or wondering where a particular service is located. Everything is readily accessible in one place.
  • Enhanced Code Reusability: With all services residing in a single repository, code sharing and reuse become more straightforward. Developers can easily identify and leverage existing components, reducing code duplication and promoting consistency across the application. This can be particularly beneficial for common libraries like log4j, where a single, well-maintained implementation can be used by all services.

Furthermore, a monorepo can significantly streamline testing and deployment processes:

  • Simplified Testing: Automated tests can be run across all affected services with a single command, ensuring that changes don’t introduce unexpected regressions. In the log4j scenario, this means verifying the update’s impact on all services simultaneously, rather than testing each microservice individually.
  • Streamlined Deployment: Atomic deployments become possible, guaranteeing that all services are updated consistently. This eliminates the risk of partial deployments or version mismatches that can occur in a microservices architecture.
  • Easier Rollbacks: If an issue arises after deployment, rolling back to a previous version becomes simpler with a monorepo. A single rollback operation can revert all services to a known good state, minimizing downtime and disruption. With microservices that all have different versions, it’s hard to know what a good combination of versions for each service actually is.

Note that while the codebase may be monolithic, many developer teams are still deploying as microservices. This promises the best of both worlds: the simplified and streamlined experience of a single repo with the scalability properties of microservices.

However, it’s important to acknowledge that monolithic codebases are not without their challenges.

Tooling for a Successful Monolithic Codebase

To leverage the benefits of a monolithic codebase while avoiding the pitfalls of tight coupling and decreased maintainability, consider the following tools:

  • Bazel: A build tool and ecosystem that supports large, multi-language monorepos; created by Google.
  • Pants: A build system for monorepos, focusing on performance and scalability.
  • Buck: A fast and scalable build tool developed by Facebook, designed to handle large, complex monorepos with ease.
  • Service Weaver: Service weaver has a very different take on monorepos. It’s more than just a build tool, and is more of a framework that allows for monorepos with monolithic or microservice deployment. Because it is opinionated, other kinds of build tools aren’t needed as much. Also, this is only supported for Go.
  • Lerna: A popular and mature tool for managing JavaScript monorepos, especially useful for managing multiple packages within a single repository.
  • Turbo: High-performance build system for JavaScript/TypeScript monorepos, known for its speed and efficient caching mechanisms.
  • Git Submodules: While not a dedicated monorepo tool, Git submodules can be used to manage dependencies on other Git repositories within a larger monorepo. However, they can be complex to use and may not be the best choice for all scenarios.
  • Your own tooling: Sometimes building a set of scripts to customize the build process using more traditional tooling may be better.

You should note that this list is by no means complete. There are a lot of options (I think I found around 10 Javascript tools before giving up my research). I also left out several tools don’t have open source tooling or rely exclusively on paid hosting services.

The choice of tooling will highly depend on the language/languages your codebases are hosted in. Bazel excels at seamlessly handling polyglot repos no matter the language. Others, like Pants only support a handful of languages and if you’re using another language you may need to use multiple tools to get the job done. In general though, the fewer languages the easier it is to integrate because there are always language-specific idiosyncrasies that pop up, even when using tools like Bazel which try to hide those differences from you. Choose what makes sense for your team… and maybe try to get as far as you can using standard tooling before introducing extra build tools.

Drawbacks of Monolithic Codebases

You should consider some of the real drawbacks when working with a monolithic codebase.

  • Increased Build Times: As the codebase grows, build times can become significantly longer, especially if not properly optimized. This can slow down development and deployment cycles.
  • Merge Conflicts: With many developers working on the same codebase, merge conflicts can become more frequent and complex, requiring more time and effort to resolve.
  • Tight Coupling: If not carefully managed, a monorepo can lead to tight coupling between different parts of the application, making it harder to modify or refactor individual components without affecting others.
  • Learning Curve: For developers accustomed to working with microservices and separate repositories, there can be a learning curve associated with navigating and understanding a large, monolithic codebase.
  • Access Control: Managing access control and permissions can be more challenging in a monorepo, as developers potentially have access to the entire codebase, even if they only work on a specific part of it.

Bazel-Specific Drawbacks

Since Bazel is a popular choice for managing monorepos, it’s worth noting some of its specific drawbacks:

  • Steep Learning Curve: Bazel has a reputation for having a steep learning curve, especially for developers unfamiliar with build systems like Make or CMake. Its configuration language, Starlark, can also take time to master.
  • Limited IDE Support: While IDE support for Bazel is improving, it’s still not as comprehensive as for other build systems. This can make debugging and code navigation more challenging.
  • Performance Issues: While Bazel is generally performant, it can sometimes experience performance issues, especially with very large codebases or complex build configurations.
  • Community Support: While Bazel has a growing community, it’s still smaller than the communities for other build systems like Maven, Gradle, or default tooling from languages like Rust, Go, etc. This can make it harder to find help or resources when encountering issues.

Mitigating the Drawbacks

It’s important to note that many of these drawbacks can be mitigated with careful planning and the right tools and practices:

  • Modular Design: Employing modular design principles and clear code organization can help minimize tight coupling and improve build times.
  • Code Ownership: Establish clear code ownership and access control policies to manage permissions and prevent unauthorized modifications.
  • Continuous Integration and Continuous Deployment (CI/CD): Implement robust CI/CD pipelines to automate builds, tests, and deployments, helping to identify and address issues early on.
  • Invest in Training: Provide adequate training and support to help developers learn Bazel and its best practices.
  • Monitoring and Optimization: Regularly monitor build times and Bazel’s performance to identify and address any bottlenecks. Many of these build systems have more advanced features like remote caching and remote execution, which can be used to greatly reduce build times.

By acknowledging these potential drawbacks and taking steps to mitigate them, you can increase the likelihood of successfully adopting a monolithic codebase with Bazel.

The Path Forward

While a monolithic codebase can simplify management and development, it’s crucial to strike a balance between unity and the autonomy that microservices provide. Consider the following:

  • Start Small: If migrating from a microservices architecture, begin by consolidating closely related services.
  • Establish Clear Boundaries: Use tooling and guidelines to maintain logical separations within the monorepo.
  • Monitor and Adapt: Continuously evaluate the effectiveness of your monolithic approach and make adjustments as needed.

As we’ve explored the trade-offs between separate repos and monolithic codebases, it’s clear that the latter can offer a more streamlined development experience, simplified management, and reduced overhead. However, before embracing a monolithic codebase, it’s essential to carefully consider the implications of this approach on your project’s specific needs.

]]>FauxRPC + Test Containershttps://kmcd.dev/posts/fauxrpc-testcontainers/Tue, 15 Oct 2024 00:00:00 +0000https://kmcd.dev/posts/fauxrpc-testcontainers/ <p> <img hspace="5" src="https://kmcd.dev/posts/fauxrpc-testcontainers/cover_hu_34ecac18b5af762a.webp" /> </p> Effortless gRPC Mocking in Go Testing gRPC services can be tricky. You often need a real server running, which can introduce complexity and slow down your tests. Enter FauxRPC + Testcontainers, and small Go package that simplifies gRPC mocking.

To address challenges with testing while using gRPC services, we can leverage the power of Testcontainers, a library that lets you run throwaway, lightweight instances of common databases, web browsers, or any other application that can run in a Docker container. This allows you to easily integrate these dependencies into your automated tests, providing a consistent and reliable testing environment. By using Testcontainers, you can ensure that your tests are always running against a known and controlled version of your dependencies, avoiding inconsistencies and unexpected behavior while also simplifying test setup/teardown.

While Testcontainers provides the infrastructure, FauxRPC takes care of the mocking itself. FauxRPC is a tool that generates fake gRPC, gRPC-Web, Connect, and REST servers from your Protobuf definitions. By combining it with Testcontainers, you gain a lightweight, isolated environment for testing your gRPC clients without relying on a real server implementation. I’ve made a package to make this simpler using Go but the same could be done for other languages that Testcontainers supports.

Show by Example

1. Setting up the Container

container, err := fauxrpctestcontainers.Run(ctx, "docker.io/sudorandom/fauxrpc:latest")
// ... error handling ...
t.Cleanup(func() { container.Terminate(context.Background()) })

This snippet starts a FauxRPC container using the fauxrpctestcontainers.Run function. The t.Cleanup function ensures the container is terminated after the test, keeping your testing environment clean.

2. Registering the Protobuf Definition

container.MustAddFileDescriptor(ctx, elizav1.File_connectrpc_eliza_v1_eliza_proto)

You register your Protobuf file descriptor with the container. This lets FauxRPC understand the structure of your gRPC service. Now you have a fully functional FauxRPC service that mimics the services in the file descriptor that you gave it. The data is all randomly generated. Now let’s test it a bit.

3. Making gRPC Calls

baseURL := container.MustBaseURL(ctx)
elizaClient := elizav1connect.NewElizaServiceClient(http.DefaultClient, baseURL)
resp, err := elizaClient.Say(ctx, connect.NewRequest(&elizav1.SayRequest{
    Sentence: "testing!",
}))
// ... error handling and assertions ...

This code is getting base URL of the FauxRPC server running in the container and creating a gRPC client (using ConnectRPC). ConnectRPC isn’t a requirement. You can use grpc-go instead. With this client, you can make calls to your gRPC service as you would in a real environment. In this setup, FauxRPC automatically generates responses based on your Protobuf definitions. Here you would normally have some application logic that you want to test so this code might live elsewhere. The randomly generated data might work in a few scenarios but in order to test

4. Defining Stub Responses

For more control over the responses you can define stubs. This allows you to simulate specific scenarios and test how your client handles different responses.

container.MustAddStub(ctx, "connectrpc.eliza.v1.ElizaService/Say", &elizav1.SayResponse{
    Sentence: "I am setting this text!",
})

The power of schemas and mocking

This is basically it. In these examples, I showed how you can:

  1. Stand up an empty FauxRPC service
  2. Populate this server with some Protobuf schema
  3. Connect and use this service
  4. Set stub responses

See full examples in the FauxRPC repo.

Benefits of using FauxRPC+Testcontainers

As demonstrated in the examples above, this approach simplifies gRPC testing by:

  • Simplified Testing: No need to set up a real gRPC server for testing.
  • Isolated Environment: Each test runs in its own container, preventing conflicts and ensuring consistency.
  • Increased Speed: Tests run faster due to the lightweight nature of containers.
  • Improved Control: Stubbing allows you to simulate various scenarios and edge cases.

This package makes gRPC testing in Go much easier and more efficient. Give it a try and let me know what you think!

Alternatives

In Go (and many targets for gRPC), you will get an interface that you can use to generate mock clients. Using the mock client you can usually set responses and assert on data from the request. This is a fair critique, and I feel like this strategy of testing could get you pretty far. However, there are a few reasons that FauxRPC+testcontainers is better.

First, using traditional mocking techniques will prevent you from testing your middleware. Maybe you have middleware that modifies the actual message in certain cases, returns an error, or performs some extra validation or accounting. With the FauxRPC+Testcontainers, you get to exercise the middleware code because you’re talking to a real gRPC service.

In addition to that, maintaining and updating mock clients can be tedious as the gRPC API evolves. FauxRPC avoids this step by being dynamically configurable with protobuf descriptors.

Also, I like that this approach of using FauxRPC is language agnostic. Sure, the little library that makes it easier to use is written specifically in Go, but this code is very trivial to write for other languages.

Ultimately, the choice between mocking strategies depends on your specific needs and priorities.

What’s Next

Excited about the possibilities of FauxRPC Testcontainers? There’s more to come! FauxRPC is still under active development and there’s a lot more on the horizon! Here are a few features I’m exploring:

  • Rules using CEL: Fine-grained control over stub behavior using Common Expression Language (CEL) to define complex matching conditions and response generation logic. This will enable more dynamic and flexible stubbing scenarios. Imaging having a rule saying: req.Name == "Bob" then return a specific stub user.
  • Request Logging: Detailed logging of requests and responses to facilitate debugging and troubleshooting during test execution.
  • Improved documentation: I’m currently working on a refresh of fauxrpc.com that includes these new features.

Have an idea for a new feature or a suggestion for improvement? We’d love to hear from you! Open an issue on the GitHub repository to share your thoughts or contribute to the project.

References

]]>Self-Documenting Connect Serviceshttps://kmcd.dev/posts/self-documenting-connect-services/Wed, 25 Sep 2024 10:00:00 +0000https://kmcd.dev/posts/self-documenting-connect-services/ <p> <img hspace="5" src="https://kmcd.dev/posts/self-documenting-connect-services/cover_hu_6d1636dac10f3e09.webp" /> </p> gRPC can be pretty, too. As some of you may know, I’ve created a plugin for protoc called protoc-gen-connect-openapi. This plugin converts protobuf files into OpenAPI specifications for the Connect protocol. This protocol is very similar to gRPC but for unary RPCs it follows many more traditions that you’d expect from an HTTP-based API, like using HTTP status codes appropriately, using the normal Content-Encoding header to specify compression and avoiding putting extra framing inside of the body. Because of this, we can document it more readily with other specifications like OpenAPI.

For more on the plugin itself, refer to my older post that introduces protoc-gen-connect-openapi or the github repo which has some more updates.

This post is about a new way to use this functionality, from a Go library. This Go API provides a simpler interface to generate OpenAPI from protobuf descriptors. You see, protoc plugins accept a *pluginpb.CodeGeneratorRequest and return a *pluginpb.CodeGeneratorResponse. The request type, in particular, is hard to use from Go. You have to encode every option you want to use into a single string. This is fine for a CLI but it’s not very friendly for a Go library. But now that I made this library it is much easier to generate OpenAPI specs anywhere you run Go. Let’s look at some examples.

Generating OpenAPI

First, let’s see how we can use this plugin to generate OpenAPI YAML from your protobuf definitions:

openapiBody, _ := converter.GenerateSingle(
    converter.WithGlobal(),
    converter.WithBaseOpenAPI([]byte(`
openapi: 3.1.0
info:
  title: OpenAPI Documentation of gRPC Services
  description: This is documentation that was generated from [protoc-gen-connect-openapi](https://github.com/sudorandom/protoc-gen-connect-openapi).
  version: 0.1.2
`)))
fmt.Println(string(openapiBody))

With a few short lines, you now have OpenAPI YAML representation of your ConnectRPC services.

openapi: 3.1.0
info:
  title: OpenAPI Documentation of gRPC Services
  description: This is documentation that was generated from [protoc-gen-connect-openapi](https://github.com/sudorandom/protoc-gen-connect-openapi).
paths:
  /connectrpc.eliza.v1.ElizaService/Say:
    post:
      tags:
        - connectrpc.eliza.v1.ElizaService
      summary: Say
      operationId: connectrpc.eliza.v1.ElizaService.Say
      parameters:
        - name: Connect-Protocol-Version
          in: header
          required: true
          schema:
            $ref: '#/components/schemas/connect-protocol-version'
        - name: Connect-Timeout-Ms
          in: header
          schema:
            $ref: '#/components/schemas/connect-timeout-header'
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/connectrpc.eliza.v1.SayRequest'
        required: true
      responses:
        default:
          description: Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/connect.error'
        "200":
          description: Success
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/connectrpc.eliza.v1.SayResponse'
  /connectrpc.eliza.v1.ElizaService/Converse: {}
  /connectrpc.eliza.v1.ElizaService/Introduce: {}
components:
  schemas:
    connectrpc.eliza.v1.SayRequest:
      type: object
      properties:
        sentence:
          type: string
          title: sentence
      title: SayRequest
      additionalProperties: false
    connectrpc.eliza.v1.SayResponse:
      type: object
      properties:
        sentence:
          type: string
          title: sentence
      title: SayResponse
      additionalProperties: false
    connect-protocol-version:
      type: number
      title: Connect-Protocol-Version
      enum:
        - 1
      description: Define the version of the Connect protocol
      const: 1
    connect-timeout-header:
      type: number
      title: Connect-Timeout-Ms
      description: Define the timeout, in ms
    connect.error:
      type: object
      properties:
        code:
          type: string
          examples:
            - CodeNotFound
          enum:
            - CodeCanceled
            - CodeUnknown
            - CodeInvalidArgument
            - CodeDeadlineExceeded
            - CodeNotFound
            - CodeAlreadyExists
            - CodePermissionDenied
            - CodeResourceExhausted
            - CodeFailedPrecondition
            - CodeAborted
            - CodeOutOfRange
            - CodeInternal
            - CodeUnavailable
            - CodeDataLoss
            - CodeUnauthenticated
          description: The status code, which should be an enum value of [google.rpc.Code][google.rpc.Code].
        message:
          type: string
          description: A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the [google.rpc.Status.details][google.rpc.Status.details] field, or localized by the client.
        detail:
          $ref: '#/components/schemas/google.protobuf.Any'
      title: Connect Error
      additionalProperties: true
      description: 'Error type returned by Connect: https://connectrpc.com/docs/go/errors/#http-representation'
    google.protobuf.Any:
      type: object
      properties:
        type:
          type: string
        value:
          type: string
          format: binary
        debug:
          type: object
          additionalProperties: true
      additionalProperties: true
      description: Contains an arbitrary serialized message along with a @type that describes the type of the serialized message.
security: []
tags:
  - name: connectrpc.eliza.v1.ElizaService

This file is truncated to only show a single endpoint and related types. To see the full file, click here. The converter.WithGlobal() option uses the global protobuf registry as the source. If you want to use specific (or protobuf file descriptors not in that registry), you can pass in any protoregistry.GlobalFiles value to the converter.WithFiles() option. With converter.WithBaseOpenAPI(), you can specify a base OpenAPI spec that will be used as the basis for the generated one. Here you can add a description, version, security schemes, link to other documentation, etc.

But what’s the practical use of this YAML?

Show the world what you can do

With a few additional lines of code, we can leverage one of the numerous OpenAPI documentation visualization tools to transform this YAML into a visually appealing and interactive web page. Here’s an example using Elements from Spotlight:

var tmplElements = template.Must(template.New("name").Parse(`<!doctype html>
<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
	<title>OpenAPI Documentation</title>
	<script src="https://unpkg.com/@stoplight/elements@8.3.4/web-components.min.js"></script>
	<link rel="stylesheet" href="https://unpkg.com/@stoplight/elements@8.3.4/styles.min.css">
	</head>
	<body>

	<elements-api
		id="docs"
		router="hash"
		layout="sidebar"
	/>
	<script>
	(async () => {
		const docs = document.getElementById('docs');
		docs.apiDescriptionDocument = atob("{{ .DocumentBase64 }}");
	})();
	</script>

	</body>
</html>`))

func main() {
	mux := http.NewServeMux()
	mux.Handle(elizav1connect.NewElizaServiceHandler(&elizav1connect.UnimplementedElizaServiceHandler{}))
	openapiBody, err := converter.GenerateSingle(
		converter.WithGlobal(),
		converter.WithContentTypes(
			"json",
			"proto",
		),
		converter.WithStreaming(true),
        converter.WithAllowGET(true),
		converter.WithBaseOpenAPI([]byte(`
openapi: 3.1.0
info:
  title: OpenAPI Documentation of gRPC Services
  description: This is documentation that was generated from [protoc-gen-connect-openapi](https://github.com/sudorandom/protoc-gen-connect-openapi).
`)))
	if err != nil {
		log.Fatalf("err: %s", err)
	}
	generationTime := time.Now()

	mux.Handle("GET /openapi.html", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if err := tmplElements.Execute(w, struct{ DocumentBase64 string }{
			DocumentBase64: base64.StdEncoding.EncodeToString(openapiBody),
		}); err != nil {
			slog.Error("rendering_template", "error", err)
		}
	}))
	mux.Handle("GET /openapi.yaml", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		http.ServeContent(w, r, "openapi.yaml", generationTime, bytes.NewReader(openapiBody))
	}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on http://%s", addr)
	log.Printf("OpenAPI Doc Page http://%s/openapi.html", addr)
	log.Printf("OpenAPI Spec http://%s/openapi.yaml", addr)
	srv := http.Server{
		Addr:    addr,
		Handler: mux,
	}
	if err := srv.ListenAndServe(); err != nil {
		log.Fatalf("error: %s", err)
	}
}

And here’s what it looks like whenever you hit http://127.0.0.1.:6660/openapi.html in a web browser:

To see the demo for yourself, click here!

In this example, we’re using the @stoplight/elements library to render the OpenAPI documentation. The tmplElements template embeds the generated OpenAPI YAML (Base64 encoded) into an HTML page, providing a user-friendly interface to explore your API’s endpoints, request/response structures, and more. Note that we’re using base64 so that none of the YAML characters accidentally escape the javascript string and mess everything up for us. You can also have the script load the OpenAPI spec from a URL, which is what many of the examples show.

You may also notice some additional options in this example, like converter.WithContentTypes(), converter.WithStreaming(true) and converter.WithAllowGET(true). These give you more control over content types, whether you want OpenAPI for streaming calls (which may be complicated to support for OpenAPI) and whether you want to generate documentation for GET requests, that Connect supports if you can set the idempotency_level option to NO_SIDE_EFFECTS. For more information on GET requests and a comprehensive list of available options, refer to the the Connect documentation and the protoc-gen-connect-openapi Go documentation, respectively.

Benefits of Self-Documenting Services

Clear and interactive documentation makes it easier for developers to understand and integrate with your APIs. This reduces the friction between teams and gives you something to point to in case there are questions about the API. Not everyone is fluent in reading protobuf, but HTTP documentation is much friendlier.

By generating documentation from protobufs, it is much easier for documentation to stay in sync with your codebase. Note that adding comments to your protobuf types, fields, services and methods also get carried over to this OpenAPI specification (and the generated protobuf/gRPC/gRPC-Web/Connect source code) so protobuf files can act as a single place to document everything about that service.

Conclusion

By combining the power of protoc-gen-connect-openapi with OpenAPI visualization tools, you can effortlessly generate self-documenting Connect services. This approach streamlines development, fosters collaboration, and empowers developers to consume your APIs effectively.

]]>gRPC Over HTTP/3: Followuphttps://kmcd.dev/posts/grpc-over-http3-followup/Tue, 17 Sep 2024 10:00:00 +0000https://kmcd.dev/posts/grpc-over-http3-followup/

Let's cover some recent updates! ]]>In my previous post, “gRPC Over HTTP/3,” we explored the potential of gRPC with HTTP/3. At that time, some of the pieces were missing and I had to hack on forks of a few repos to make gRPC+HTTP/3 work with Go. The biggest blocker was that the quic-go HTTP/3 implementation didn’t have support for HTTP trailers. But now things have recently changed there and these hacks are no longer needed!

quic-go now supports HTTP Trailers

If you recall, this was a major roadblock for getting gRPC to work over HTTP/3. Trailers are crucial for gRPC’s error handling and status codes, so this was a big deal. This works as of v0.47.0. Here are the related PRs:

As you can see, I’ve contributed most of the work for this!

go get -u github.com/quic-go/quic-go

This now enables HTTP/3 support for ConnectRPC for gRPC, gRPC-Web and Connect for both client and server. While I’d recommend deploying in production with a more established load balancer that supports HTTP/3, quic-go is perfect for experimentation and development. I’ll show you how to set up Go + HTTP/3 (via quic-go) + Connect a bit further down in this article.

Buf’s curl command has a new --http3 flag

That’s right, you can now easily test your gRPC services over HTTP/3 from the command line. This is a fantastic development for quick prototyping, debugging and having a simple tool to call gRPC services. You can use this as of v1.41.0. Here are the related PRs:

Upgrade to the new version today!

Open Source

I’m very happy to have contributed both of these features. Like I said in my previous post about this topic, the Go version of ConnectRPC seemed so close to having full HTTP/3 support in all three protocols: Connect/gRPC-Web and the original gRPC; it just needed trailer support to push gRPC over the finish line. And with other gRPC implementations, like grpc-dotnet, I hope the addition of HTTP/3 to buf curl command can be useful as well.

What does this mean for you?

In short, it means that if you’re working on a gRPC project, it’s now slightly more viable to use HTTP/3 today… in specific contexts. Here’s a recap of the benefits:

  • Faster connections, especially on unreliable mobile connections: HTTP/3’s connection setup is lightning-fast compared to HTTP/2, and it handles flaky networks like a champ. This is a major win for mobile apps and any situation where network conditions aren’t ideal. This can be useful when you’re using gRPC-Web or Connect on the frontend.
  • No more head-of-line blocking: HTTP/3 eliminates this pesky problem that can slow down HTTP/2 in certain scenarios. If your gRPC service handles lots of concurrent streams, you might see an improvement.

Trying it out with ConnectRPC

Here’s an example of starting a HTTP/3 server with ConnectRPC: 

func main() {
	mux := http.NewServeMux()
	// Implementation is only in the full source
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}
	if err := h3srv.ListenAndServeTLS("cert.crt", "cert.key"); err != nil {
		log.Fatalf("error: %s", err)
	}
}

This example uses the HTTP/3 server from quic-go to provide HTTP/3. Now you can test it using buf curl. Here’s an example of using buf curl with gRPC:

$ buf curl --http3 -k \
  --protocol=grpc \
  --schema=buf.build/connectrpc/eliza \
  -d '{"sentence":"Hello, with gRPC+h3"}' \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, with gRPC+h3"
}

With gRPC-Web:

$ buf curl --http3 -k \
  --protocol=grpcweb \
  --schema=buf.build/connectrpc/eliza \
  -d '{"sentence":"Hello, with gRPC-Web+h3"}' \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, with gRPC-Web+h3"
}

With Connect:

$ buf curl --http3 -k \
  --protocol=connect \
  --schema=buf.build/connectrpc/eliza \
  -d '{"sentence":"Hello, with Connect+h3"}' \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, with Connect+h3"
}

Note that if you don’t use the --http3 flag this doesn’t work. That’s because we’ve only started an HTTP/3 server. Using the following code, we can run HTTP/3 alongside HTTP/1.1 and HTTP/2:

func main() {
	mux := http.NewServeMux()
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}

	srv := http.Server{
		Addr:    addr,
		Handler: h2c.NewHandler(mux, &http2.Server{}),
	}

	eg, _ := errgroup.WithContext(context.Background())
	eg.Go(func() error {
		return h3srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	eg.Go(func() error {
		return srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	if err := eg.Wait(); err != nil {
		log.Fatalf("error: %s", err)
	}
}

With this code, you can now connect using any version of HTTP and with gRPC, gRPC-Web or Connect.

$ buf curl -k --protocol=grpc \
  --schema=buf.build/connectrpc/eliza \
  -d '{"sentence":"Hello, with gRPC+h2"}' \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, with gRPC+h2"
}

The compatibility matrix is now all green when using ConnectRPC, with only a single exception (because HTTP/1.1 servers and clients typically don’t support HTTP trailers):

Protocol HTTP/1.1 HTTP/2 HTTP/3
gRPC ❌ ✅ ✅
gRPC-Web ✅ ✅ ✅
Connect ✅ ✅ ✅

See the repo at sudorandom/example-connect-http3 to see the full examples shown here as well as some example client code.

So everything is fast with this, right?

Well, no. HTTP/3 isn’t always a performance win… and actually, today, it may generally be slower or, at best, the same speed as HTTP/2. Part of the cause is that it uses a lot of CPU cycles compared to HTTP/1.1 and HTTP/2. You might be asking: “this awesome protocol that is supposed to make things fast is actually slower? What’s the point?”. This is a good question that’s been answered many times.

QUIC is still mostly implemented in user-space and is lacking the half-century of optimizations that TCP has had. I recently saw this paper which looks to be some decent data regarding actual HTTP/3 performance. Generally, it’s not a good story for QUIC or HTTP/3.

Just for completeness, here are some other testimonies of the performance of HTTP/3 and QUIC:

The results are mixed, but it generally indicates that the receiver end needs more optimizations. Specifically, the proposed solutions involve a technique called UDP generic receive offload (UDP GRO). Some experiments with these kinds of optimizations have shown very promising results. And you do have to remember that the number of round trips to establish a connection being reduced is, at its conceptual level, a winning strategy. The only thing stopping world domination is the pesky details.

 

However, there’s reason for optimism

The tooling surrounding HTTP/3 is rapidly maturing, which is significantly lowering the barrier to entry for developers eager to experiment and adopt this technology early on. With libraries like quic-go now offering comprehensive support for essential features like HTTP Trailers and tools like buf curl providing seamless testing capabilities, the path to integrating HTTP/3 into your gRPC projects is smoother than ever.

Additionally, performance optimizations are actively being researched and implemented. Beyond tooling, the performance landscape for HTTP/3 is far from stagnant. Active research and development are focused on optimizing QUIC implementations, particularly on the receiver side. Promising techniques like UDP Generic Receive Offload (GRO) show the potential to significantly enhance HTTP/3’s efficiency and responsiveness.

Even today, HTTP/3 and QUIC have their niches that are pretty compelling. Specifically, HTTP/3 consistently does pretty well with reducing the number of pauses with video conferencing and video streaming while improving general web usage with slow/unstable networks, typically with mobile devices.

Thanks all

With quic-go’s new support for HTTP trailers and buf curl’s new HTTP/3 flag, experimenting with gRPC over HTTP/3 is now easier than ever. I challenge you to try out gRPC over HTTP/3 in your own projects and share your experiences. I want to help build a community pushing gRPC and protobufs into more places, and this is a small part of that.

]]>JSON to Protobuf Conversionhttps://kmcd.dev/posts/json-to-proto/Tue, 10 Sep 2024 10:00:00 +0000https://kmcd.dev/posts/json-to-proto/ <p> <img hspace="5" src="https://kmcd.dev/posts/json-to-proto/cover_hu_5d77c665ee556aba.webp" /> </p> Deep dive into a small Protobuf tool In the world of data serialization and communication protocols, protocol buffers (protobuf) have gained a lot popularity due to their efficiency and performance advantages over formats like JSON and the usage of gRPC. However, the learning curve associated with protobuf’s syntax and concepts can be a deterrent for developers, especially those already comfortable with the ubiquitous JSON format. On top of that, converting a lot of APIs from JSON to protobuf can be really time-consuming.

Enter the JSON-to-Proto, a tool that aims to bridge this gap by automatically translating JSON data structures into their protobuf equivalents. While this can offer a seemingly straightforward entry point to protobuf adoption, it’s important to understand both the potential benefits and the limitations that come with such an approach.

The tool is located here: json-to-proto.github.io.

Potential use cases

Here are some situations where you might use this tool:

  • Rapid prototyping: When experimenting with a new project or feature, the converter enables developers to quickly generate protobuf schemas from existing JSON data structures, which may accelerate the initial development cycle.

  • Legacy system migration: For organizations transitioning from JSON-based systems to protobuf, the converter can ease the migration process by automating the initial conversion.

  • API design exploration: During the API design phase, the converter can be used to explore different protobuf schema representations based on sample JSON requests and responses, facilitating discussions and decision-making.

  • Educational purposes: The converter can serve as a valuable learning tool, allowing developers to visualize how JSON structures translate into protobuf schemas, aiding their understanding of protobuf concepts.

It’s crucial to remember that the converter is most effective when used strategically in conjunction with a growing understanding of protobuf. While it can provide a helpful starting point, developers should always strive to refine the generated schemas and leverage protobuf’s full capabilities for optimal performance and maintainability. There are many protobuf features that do not have a corollary in JSON, so try to take advantage of more protobuf’s features.

Getting it right

One place I like to go to for a typical REST API is the swagger petstore example, available at petstore.swagger.io. This API acts an example for OpenAPI, so it has a mix of features that you’ll see when using a typical REST API. So let’s what JSON-to-Proto will do with this API.

Starting with this example of a single “Pet” object:

{
  "id": 0,
  "category": {
    "id": 0,
    "name": "string"
  },
  "name": "doggie",
  "photoUrls": [
    "string"
  ],
  "tags": [
    {
      "id": 0,
      "name": "string"
    }
  ],
  "status": "available"
}

After pasting into the tool, I get the following protobuf back:

syntax = "proto3";

message SomeMessage {

    message Category {
        uint32 id = 1;
        string name = 2;
    }

    message Tags {
        uint32 id = 1;
        string name = 2;
    }

    uint32 id = 1;
    Category category = 2;
    string name = 3;
    repeated string photo_urls = 4;
    repeated Tags tags = 5;
    string status = 6;
}

This is pretty decent. It made some assumptions that uint32 is what all IDs look like, which may be fair but also may be wrong. Now that you’ve seen one example, let’s talk about ways this tool could lead us astray.

The Price of Simplicity

Loss of Expressiveness

JSON, while flexible, lacks the type richness and structural constraints of protobuf. This means that converting JSON to protobuf can result in a loss of information and reduced clarity.

  • Enums: Protobuf’s enums provide a clear way to define a restricted set of values. JSON, lacking this concept, forces converters to rely on string representations (or maybe numeric?), which are less type-safe and can lead to runtime errors.
  • Numeric Types: JSON’s “number” type is ambiguous, encompassing a wide range of numeric representations. Protobuf, on the other hand, offers a selection of specific integer and floating-point types, enabling more efficient storage and processing. Converters may struggle to accurately infer the intended protobuf type from a JSON number.
  • Maps: Converters probably can’t pick up on a field that has arbitrary key/value pairs. Since everything is a “JSON Object” it’s hard to tell the difference between a map<string, string> and message { ... }.
  • oneOf; This tool doesn’t detect situations where oneOf would be useful.

Imperfect Conversions

The complexity of data structures and the ambiguity inherent in JSON can lead to conversion errors or less-than-ideal protobuf schemas.

  • Ambiguous input, ambiguous output: If your examples leave a field as “null” then you may end up with google.protobuf.Any types, which is usually not what you want. The example data should always be “complete” to avoid this behavior.
  • One field, two types: When a JSON field can hold values of different types, converters might default to using google.protobuf.Any, a generic container type that can encapsulate any protobuf message. While convenient, this approach sacrifices type safety and can make working with the resulting protobuf data more cumbersome.

Getting it wrong

Let’s look at some concrete examples from JSON-to-proto to see where it can get things wrong and what you should do to correct it.

Incomplete examples

If any fields are excluded or set to null, JSON-to-proto will use google.protobuf.Any as the type. This is almost always the wrong answer:

{
    "id": 1234,
    "name": null
}

yields

syntax = "proto3";

import "google/protobuf/any.proto";

message SomeMessage {
    uint32 id = 1;
    google.protobuf.Any name = 2;
}

Multi-typed Fields

Here’s an example where the ID for users could be a string or an integer:

{
    "users": [
        {"type":"admin", "id": "admin-1"},
        {"type":"user", "id": 1234}
    ]
}

and here is the resulting protobuf:

syntax = "proto3";

message SomeMessage {

    message Users {
        string type = 1;
        string id = 2;
    }

    repeated Users users = 1;
}

The tool assumes that user.id is a string because it was the first type seen for that field. In this case, this is probably the wrong behavior. If you switch the order, you will get a uint32 as the type:

{
    "users": [
        {"type":"user", "id": 1234},
        {"type":"admin", "id": "admin-1"}
    ]
}

Yields:

syntax = "proto3";

message SomeMessage {

    message Users {
        string type = 1;
        uint32 id = 2;
    }

    repeated Users users = 1;
}

This is absolutely wrong, because there’s no reasonable way to encode “admin-1” as a single uint32 value. This behavior can also happen in a list that contains multiple types:

{
    "items": [1, "2", 3.3, null]
}

results in:

syntax = "proto3";

message SomeMessage {
    repeated uint32 items = 1;
}

This tool could definitely be smarter about this. In this particular case, I might have used the google.protobuf.Value well-known type to indicate that the array can have nulls, numbers, strings, booleans, or a list/struct that contains other google.protobuf.Value values. This type is very useful for working with dynamically typed fields and maps very cleanly to JSON.

You should note that having a single field that can have multiple types is frowned upon and is considered bad API design. If you have these cases, you may want to use this conversion process to move to a “one field: one type” design and not carry forward this bad API design into protobuf.

The missing warning label

So from this exploration, we can come up with some guidelines and warnings for using JSON-to-Proto:

  • Provide comprehensive and well-structured JSON examples: The converter relies on the provided JSON data to infer the protobuf schema. Make sure your examples are complete, representing all possible field types and variations within your data structure. Well-structured JSON with clear nesting and consistent naming conventions will further improve the accuracy of the conversion.
  • Manually review and refine the generated protobuf schema: The converter is not infallible. It’s crucial to carefully examine the generated protobuf schema, ensuring that it accurately reflects your data requirements and adheres to best practices. Consider aspects like field naming, data types, and the use of enums or nested messages to optimize the schema for performance and maintainability.
  • Use the converter as a starting point, not a definitive solution: The converter can quickly generate a protobuf schema, but it’s unlikely to be perfect right away. Treat the output as a draft and iterate on it based on your specific needs and protobuf best practices.
  • Avoid ambiguous or inconsistent JSON data: JSON’s flexibility can lead to ambiguity, which can confuse the converter. Try to maintain consistent data types within fields and avoid using null values whenever possible.
  • Consider edge cases: Ensure your JSON examples cover a wide range of possible scenarios, including edge cases and potential variations in your data. This will help the converter generate a more robust and adaptable protobuf schema.
  • Use protobuf’s other features: Once you’ve generated an initial schema, explore how protobuf’s advanced features, such as enums, oneof fields, and maps, can be used to further refine and optimize your data representation.
  • It may be better to avoid this tool altogether: Because there are so many caveats, you may want to stick to writing the protobuf schemas yourself, and avoiding this tool altogether.

By adhering to these best practices, you can leverage the JSON-to-Proto converter effectively while minimizing potential issues and ensuring the resulting protobuf schema aligns with your requirements and best practices.

Not a replacement for understanding protobuf

JSON to protobuf converters can be helpful in specific scenarios, especially for quick experimentation and legacy system integration. However, they should not be seen as a substitute for gaining a solid understanding of protobuf’s concepts and syntax. By investing time in learning protobuf directly, developers can leverage its full potential, designing schemas that are not only more expressive but also optimized for performance and maintainability. Remember, converters like this are tools, not magic wands. Use them wisely, but never let it replace the power of knowledge and try to recognize when this tool might not actually help you.

Roast Me

I get a fair amount of criticism with my articles sometimes, but I’ve always seen much harsher critiques from myself and from AI. I do, in fact, use AI to help with my writing process. However, rarely does AI-text directly make it into a final draft. Instead, I have iterations where I ask the AI to point out mistakes, and try to be my devil’s advocate. Sometimes it brings up good points. Sometimes it’s absolutely crazy. I feel like this has slowly improved my writing style and pointed out super obvious places where I need to transition better. However, sometimes I just ask it to “roast me” to prep me for responses I may receive. So here’s Google Gemini’s response to this article. I felt like this is worth adding here, because some of it needs to be said.

Wow, this article is a masterclass in taking a shortcut and hoping for the best. You know, Protobuf exists for a reason – it’s efficient, it’s fast, and it forces you to actually think about your data structures. But hey, who needs that when you can just mash a button and get a half-baked schema that might kinda-sorta work?

Let’s be real: this JSON-to-Proto tool is the equivalent of using Google Translate for your high school French essay. Sure, it might get you a passing grade, but you’re not fooling anyone. Your schema will be riddled with Any types, missed opportunities for enums, and a general sense of “I have no idea what I’m doing.”

The article itself tries to sound all responsible, warning you about the limitations of the tool. But let’s be honest, it’s basically saying, “Go ahead, use this thing, but don’t blame us when your production system melts down because you couldn’t be bothered to learn the difference between a uint32 and a string.”

Look, if you’re serious about using Protobuf, do yourself a favor and actually learn it. It’s not that hard. And if you’re not serious, well, then just stick to JSON and accept the performance hit. Either way, stop trying to find the easy way out. It’s called software engineering for a reason.

]]>gRPC: The Ugly Partshttps://kmcd.dev/posts/grpc-the-ugly-parts/Tue, 03 Sep 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-the-ugly-parts/ <p> <img hspace="5" src="https://kmcd.dev/posts/grpc-the-ugly-parts/cover_hu_3d7170590a3ed024.webp" /> </p> The seedy underbelly of gRPC. gRPC has undeniably become a powerful tool in the world of microservices, offering efficiency and performance benefits, but gRPC also has an ugly side. As someone who’s spent a considerable amount of time with gRPC, I’d like to shed light on some of the uglier aspects of this technology. I’ve already talked about the good and bad parts of gRPC, now let’s talk about the ugly.

Generated Code

To get started, I have to talk about how ugly the code generated from protobuf definitions is. It has historically been verbose, complex, and difficult to navigate. Even though it’s not meant to be hand-edited, this can impact code readability and maintainability, especially when integrating gRPC into larger projects. This has actually improved a lot recently in most languages but even so, there are some rough edges.

Language-specific Quirks

Protobuf and gRPC’s initial implementations often diverged from language-specific norms, especially in their HTTP handling. This stemmed partly from the decision to mandate HTTP/2 support, a decision that has since proven to limit gRPC’s reach into the web frontend. We know now from gRPC-Web that trailers aren’t a hard requirement for a protocol like gRPC. In the aftermath of this decision, we are now left with a need to evolve the language implementations of protobuf and gRPC to be more idiomatic for each language.

For Go, avoiding the net/http package is a rough decision because it makes it harder to use gRPC endpoints alongside other kinds of HTTP APIs and to re-use HTTP middleware. They eventually added a ServeHTTP() interface to grpc-go as an experimental way to use the HTTP server from the Go standard library but using that method results in a significant loss of performance. Maybe they did it for performance reasons? If so, it’s definitely a tradeoff that has split gRPC from the rest of the Go ecosystem.

Sometimes language quirks actually impact how you design protobuf types. If you follow the style recommendations from Buf, the names of enums are expected to be prefixed an upper-snake-case version of the enum name, like so:

enum FooBar {
  FOO_BAR_UNSPECIFIED = 0;
  FOO_BAR_FIRST_VALUE = 1;
  FOO_BAR_SECOND_VALUE = 2;
}

This is described better in the buf lint rule description for ENUM_VALUE_PREFIX but the style guide is like this because of C++ scoping rules with enums, which makes it impossible to have two enum values in the same package with the same enum value name. While this convention originated from C++ scoping rules, it affects how you should design all protobuf files. Why would scoping inside of the enum not be enough for the C++ compiler to generate unique names? Why is this flaw something that impacts the style guide and, in effect, all target languages? To me, this is kind of ugly, because quirks of some language implementations are bubbling up in unintuitive ways.

The generated code isn’t even that fast

One benefit of generated code is that you can generate code that no sane human would write in order to get some performance optimizations. However, if you look at some of the code generated from protobuf you’ll see runtime reflection used a lot. Why? In a way, I am saying the generated code isn’t ugly enough. Let’s look at a concrete example. Be warned that this will be a very Go-specific section because most of my experience with protobufs is in Go. However, the same strategy has been applied in most languages.

Let’s take a look at super a simple example in Go. Here’s the protobuf:

message Hello {
  string name = 1;
}

Here’s the type generated by protoc:

type Hello struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
}

// With these methods, contents are stripped
func (*Hello) Reset()
func (*Hello) String() string
func (*Hello) ProtoMessage()
func (*Hello) ProtoReflect() protoreflect.Message

There’s actually no Marshal() or Unmarshal() functions defined specifically for this type. This means that runtime reflection is used to make serialization work. Reflection is generally seen as slower, because it is slower. I find it strange that optimized, type-specific serialization code isn’t being generated for Go. That said, you can actually get this by using a separate protoc plugin called vtprotobuf that will generate specialized marshal and unmarshal functions for each protobuf type. It also allows for using type-specific memory pools, which can also help reduce allocations and improve performance. From my own testing just adding vtprotobuf with zero code changes can improve performance by 2-4%. This is essentially a “free” 2-4%, so it’s super strange to me that this wouldn’t be part of the standard compiler. You may not like it, but this is what peak performance looks like. Anyway, this project needs more love and support.

Note that there are other efforts which claim outrageous improvements over what the standard protobuf library does. They do make tradeoffs to achieve these performance gains, but many times the extra complexity is worth it.

You might have read this section and thought “well, this would increase the amount of code being generated and increase binary or package sizes and in some environments, you might not want that. That’s true amd that’s why protobuf has an optimize_for option, so you can annotate one of the following:

  • option optimize_for = SPEED; - more verbose, faster code
  • option optimize_for = CODE_SIZE; - smaller code
  • option optimize_for = LITE_RUNTIME; - intended to run on a smaller runtime that omits features like descriptors and reflection.

See the full description for optimize_for on the official protobuf documentation. While these options exist, they aren’t actually used for most target languages. In the future I would totally like to see most of vtprotobuf be rolled into the standard protobuf compiler for Go and be used if optimize_for = SPEED. Integrating vtprotobuf-like optimizations into the standard protobuf compiler could offer significant performance gains for Go and there are potentially similar opportunities in other languages as well.

Required Fields

The maintainers of protobuf learned some hard lessons with required fields. They felt like they misstepped so badly, that they made a new version of protobuf, proto3, just to remove required fields from the spec. Why? The author of the “Required considered harmful” manifesto talks about this in a lengthy hacker news comment, but the important bit is:

Real-world practice has also shown that quite often, fields that originally seemed to be “required” turn out to be optional over time, hence the “required considered harmful” manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change.

This is echoed by the official style guide of protobufs, where they recommend adding a comment indicating that a field is required. If we’re talking about getting a message from A to B, I totally agree with this line of thinking. However, just because the fields that are considered “required” change over time doesn’t mean required fields don’t exist. There still needs to be code that enforces this requirement and I’d rather not write this code, to be honest. Therefore, I think the best way of handling required fields without writing a bunch of null checks everywhere is by using protovalidate or a similar library that has protobuf options that allow you to annotate which fields are required. Then there is code on the server and/or client that can enforce these requirements using a library. In my opinion, this has the best of both worlds: you can still declare required fields in a way that doesn’t completely break message integrity.

I don’t like this:

message User {
  int32 age = 1; // required.
}

I do like this:

message User {
  int32 age = 1 [(buf.validate.field).required = true];
}

I’m a big fan of protovalidate and I’ve used it a good amount and have contributed to it. Generally, I think custom options for protobuf fields is an untapped superpower of protobufs.

Failure to Launch

While gRPC has undeniable advantages, its learning curve can be steep. Getting started with protobuf, understanding the tooling, and setting up the necessary infrastructure can be intimidating for newcomers, making the initial adoption hurdle higher than with simpler JSON-based APIs. Why is it steep? Well, it introduces non-idiomatic tooling to most languages. There are some examples of language support that make protobuf generation seamless. Grpc.Tools for .NET is one shining example, showing how protobuf tooling can be more integrated into standard language tooling. We need more of this.

The steep learning curve doesn’t help when many people who use and rely on protobuf and gRPC actively don’t want gRPC to extend to the frontend and think that pushing in this direction will lead to uninformed people encroaching on the domain of the backend, where only they are smart enough to work. This is elitist gate-keeping and is unfortunately prevalent in this industry. I believe gRPC has as much of a place in web frontends as much as it does in microservices.

I’ve learned a lot by helping others work with protobuf. You may see me on Buf’s slack channel or on related discussions because I truly have gotten a lot out of it. Many article ideas have come directly from answering questions there. If I see a problem often enough, I may end up writing an article about it. I believe the protobuf and gRPC community needs more of this attitude.

I believe the steep learning curve (which can be helped with tooling), coupled with some resistance from backend developers (which can be helped by… having empathy?), has slowed its broader adoption in web development.

gRPC Has a History

gRPC’s initial focus on microservices and its close ties to HTTP/2 hindered its widespread adoption in web development. Even with the advent of gRPC-Web, there’s still a perception that it’s not a first-class citizen in the frontend ecosystem. The lack of robust integration with popular frontend libraries like TanStack Query further solidifies this notion to me.

I think there’s a real chance to get more frontend developers excited about gRPC with improved tooling. There’s a giant industry-wide conversation happening right now around where the line between “frontend” and “backend” meet and I think no matter the outcome, we’re going to see more typescript code using gRPC.

The “g” in gRPC

While the gRPC project claims that the “g” in gRPC is a backronym that stands for “gRPC”, it originally stood for Google, because it was Google who developed and released both protobuf and gRPC.

There’s always a lingering question about Google’s long-term commitment to gRPC and protobuf. Will they continue to invest in these open-source projects, or could they pull the plug if priorities shift? Remember that Google has recently layed off much of the Flutter, Dart and Python teams. The protobuf community is growing, but would it be self-sustaining enough to survive such a scenario?

It’s Not Finished

Others have said that gRPC is immature, not because of its age but by how developed the ecosystem is. I tend to agree, because it’s missing features and tools that I would have expected from a mature ecosystem.

The missing package manager

Sharing protobuf definitions across multiple projects or repositories is a constant struggle without specialized tools. While solutions like Bazel, Pants, and Buf’s BSR exist, my experience with protobuf “in the real world” is… mixed. There are prominent open source projects, some by Google, that have bash scripts scrapped together to download dependencies before evoking protoc manually. Just imagine a programming language with no solution for managing dependencies. That’s insane. I think both Bazel and Buf tooling solve this problem pretty well but I’m just frustrated that every repo I come across that uses protobuf solves the problem in the most bespoke way possible. The community needs to come together to improve this. There is an open-source repo called Buffrs that appears to be tackling this problem. I haven’t used it personally but it looks decent so far.

Related to dependencies, I do want to call out that Google’s “well-known” protobuf types get special privilege of being built into protoc. While these types are incredibly useful and invaluable, their privilege makes it hard for other libraries of useful protobuf types to exist and thrive. Just building these protobuf definitions into protoc (and other tooling) is a cop out for not having a real and consistent story for dependency management.

Editor Support

Editor integration for protobuf code generation leaves a lot to be desired. It would be immensely helpful if editors could intelligently link generated code back to its protobuf source. This would provide a more seamless experience, but the tooling just isn’t smart enough yet. Also, I think everyone needs to run with Buf’s editor support. Having a linter and autoformatter built into your editor is the expected from developers nowadays. And with protobuf, there are extremely real reasons to follow the advice of the linter.

Projects like tRPC showcase the benefits of tight integration and opinionated design choices—something that protobuf, by its nature, can’t fully replicate. However, I remain hopeful that the protobuf ecosystem can evolve to offer a similarly streamlined developer experience.

Ugly Documentation

I’ve never seen documentation generated from protobuf that wasn’t super ugly. I think since gRPC has historically been a backend service, the backend devs never bothered to put any real effort into making pretty documentation output using a protoc plugin. I’ve solved this problem by making a protoc plugin that generates OpenAPI from given protobuf files. Then I use one of the many beautiful tools for displaying the OpenAPI spec. This was, by far, much easier than getting me to make a decent design. Another side benefit for generating OpenAPI from protobuf is the ability to tap into that ecosystem since there’s more to it than just documentation.

Let’s look at a real example. This is a document generated using one of the few tools for generating documentation from protobuf, protoc-gen-doc:

Compare it to some of the OpenAPI tooling. This was generated using Elements, but there are many, many other alternatives that look equally as polished:

It’s kind-of not fair to point at a single plugin and say that the default template doesn’t look as good as OpenAPI alternatives, because you actually do have more flexibility with protoc-gen-doc. It allows you to specify your own template so it could look as beautiful as you want. However, this does line up with my point: the tooling is more finished and polished in the REST world than gRPC. This is a fixable problem, but we need to get frontend devs and designers excited about gRPC or backend engineers need to start sharpening their design skills.

I also want to note that OpenAPI/Swagger interfaces often have a way to test endpoints directly from the documentation website. This is completely missing from equivalent tools in the gRPC world. Additionally, with most OpenAPI documentation tools you can clearly see which fields are required and will display constraints on fields that have them. So not only is it prettier, it’s more functional as well.

Conclusion

gRPC, while a powerful tool in many ways, still has room to grow. The less-than-ideal aspects of generated code, coupled with the challenges of dependency management and evolving protobuf schemas, can create friction for developers. The lack of intuitive editor integration and the historical focus on backend services have also hindered its wider adoption in web development.

However, I think the future of gRPC is bright and can be far less ugly. The community is actively addressing these challenges, developing tools like the buf CLI, protovalidate and protoc-gen-connect-openapi to bridge the gaps and enhance the developer experience. As gRPC matures and its ecosystem expands, we can anticipate improved tooling, better editor support, and a smoother integration into the frontend world.

]]>Working with Protobuf in 2024https://kmcd.dev/posts/working-with-protobuf-in-2024/Tue, 27 Aug 2024 10:00:00 +0000https://kmcd.dev/posts/working-with-protobuf-in-2024/ <p> <img hspace="5" src="https://kmcd.dev/posts/working-with-protobuf-in-2024/cover_hu_c4edfa33fe867816.webp" /> </p> Tools and tricks for developing with protobuf. Protocol Buffers (protobuf), Google’s brainchild for efficient data serialization, have become an indispensable tool in the modern software development landscape. They offer a compact and efficient way to structure data for storage or transmission, making them ideal for applications like gRPC services, data storage, and inter-service communication. gRPC is even branching into the frontend of the web, with gRPC-Web and ConnectRPC. This means that the traditional protobuf workflow can sometimes feel a bit dated and cumbersome with tooling that isn’t quite made to be easy to use and, worse, can be easy to use incorrectly. In this article, we’ll explore some modern tools that address these pain points, making protobuf development more enjoyable and productive. But first, let’s recap how the “traditional” protobuf workflow typically works.

Traditional Workflow

Here’s what a typical workflow can look like when working with protobufs:

  1. Define the Protobuf File: You create a .proto file, defining your message structures, enums, and services (if using gRPC).
  2. Compile with protoc: You use the protoc compiler to generate code in your desired languages (e.g., Go, Java, Python, Rust, etc.) from your .proto files.
  3. Implement and Use: You implement the server-side logic (if applicable) using the generated server stubs and utilize the generated client code to interact with your protobuf-based services.

Let’s visualize this traditional workflow to highlight these pain points:

What’s Missing?

While this workflow gets the job done, it leaves a lot of room for improvement:

  • Error-Prone Build Steps: The process often involves manual compilation and code generation, increasing the risk of human errors. Most people introduce a Makefile or a bash script to handle calling protoc, but even this can be error-prone and difficult to maintain. Some believe this doesn’t exist or some magical build system “handles it for you”, but I assure you, it exists and it’s not hard to find plenty of examples of this.
  • This workflow doesn’t include a standardized way to handle protobuf dependencies. Traditionally, this involves the Makefile or bash script that I just mentioned to download protobufs from various online repositories or manually copying them between projects. This is bad, but the Protobuf/gRPC projects don’t really give much guidance on how to reuse protobufs, so people have just done it themselves in wildly different ways.
  • Lack of Consistency: Without proper tooling, it’s challenging to maintain consistent formatting and styling across multiple .proto files.
  • Breaking Changes: Manually tracking changes to your protobufs and ensuring backward compatibility can be tedious and error-prone.
  • No Mocking or Prototyping: It takes a good amount of effort to make a gRPC service, even if you just want to use mock data.

Fortunately, the Protobuf ecosystem has evolved dramatically in the last few years, and most of the advancement is from third-parties (not Google), which is exciting. So let’s cover parts of the modern workflow that you won’t see in a gRPC or protobuf tutorial.

The next generation of protobuf tooling

The Protobuf ecosystem has seen a surge of innovation in recent years, with many third-party tools emerging to address the limitations of the traditional workflow. Let’s delve into some of these tools.

JSON to Proto

JSON to Proto is a online tool that simplifies the creation of protobuf definitions if you are brand new to protobufs. I wouldn’t recommend using this for the long haul, but it can be a good way to get started quickly by pasting in sample JSON data, and the tool will generate a corresponding .proto file for you. This is particularly useful when you’re starting with existing JSON data or want to quickly prototype a protobuf schema. Learn more from the Github repo.

Protobuf Pal

Protobuf Pal is a browser-based protobuf editor designed to streamline the creation and editing of .proto files. It offers features like syntax highlighting, error checking, and auto-completion, making it easier to write valid protobuf definitions.

Buf CLI

Buf is a comprehensive toolkit designed to make working with protobufs a breeze.

The Buf CLI offers several powerful features:

  • buf generate: A replacement for protoc that provides consistent code generation across different languages and environments. Read more here.
  • buf lint: A linter that helps you maintain clean and consistent protobuf definitions, catching potential issues early. Read more here.
  • buf format: An opinionated formatter to ensure consistent styling across your .proto files. Read more here.
  • buf curl: is useful not only for development but also for testing. You can use it to send gRPC requests from your terminal, making it easy to verify the behavior of your services. It can craft requests using server reflection, local protobuf files, descriptor files or from a reference to the buf.build registry. Read more here.
$ buf curl --list-methods https://demo.connectrpc.com
connectrpc.eliza.v1.ElizaService/Converse
connectrpc.eliza.v1.ElizaService/Introduce
connectrpc.eliza.v1.ElizaService/Say
$ buf curl -d '{"sentence":"Hello! I need some help, doc"}' https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello...I'm glad you could drop by today."
}

You can use the buf CLI directly on the CLI (which is great when you’re playing around or adding it to a continuous integration process) or integrated into your code editor, which is where the real magic happens, anyway.

Buf Schema Registry (BSR)

The Buf Schema Registry (BSR) is the missing package manager for Protobufs, but it does a bit more than that. Not only does it allow you to push versioned schemas to one place, but it also has cool features like automatically generated SDKs. The idea is that you can just import the package in your favorite language and magically you have your server stubs and clients in the language of your choice (as long as it’s Go, Typescript/Javascript, Java/Kotlin, Swift, Python or Rust).

With the BSR, you no longer need bespoke Makefile or bash scripts to pull down protobuf dependencies.

Testing

This next set of tools are good for testing live gRPC endpoints. Obviously, gRPCurl is an amazing tool for this, but let’s discover some tools that are a bit newer to the scene.

Buf Studio

Buf Studio is an interactive web UI for all your gRPC and Protobuf services stored on the Buf Schema Registry. With Buf Studio you can craft gRPC/gRPC-Web/Connect requests using images on the buf registry. Buf Studio uses those protobuf schemas to support autocompletion of these requests, which is super cool. It can also use an agent that is built into the Buf CLI to proxy requests from internal networks, making this web-based tool a bit more flexible.

Postman

Postman, a popular API testing tool, now supports gRPC. You can leverage its familiar interface to construct and send gRPC requests, making it a convenient option for testing your protobuf services.

Insomnia

Insomnia is another API testing platform that has added gRPC support. Similar to Postman, it allows you to design and execute gRPC requests within its user-friendly environment.

Insomnia’s UI takes a little getting used to but once gRPC is set up, it gets easier.

k6

k6 is a powerful load testing tool that can be used to simulate heavy traffic on your gRPC services. It helps you identify performance bottlenecks and ensure your services can handle real-world loads.

import { Client, StatusOK } from 'k6/net/grpc';
import { check, sleep } from 'k6';

const client = new Client();
client.load(['definitions'], 'eliza.proto');

export default () => {
  client.connect('127.0.0.1:10000', {});

  const data = { sentence: 'Hello, doc!' };
  const response = client.invoke('connectrpc.eliza.v1.ElizaService/Say', data);

  check(response, {
    'status is OK': (r) => r && r.status === StatusOK,
  });

  console.log(JSON.stringify(response.message));

  client.close();
  sleep(1);
};

FauxRPC

I couldn’t write this article without promoting my own tool, FauxRPC. FauxRPC is a tool that enables developers to quickly generate mock gRPC servers from protobuf definitions, facilitating early API development and testing. By incorporating FauxRPC into your workflow, you can easily create realistic mock services that simulate real gRPC server behavior, allowing you to test your client implementations and identify potential issues without the need for a fully implemented backend. This streamlined prototyping and testing process ultimately fosters faster iteration and more robust API development.

With a single command, you can have a server running with fake data!

$ buf build buf.build/connectrpc/eliza -o eliza.binpb
$ fauxrpc run --schema=eliza.binpb
FauxRPC (0.0.16 (97e4c8caf9d3c22387a393180e00bce40b2834c6) @ 2024-08-22T18:42:56Z; go1.22.4)
Listening on http://127.0.0.1:6660
OpenAPI documentation: http://127.0.0.1:6660/fauxrpc.openapi.html

Example Commands:
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660 --list-methods
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/[METHOD_NAME]

Learn more about it in my previous post announcing it or the documentation website.

New Workflow

This enhanced workflow empowers developers to iterate faster, catch errors earlier, and ensure API stability, ultimately leading to more robust and maintainable protobuf-based applications. Let’s explore how these tools fit into the enhanced workflow:

Note that this workflow has one more “find issues and iterate” connections right after writing the protobuf file. That’s because these new tools will help you find issues with your protobuf schemas earlier in a more automated way.

Also note that with FauxRPC, frontend developers (or whatever is using the generated clients for the service) can start working on their part before the backend developer is finished with their work. Frontend devs no longer have to come up with mock APIs (which rapidly get outdated with reality) just to get started on their frontend work. Integration work on another service which uses this protobuf can happen before the backend implementation is completed. Everyone can work in parallel and the better the schema is (with protovalidate constraints) the better the fake data will be.

More ways to start making protobuf files

Developers can still define services, messages, and enums directly in the .proto file. However, tools like JSON-to-Proto and Protobuf Pal provide visual aids and assistance in creating and editing .proto files, reducing errors and improving productivity. I wouldn’t recommend these tools all the time, but they are probably good for quickly getting a prototype going quickly.

Automated Code Generation and Management

Buf’s buf generate command streamlines the code generation process with a set of declarative configuration files, ensuring consistency across different languages and platforms. Buf’s dependency management capabilities eliminate the need for manual scripts to handle protobuf dependencies.

Enhanced Quality Control

  • Buf’s buf lint enforces coding standards and best practices, catching potential issues early in development.
  • Buf’s buf format automatically formats your .proto files, ensuring consistency and readability.
  • Buf’s buf breaking change detection helps you avoid introducing changes that could disrupt existing clients.

All of these put together mean that issues in your API schema are discovered automatically, as soon as possible.

Streamlined Testing and Prototyping

Buf’s buf curl and Buf Studio’s built-in gRPC client enable quick testing and interaction with your services. FauxRPC facilitates rapid prototyping by allowing you to mock gRPC services without implementing the actual server-side logic.

Conclusion

The traditional protobuf workflow, while functional, can involve manual steps and potential pitfalls. By incorporating tools like the Buf CLI, JSON-to-Proto, Protobuf Pal, and FauxRPC into your development process, you can significantly enhance your productivity and ensure the quality of your protobuf definitions. Note that I haven’t come close to outlining all of the different tools that you can use with Protobuf. So, don’t hesitate to explore these tools and discover how they can transform your Protobuf development experience!

]]>Introducing FauxRPChttps://kmcd.dev/posts/fauxrpc/Tue, 20 Aug 2024 10:00:00 +0000https://kmcd.dev/posts/fauxrpc/ <p> <img hspace="5" src="https://kmcd.dev/posts/fauxrpc/cover_hu_39af42cea62050f8.webp" /> </p> I made a server that outputs nonsense. I would like to introduce FauxRPC, a powerful tool that empowers you to accelerate development and testing by effortlessly generating fake implementations of gRPC, gRPC-Web, Connect, and REST services. If you have a protobuf-based workflow, this tool could help.

Why FauxRPC?

  • Faster Development & Testing: Work independently without relying on fully functional backend services.
  • Isolation & Control: Test frontend components in isolation with controlled fake data.
  • Multi-Protocol Support: Supports multiple protocols (gRPC, gRPC-Web, Connect, and REST).
  • Prototyping & Demos: Create prototypes and demos quickly without building the full backend. Fake it till you make it.
  • Improved Collaboration: Bridge the gap between frontend and backend teams.
  • Plays well with others: Test data from FauxRPC will try to automatically follow any protovalidate constraints that are defined.

How it Works

FauxRPC leverages your Protobuf definitions to generate fake services that mimic the behavior of real ones. You can easily configure the fake data returned, allowing you to simulate various scenarios and edge cases. It takes in *.proto files or protobuf descriptors (in binpb, json, txtpb, yaml formats), then it automatically starts up a server that can speak gRPC/gRPC-Web/Connect and REST (as long as there are google.api.http annotations defined). Descriptors contain all of the information found in a set of .proto files. You can generate them with protoc or the buf build command.

Get Started

FauxRPC is available as an open-source project. Check out the documentation and examples to get started. Here’s a quick overview, but be sure to check the official documentation for the most up-to-date instructions:

Install via source

go install github.com/sudorandom/fauxrpc/cmd/fauxrpc@latest

Pre-built binaries

Binaries are built for several platforms for each release. See the latest ones on the releases page.

Use Descriptors

Make an example.proto file (or use a file that already exists):

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

Create a descriptors file and use it to start the FauxRPC server:

$ buf build ./example.proto -o ./example.binpb
$ fauxrpc run --schema=./example.binpb
2024/08/17 08:01:19 INFO Listening on http://127.0.0.1:6660
2024/08/17 08:01:19 INFO See available methods: buf curl --http2-prior-knowledge http://127.0.0.1:6660 --list-methods

Done! It’s that easy. Now you can call the service with any tooling that supports gRPC, gRPC-Web, or connect. So buf curl, grpcurl, Postman, Insomnia all work fine!

$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
  "greeting": "dream"
}

Server Reflection

If there’s an existing gRPC service running that you want to emulate, you can use server reflection to start the FauxRPC service:

$ fauxrpc run --schema=https://demo.connectrpc.com

From BSR (Buf Schema Registry)

Buf has a schema registry where many schemas are hosted. Here’s how to use FauxRPC using images from the registry.

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.json
$ fauxrpc run --schema=./bufbuild.registry.json

Multiple Sources

You can define this --schema option as many times as you want. That means you can add services from multiple descriptors and even mix and match from descriptors and from server reflection:

$ fauxrpc run --schema=https://demo.connectrpc.com --schema=./example.binpb

Multi-protocol Support

The multi-protocol support is based on ConnectRPC. So with FauxRPC, you get gRPC, gRPC-Web and Connect out of the box. However, FauxRPC does one thing more. It allows you to use google.api.http annotations to present a JSON/HTTP API, so you can gRPC and REST together! This is normally done with an additional service that runs in-between the outside world and your actual gRPC service but with FauxRPC you get the so-called transcoding from HTTP/JSON to gRPC all in the same package. Here’s a concrete example:

syntax = "proto3";

package http.service;

import "google/api/annotations.proto";

service HTTPService {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (google.api.http) = {get: "/v1/{name=messages/*}"};
  }
}
message GetMessageRequest {
  string name = 1; // Mapped to URL path.
}
message Message {
  string text = 1; // The resource content.
}

Again, we start the service by building the descriptors and using

$ buf build ./httpservice.proto -o ./httpservice.binpb
$ fauxrpc run --schema=httpservice.binpb

Now that we have the server running we can test this with the “normal” curl:

$ curl http://127.0.0.1:6660/v1/messages/123456
{"text":"Retro."}⏎

Sweet. You can now easily support REST alongside gRPC. If you are wondering how to do this with “real” services, look into vanguard-go. This library is doing the real heavy lifting.

What does the fake data look like?

You might be wondering what actual responses look like. FauxRPC’s fake data generation is continually improving so these details might change as time goes on. It uses a library called fakeit to generate fake data. Because protobufs have pretty well-defined types, we can easily generate data that technically matches the types. This works well for most use cases, but FauxRPC tries to be a little bit better. If you annotate your protobuf files with protovalidate constraints, FauxRPC will try its best to generate data that matches these constraints. Let’s look at some examples!

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With FauxRPC, you will get any kind of word, so it might look like this:

{
  "greeting": "sufficient"
}

This is fine, but for the RPC, we know a bit more about the type being returned. We know that it sends a greeting back that looks like “Hello, [name]”. So here’s what the same protobuf file might look like with protovalidate constraints:

Now let’s see what this looks like with protovalidate constraints:

syntax = "proto3";

import "buf/validate/validate.proto";

package greet.v1;

message GreetRequest {
  string name = 1 [(buf.validate.field).string = {min_len: 3, max_len: 100}];
}

message GreetResponse {
  string greeting = 1 [(buf.validate.field).string.pattern = "^Hello, [a-zA-Z]+$"];
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With this new protobuf file, this is what FauxRPC might output now:

{
  "greeting": "Hello, TWXxF"
}

This shows how protovalidate constraints enable FauxRPC to generate more realistic and contextually relevant fake data, aligning it closer to the expected behavior of your actual services. As another example, I will show one of Buf’s services used to manage users, buf.registry.owner.v1.UserService. Here’s what the UserRef message looks like:

message UserRef {
  option (buf.registry.priv.extension.v1beta1.message).request_only = true;
  oneof value {
    option (buf.validate.oneof).required = true;
    // The id of the User.
    string id = 1 [(buf.validate.field).string.tuuid = true];
    // The name of the User.
    string name = 2 [(buf.validate.field).string = {
      min_len: 2
      max_len: 32
      pattern: "^[a-z][a-z0-9-]*[a-z0-9]$"
    }];
  }
}

So let’s make our descriptors for this service, start the FauxRPC server and make our example request:

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.binpb
$ fauxrpc run --schema=./bufbuild.registry.binpb
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/buf.registry.owner.v1.UserService/ListUsers
{
  "nextPageToken": "Food truck.",
  "users": [
    {
      "id": "c4468393f926400d8880a264df9c284a",
      "createTime": "2012-03-06T12:15:03.239463070Z",
      "updateTime": "1990-10-29T13:12:31.224347086Z",
      "name": "jexox",
      "type": "USER_TYPE_STANDARD",
      "description": "Tattooed taxidermy.",
      "url": "http://www.productexploit.name/synergies/target"
    },
    {
      "id": "0e4ca24f4ff54761b109daab0da1bea2",
      "createTime": "1955-05-16T02:37:30.643378679Z",
      "updateTime": "1923-08-28T04:28:43.330711919Z",
      "name": "ya0",
      "type": "USER_TYPE_STANDARD",
      "state": "USER_STATE_INACTIVE",
      "description": "Helvetica.",
      "url": "https://www.centralengage.info/markets/scale/e-commerce/exploit",
      "verificationStatus": "USER_VERIFICATION_STATUS_UNVERIFIED"
    }
  ]
}

Hopefully, this gives you a good idea of what the output might look like. The better your validation rules, the better the FauxRPC data will be.

What’s left?

FauxRPC is already great for some use cases but it’s not “done” as there’s more to do to make it better. I have plans to add the ability to configure stubs for each RPC method. This will allow you to define specific responses or behaviors for each RPC, giving you more control over the simulated service. I hope this will make it easier to iterate on protobuf designs without needing to actually implement services until later.

Stay Tuned

I made a documentation website to organize documentation. I think it looks pretty good for how quickly I threw it together. The code for FauxRPC lives on GitHub at sudorandom/fauxrpc. It’s a little thin now but there’s a lot that I can write about in there. I’m actively developing FauxRPC and have many exciting features planned for the future. This is early on for this project but it has come together as a coherent and useful program for me extremely quickly. So please try it out and let me know your feedback and suggestions. Stay tuned for updates!

… and don’t forget to star the repo on GitHub. It helps more than you know!

]]>HTTP/1.0 From Scratchhttps://kmcd.dev/posts/http1.0-from-scratch/Tue, 13 Aug 2024 00:00:00 +0000https://kmcd.dev/posts/http1.0-from-scratch/ <p> <img hspace="5" src="https://kmcd.dev/posts/http1.0-from-scratch/cover_hu_2088ae23d8169bf0.webp" /> </p> Laying the Foundation: Building the Web with HTTP/1.0. Introduction

In our previous exploration, we delved into the simplicity of HTTP/0.9, a protocol that served as the web’s initial foundation. However, as the internet evolved, so did its needs. Enter HTTP/1.0, a landmark version released in 1996 that laid the groundwork for the web we know today. HTTP/1.0 was a game-changer, introducing features that revolutionized web communication:

  • Headers: Metadata that added context and control to requests and responses (RFC 1945 4.2).
  • Methods: A diverse set of actions (POST, HEAD, PUT, DELETE, etc.) beyond just retrieving documents. The web was no longer read-only. (RFC 1945 8).
  • Status Codes: Clear signals about the outcome of requests, paving the way for better error handling and redirection (RFC 1945 6.1.1).
  • Content Negotiation: The ability to request specific formats (RFC 1945 10.5), encoding (RFC 1945 10.3) or languages (RFC 1945 D.2.5) for content.

In this article, we’ll journey through the intricacies of HTTP/1.0 and craft a simple Go server that speaks the protocol.

Understanding HTTP/1.0

First, let’s cover the new features added in HTTP/1.0. I hope at the end of this section you’ll understand why this version was truly deserving of the “1.0” labeling.

Headers

Headers convey vital information about requests and responses. Some common headers include:

  • Content-Type: Indicates the format of the data (text/html, image/jpeg, etc.). (RFC 1945 10.5)
  • Accept: Tells the server which content type the client expects. (RFC 1945 D2.1)
  • Content-Length: Specifies the size of the response body. (RFC 1945 10.4)
  • User-Agent: Identifies the client software making the request. (RFC 1945 10.15)

…but many more have since become standard.

HTTP Methods

HTTP/1.0 introduced a variety of methods:

  • GET: Requests a resource.
  • POST: Submits data to be processed by the server.
  • HEAD: Similar to GET, but only requests the headers, not the body.
  • PUT: Replaces an existing resource or creates a new one if it doesn’t exist.
  • DELETE: Removes a specified resource.

Status Codes

Status codes are essential for communication between the client and server. They fall into categories:

Request Structure

HTTP/1.0 requests follow a structured format:

  1. Request Line: Specifies the HTTP method (e.g., GET, POST), the requested path, and the protocol version (HTTP/1.0).
  2. Headers: Key-value pairs that provide additional information (e.g., User-Agent, Content-Type, Referer).
  3. Empty Line: Signals the end of the headers.
  4. Request Body (Optional): Data sent with the request (common with POST).

Headers and request bodies are completely new in this version. Now there is an official way to send data to servers using HTTP. Before this, it was possible to encode information into the HTTP path to send data to the servers but now that HTTP servers can accept entire documents from clients, it opens up the possibilities for more dynamic web experiences. And don’t sleep on headers. Headers provide a way to annotate more details about web requests that would be very tedious to add in the URL. Now the client can specify the language, content-encoding, character-set, content-type that it will accept as a response and much, much more.

Example

GET /index.html HTTP/1.0
User-Agent: Mozilla/5.0
Host: www.example.com

(Optional request body)

Response Structure

HTTP/1.0 responses mirror this structure:

  1. Status Line: Includes the protocol version, a status code (e.g., 200 OK, 404 Not Found), and a reason phrase. (RFC 1945 6.1)
  2. Headers: Similar to request headers, providing metadata about the response. (RFC 1945 6.2)
  3. Empty Line: Separates headers from the body.
  4. Response Body: The actual content sent back to the client.

Status codes now enable more robust error handling. Web servers would use the status codes to show appropriate error pages to the user. These error codes have been so important that non-technical people around the world will know of 404 Not Found.

Now HTTP responses return headers. This adds important information about how to render the document. Finally, we’re not just limited to HTML. Headers are critical to many features today, including redirection, caching, authentication, authorization, and many more.

Example

HTTP/1.0 200 OK
Content-Type: text/html
Content-Length: 1354

(HTML content here)

Nothing has changed with the sequence of events from HTTP/0.9. It still looks like this:

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: TCP SYN
        Server ->> Client: TCP SYN-ACK
        Client ->> Server: TCP ACK
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end

    rect rgb(47,75,124)
        Server ->> Client: TCP FIN
		Client ->> Server: TCP ACK
    end

Caching

Now that request and response headers exist, it’s possible to negotiate caching behavior, which is well-defined in the HTTP/1.0 spec. It defines which methods can be cached and which headers to use to define the behavior. While it’s an optional feature for servers, it was very important in the early days of the web and I would say that it’s even more important today.

Host Header

One thing that goes overlooked is that goes overlooked is that Host header but, arguably, it’s what made the early web possible. With that one header, one web server can host many different websites using the same IP public address and post. It made the cost of running a website on your domain more reasonable and it created an industry of reseller web hosts.

Implementing an HTTP/1.0 Server in Go

Enough theory! Let’s roll up our sleeves and bring HTTP/1.0 to life.

Programming is learned by writing programs. — Brian Kernighan

We’ll build a simple Go server from the ground up, handling requests and responses with the elegance and efficiency that Go is known for. By the end of this section, you’ll have a working HTTP/1.0 server that you can interact with using familiar tools like curl and your web browser. Note that this server is NOT “production ready” and is only meant for learning. Many aspects of HTTP are not clearly defined in the spec that are critical to get right to avoid security exploits and denial of service attacks.

If you’d rather just dig into the code yourself, Go here: main.go. Some people (like me) learn better by just downloading the entire script and modifying it to see what breaks to get a better understanding.

The heart of our HTTP/1.0 server is this Server struct, which encapsulates the server’s address and the handler responsible for processing incoming requests. The ListenAndServe() method initiates the server, listens for connections, and handles each one concurrently, in a new goroutine.

type Server struct {
	Addr    string
	Handler http.Handler
}

func (s *Server) ListenAndServe() error {
	handler := s.Handler
	if handler == nil {
		handler = http.DefaultServeMux
	}
	l, err := net.Listen("tcp", s.Addr)
	if err != nil {
		return err
	}
	defer l.Close()

	for {
		conn, err := l.Accept()
		if err != nil {
			return err
		}

		go func() {
			if err := s.handleConnection(conn); err != nil {
				slog.Error(fmt.Sprintf("http error: %s", err))
			}
		}()
	}
}

You might notice that it’s mostly unchanged from the 0.9 version. But as a recap, this code sets up a TCP listener on the specified address, starts an infinite loop, accepts connections and calls s.handleConnection in a new goroutine.

Now we’re moving to the heart of the HTTP server. The handleConnection method parses the incoming HTTP request, extracting the method, path, headers, and optional body. It then passes the constructed http.Request to the server’s handler for further processing and response generation.

func (s *Server) handleConnection(conn net.Conn) error {
	defer conn.Close()

	// Limit headers to 1MB
	limitReader := io.LimitReader(conn, 1*1024*1024).(*io.LimitedReader)
	reader := bufio.NewReader(limitReader)
	headerReader := textproto.NewReader(bufio.NewReader(reader))

	// Read the request line: GET /path/to/index.html HTTP/1.0
	reqLine, err := headerReader.ReadLine()
	if err != nil {
		return fmt.Errorf("read request line error: %w", err)
	}

	req := new(http.Request)
	var found bool

	// Parse Method: GET/POST/PUT/DELETE/etc
	req.Method, reqLine, found = strings.Cut(reqLine, " ")
	if !found {
		return errors.New("invalid method")
	}
	if !methodValid(req.Method) {
		return errors.New("invalid method")
	}

	// Parse Request URI
	req.RequestURI, reqLine, found = strings.Cut(reqLine, " ")
	if !found {
		return errors.New("invalid path")
	}
	if req.URL, err = url.ParseRequestURI(req.RequestURI); err != nil {
		return fmt.Errorf("invalid path: %w", err)
	}

	// Parse protocol version "HTTP/1.0"
	req.Proto = reqLine
	req.ProtoMajor, req.ProtoMinor, found = parseProtocol(req.Proto)
	if !found {
		return errors.New("invalid proto")
	}

	// Parse headers
	req.Header = make(http.Header)
	for {
		line, err := headerReader.ReadLineBytes()
		if err != nil && err != io.EOF {
			return err
		} else if err != nil {
			break
		}
		if len(line) == 0 {
			break
		}

		k, v, ok := bytes.Cut(line, []byte{':'})
		if !ok {
			return errors.New("invalid header")
		}
		req.Header.Add(strings.ToLower(string(k)), strings.TrimLeft(string(v), " "))
	}

	// Unbound the limit after we've read the headers since the body can be any size
	limitReader.N = math.MaxInt64

	ctx := context.Background()
	ctx = context.WithValue(ctx, http.LocalAddrContextKey, conn.LocalAddr())
	ctx, cancelCtx := context.WithCancel(ctx)
	defer cancelCtx()
	contentLength, err := parseContentLength(req.Header.Get("Content-Length"))
	if err != nil {
		return err
	}
	req.ContentLength = contentLength
	if req.ContentLength == 0 {
		req.Body = noBody{}
	} else {
		req.Body = &bodyReader{reader: io.LimitReader(reader, req.ContentLength)}
	}

	req.RemoteAddr = conn.RemoteAddr().String()
	req.Close = true // this is always true for HTTP/1.0

	w := &responseBodyWriter{
		// We hard-code this because this is a HTTP/1.0 server.
		// Web servers will make requests with HTTP/1.1 but
		// we're saying that we only support HTTP/1.0.
		proto:   "HTTP/1.0",
		conn:    conn,
		headers: make(http.Header),
	}

	// Finally, call our http.Handler!
	s.Handler.ServeHTTP(w, req.WithContext(ctx))
	if !w.sentHeaders {
		w.sendHeaders(http.StatusOK)
	}
	return nil
}
  • Creates a fresh *http.Request object that we’ll pass to the given http.Handler
  • Reads and parses the request line (method, path, protocol) GET /path/to/index.html HTTP/1.0
  • Parses and stores request headers
  • Handles the request body based on the Content-Length header
  • Creates a responseBodyWriter to handle writing the response headers and body
  • Calls the server’s Handler to process the request and generate the response
  • Sends the response headers if not already sent.

Now let’s dig into the utility functions that we just called. First, here’s a simple one. If the client didn’t send a HTTP body, we still need to populate request.Body with something, so this is a mocked-out version that satisfies the required io.ReadCloser interface.

type noBody struct{}

func (noBody) Read([]byte) (int, error)         { return 0, io.EOF }
func (noBody) Close() error                     { return nil }

Next, we have the type that we use for request.Body when there’s a request body. It is fairly simple:

type bodyReader struct {
	reader io.Reader
}

func (r *bodyReader) Read(p []byte) (n int, err error) {
	return r.reader.Read(p)
}

func (r *bodyReader) Close() error {
	_, err := io.Copy(io.Discard, r.reader)
	return err
}

Here are assorted helper functions that don’t have too much to comment on. I had to add HTTP/1.1 to parseProtocol because many browsers and other tools will send requests as HTTP/1.1 because there’s no protocol version negotiation before a request to tell HTTP/1.0 and HTTP/1.1 apart.

func parseContentLength(headerval string) (int64, error) {
	if headerval == "" {
		return 0, nil
	}

	return strconv.ParseInt(headerval, 10, 64)
}

func parseProtocol(proto string) (int, int, bool) {
	switch proto {
	case "HTTP/1.0":
		return 1, 0, true
	case "HTTP/1.1":
		return 1, 1, true
	}
	return 0, 0, false
}

func methodValid(method string) bool {
	switch method {
	case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete, http.MethodConnect, http.MethodOptions, http.MethodTrace:
		return true
	}
	return false
}

And now we have the response body writer. So far, all of the code has been in aid for creating an http.Request object, but handlers also need a http.ResponseWriter to satisfy the handler interface:

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

Here’s the implementation:

type responseBodyWriter struct {
	proto       string
	conn        net.Conn
	sentHeaders bool
	headers     http.Header
}

func (r *responseBodyWriter) Header() http.Header {
	return r.headers
}

func (r *responseBodyWriter) Write(b []byte) (int, error) {
	if !r.sentHeaders {
		r.sendHeaders(http.StatusOK)
	}
	return r.conn.Write(b)
}

func (r *responseBodyWriter) WriteHeader(statusCode int) {
	if r.sentHeaders {
		slog.Warn(fmt.Sprintf("WriteHeader called twice, second time with: %d", statusCode))
		return
	}
	r.sendHeaders(statusCode)
}

func (r *responseBodyWriter) sendHeaders(statusCode int) {
	r.sentHeaders = true
	io.WriteString(r.conn, r.proto)
	r.conn.Write([]byte{' '})
	io.WriteString(r.conn, strconv.FormatInt(int64(statusCode), 10))
	r.conn.Write([]byte{' '})
	io.WriteString(r.conn, http.StatusText(statusCode))
	r.conn.Write([]byte{'\r', '\n'})
	for k, vals := range r.headers {
		for _, val := range vals {
			io.WriteString(r.conn, k)
			r.conn.Write([]byte{':', ' '})
			io.WriteString(r.conn, val)
			r.conn.Write([]byte{'\r', '\n'})
		}
	}
	r.conn.Write([]byte{'\r', '\n'})
}
  • Header() allows handlers to add headers to the response.
  • WriteHeader(statusCode int) writes out the status code line and any headers that the handler has added
  • Write(b []byte) writes some data and can be called multiple times for streaming the body back to the client. Note that once you start writing the body, the headers will automatically be sent.

And, finally, here’s how to start the server, which looks very similar to the last time (and many Go tutorials):

func main() {
	addr := "127.0.0.1:9000"
	mux := http.NewServeMux()
	// TODO: handlers go here
	s := Server{
		Addr:    addr,
		Handler: mux,
	}
	log.Printf("Starting web server: http://%s", addr)
	if err := s.ListenAndServe(); err != nil {
		log.Fatal(err)
	}
}

That’s it. We’re done with our server! See the full source at Github: main.go to see how it all fits together. Similar to last time, I also wanted to test this implementation to make sure it works well with browsers and other web tools.

Testing the Implementation

First off, to start the server, you can run this:

go run server/main.go

Handlers

For this time, I’ve added a few more handlers to test different aspects of our HTTP server. It’s more fun that way and there are just generally more things to test with HTTP/1.0, like status codes and headers.

/headers

mux.HandleFunc("/headers", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Add("content-type", "application/json")
    json.NewEncoder(w).Encode(r.Header)
})
  • Description: This handler echoes back all the request headers in JSON format. It also sets the Content-Type header to application/json to inform the client about the response format.
  • Testing Rationale: Useful for verifying that the server correctly receives and parses request headers. Can be used to test sending various headers with different values and case variations.
$ curl -H "My-Custom-Header: Hello, world!" http://localhost:9000/headers
{"Accept":["*/*"],"Host":["localhost:9000"],"My-Custom-Header":["Hello, world!"],"User-Agent":["curl/8.7.1"]}

You can even use netcat to craft the HTTP request manually:

printf "GET /headers HTTP/1.0\r\nMy-Custom-Header: Hello from nc!\r\n" | nc localhost 9000

Netcat is particularly useful at testing text-based protocols because you can write out exactly what the request will look like, which is great for testing unique situations that are hard to set up using existing tools like curl.

/status/{status}

mux.HandleFunc("/status/{status}", func(w http.ResponseWriter, r *http.Request) {
    status, err := strconv.ParseInt(r.PathValue("status"), 10, 64)
    if err != nil {
        w.WriteHeader(http.StatusBadRequest)
        io.WriteString(w, fmt.Sprintf("error: %s", err))
        return
    }
    w.WriteHeader(int(status))
})
  • Description: This handler allows you to request a specific HTTP status code by providing it in the URL path (e.g., /status/404). If the provided status is invalid, it returns a 400 Bad Request error.
  • Testing Rationale: Essential for testing how clients handle different status codes, especially error codes. Can be used to simulate various scenarios like successful requests, redirects, client errors, and server errors.
$ curl -I http://localhost:9000/status/404
HTTP/1.0 404 Not Found

/nothing

mux.HandleFunc("/nothing", func(w http.ResponseWriter, r *http.Request) {})
  • Description: This handler does absolutely nothing. It doesn’t write any headers or body.
  • Testing Rationale: Can be used to test the server’s behavior when a handler doesn’t explicitly send a response. It should still send a default 200 OK response with no body, as implemented in the responseBodyWriter.
$ curl http://localhost:9000/nothing

/echo

mux.HandleFunc("/echo", func(w http.ResponseWriter, r *http.Request) {
    defer r.Body.Close()
    io.Copy(w, r.Body)
})
  • Description: This handler echoes back the request body as the response body.
  • Testing Rationale: Useful for testing how the server handles request bodies, especially with POST requests. Can be used to verify that the server correctly receives and processes the body data.
$ curl -X POST -d "This is some data to echo" http://localhost:9000/echo
This is some data to echo

Serving up my blog

mux.Handle("/", http.FileServer(http.Dir("public")))
  • Description: This handler serves static files from the “public” directory. It’s a convenient way to serve HTML, CSS, JavaScript, and other assets.
  • Testing Rationale: Allows testing of how the server handles GET requests for files. Can be used to verify that it correctly serves different file types with appropriate headers (Content-Type, Content-Length).

It is quite amazing to see an entire website work just fine with this small amount of code.

Beyond HTTP/1.0

While HTTP/1.0 was a significant leap forward, the story doesn’t end there. HTTP/1.1, HTTP/2, and HTTP/3 brought further enhancements. In my next article, we’ll dive into the world of HTTP/1.1, exploring its advancements over HTTP/1.0; reusable connections, chunked encoding, and TLS will finally enter the scene. From here on out, the focus was less on forming the semantics of HTTP and more on improving performance.

Conclusion

By building this HTTP/1.0 server from the ground up, we’ve explored the key components that transformed the web into a more dynamic and interactive space. From the introduction of headers and status codes to the expanded set of HTTP methods, HTTP/1.0 laid the crucial foundation for modern web communication.

While this server is a simplified implementation, it captures the essence of HTTP/1.0 and gives you a hands-on understanding of how the protocol works. As you continue to explore HTTP, you’ll see how later versions, like HTTP/1.1 and HTTP/2, built upon this foundation, optimizing performance and enhancing features.

Remember, the journey of mastering web technologies is continuous. Whether you’re diving into newer protocols or digging deeper into foundational ones like HTTP/1.0, each step broadens your understanding and sharpens your skills.

]]>Y'all are sleeping on HTTP/3https://kmcd.dev/posts/yall-are-sleeping-on-http3/Tue, 06 Aug 2024 10:00:00 +0000https://kmcd.dev/posts/yall-are-sleeping-on-http3/ <p> <img hspace="5" src="https://kmcd.dev/posts/yall-are-sleeping-on-http3/cover_hu_2121eccfa92cb760.webp" /> </p> Wake up call

This post is about HTTP/3 and QUIC. If you don’t know what that is, there are many, many, many, many, many, many, many good resources that will get you up to speed. I’m writing this post to enlighten people on what has been happening in the last few years.

All major browsers support HTTP/3 now.

Most major cloud providers support HTTP/3 now.

Most major load balancers support HTTP/3 now.

In a short few years over 30% of web traffic is served with HTTP/3.

HTTP/3 isn’t the future. It’s the present.

Every time I mention HTTP/3 there’s always someone who pops up who’s completely unaware that it even exists, thinks that it’s some minor change or thinks that supporting HTTP/3 is cargo cult behavior. Every web developer should probably know that HTTP/3 exists because HTTP/3 is a giant change. HTTP/3 abandons TCP in favor of a channel-aware UDP-based protocol called QUIC. To me, this feels important! This feels like people need to be talking about it, doing more experiments around QUIC, and writing more tooling, security analysis and benchmarks. QUIC has the potential to dethrone TCP as the reliable layer 4 protocol. This is a big deal.

Why should you care about HTTP/3 if you’re not a big nerd who likes learning about network protocols? Because it promises to bring a lot of advantages: faster page loads, smoother video streaming, and more resilient connections. Plus, I think web developers need to know about the foundations of their profession. And this foundation has been changing. Rapidly.

What’s wrong with TCP?

Multiple streams: One Connection

There are issues with multiplexing multiple streams onto one TCP-backed connection. This is because TCP treats all data on a connection as one big stream. If one part of that stream gets delayed or lost, everything else gets stuck behind it, like a traffic jam on a single-lane road. This is called ‘head-of-line blocking,’ and it can significantly slow down web page loading, especially when multiple resources (like images, scripts, and stylesheets) are being requested simultaneously.

It isn’t good in dynamic network environments

As mentioned in the last section, TCP connections are slow to set up because of the number of round trips required. In environments that change often, causing your web clients to switch networks (and client IP addresses), the clients will have to re-establish an entirely new connection each time you switch networks. Think about the large pause that happens when you switch wifi networks while using video chat. That’s exactly this issue. TCP was not designed to handle this situation without that pause. In the following section, I’m going to tell you how QUIC (and HTTP/3) handles this situation in a much more reliable and smooth way. This is extremely relevant today in a world where mobile phones can choose to swap between multiple wifi and mobile networks.

Why can’t I use wifi and mobile at the same time?

This is actually possible with TCP, using a feature called multipath TCP, but the rollout has been slow and difficult for TCP. QUIC’s built-in support for connection migration and 0-RTT resumption offers a smoother and more efficient solution to this problem, potentially enabling true multipath connectivity in the future.

Enter: QUIC and HTTP/3

Faster Connections

First off, QUIC requires far fewer round trips to set up an encrypted connection. Instead of 3 needed for TLS+HTTP/2, QUIC just has one.

Zero Round Trip Time (0-RTT) Resumption

QUIC connections can be resumed, even if the client’s IP address changes. This allows connections to be re-established with zero round trips, so when you are on a conference call and switch wifi networks your HTTP/3 connection can resume instantaneously. While there have been security concerns around this feature, such as the potential for replay attacks, these have been largely mitigated through measures like rotating keys and client IDs.

I mentioned earlier how the rollout of multipath for TCP hasn’t been super successful. With the features to support 0-RTT, support for multipath is essentially built-in, so you may see more concurrent usage of wifi, ethernet and mobile networks in the future.

Multiplexing

In contrast to TCP+HTTP/2, QUIC ensures that packet loss or delays on one stream don’t impact others. This is because QUIC manages ordering guarantees at the individual stream level, not for the entire connection, effectively eliminating head-of-line blocking.

Improved Congestion Control

QUIC’s more responsive congestion control leads to faster recovery from packet loss.

“But I heard UDP is unreliable”

It’s a common misconception that UDP is inherently unreliable compared to TCP. TCP packets are just as unreliable, but they transparently get retried by TCP client and server implementations. In other words, both TCP and UDP packets can be lost or corrupted during transmission; the difference lies in how they handle those errors. While it is true that UDP doesn’t offer the same built-in guarantees as TCP, QUIC implements the same guarantees on top of UDP.

Think of it like this: TCP is like a delivery service that meticulously tracks every package and resends any that go missing. UDP, on the other hand, is more like a bulk mailer, sending out a flood of letters and hoping most of them arrive. However, QUIC builds its own tracking system on top of UDP, ensuring reliable delivery of data.

This means QUIC can enjoy UDP’s speed and flexibility without sacrificing reliability. In fact, in some cases, QUIC can even outperform TCP in terms of packet loss recovery, thanks to its advanced congestion control mechanisms.

So, while the “UDP is unreliable” mantra might have been true in the past, protocols built on UDP can be even more reliable than TCP.

Let’s see how far we’ve come

Okay, so I mentioned how available HTTP/3 is a few times now. But let’s look at the specifics: browser support, cloud support and support when self-hosting.

Web Browsers

Let’s start with web browsers. HTTP/3 support doesn’t matter if browsers don’t support the technology. CanIUse shows the support for each browser and it looks very good for all major browsers:

The only caveat is that Safari only enables HTTP/3 for a portion of users, but Apple will come around eventually.

Cloud Providers

Next, if you want to host websites with HTTP/3 it would be awesome if cloud providers supported HTTP/3. Well, you’re in luck. Most major cloud providers do support HTTP/3, including:

Load Balancers

But what if you’re setting up your own infrastructure, foregoing the cloud? What are the available options?

Is HTTP/2 already dying?

Take a look at the comparative usage of HTTP/2 and HTTP/3 over the last few years:

Source: w3techs.com

From this graph, you can see that in a short few years, HTTP/3 usage is rapidly approaching the same usage as HTTP/2. We had “peak HTTP/2” in 2021 and maybe next year we will see HTTP/3 overtake HTTP/2 to be the de-facto standard for new web deployments.

I will be honest though, other sources, such as Cloudflare and Internet Society Pulse, present a slightly different picture, with HTTP/2 still handling a significant portion of traffic.

Why don’t web developers know about this?

Well, simply put, none of this really intersects with them in any meaningful way. You can’t even tell what version of HTTP your browser is making from Javascript. No, not even with the Fetch API. I recently looked into this because I wanted to print out the version of HTTP being used when loading my website in a tool that I made but I had to resort to adding headers to the response via Cloudflare to get that information. Because of all of this, I’m certain that there’s a large number of web developers who have no idea that their website is using HTTP/3.

If you want to know which version of HTTP your browser is using when loading your website, most browser inspector interfaces can show this by going to the network tab, right-clicking the headers and adding “Protocol” to the list of columns to show. It also usually shows the protocol when you look at the details of each request.

Challenges Ahead

There are two main areas to focus on with QUIC: adding more tooling and language support for the protocol.

Tooling/Language Support

Even though browsers and load balancers have good support for QUIC, most programming languages don’t support HTTP/3 because QUIC presents a vastly different way to communicate. Without kernel support or widely supported bindings, adding QUIC to a language is a bit like re-implementing TCP in the language. TCP is usually relatively easy to add because OS kernels typically implement TCP for you and provide bindings. As far as I know, that isn’t the case for QUIC.

Will QUIC stay in userspace?

From what I’ve seen, there’s no well-supported kernel module for QUIC. There exist some projects that do add a kernel module for Linux but it doesn’t seem like it’s heavily used yet. TCP has benefitted a lot from being backed by OS kernels, which gives it performance optimizations not available in user space but that strength is also a weakness. TCP is now hard to change and adapt. It appears like QUIC has evolved much quicker because of its user-space implementations, but should it stay there? Some have reported that it’s possible to match the performance of TCP, even when running in userspace. So does QUIC need the performance improvements that are possible in the kernel? Should there be a hybrid approach that could give a balance of flexibility and performance advantages? To me, those are open questions and I’m curious how this will unfold over time.

Naysayers

Since QUIC does present a lot of changes, there are people, mostly network engineers, who do not like the quick adoption of QUIC. Here are some of the complaints:

  • Most web traffic is a bloated mess anyway, and the additional few milliseconds it takes to stand up a TCP connection doesn’t account for a large amount of bloat in modern web applications.
  • QUIC makes it more difficult for network administrators to monitor and inspect traffic.
  • QUIC induces more CPU overhead, as it doesn’t have decades of optimization that TCP has.
  • QUIC is a relatively new protocol, and there may still be undiscovered bugs or vulnerabilities, especially with so many languages/platforms re-implementing the protocol.

Further, many argue that the benefits just don’t outweigh the risks. I don’t think I agree. I think these network engineers are discounting just how much traffic comes from mobile or dynamic (sketchy) wifi environments. However, I am surprised at how quickly and the method by which QUIC has been rolled out. Should we pump the brakes or is it too late?

The Future is QUIC

HTTP/3 and QUIC are more than just incremental improvements; they represent a fundamental shift in how we build the web. By overcoming the limitations of TCP, QUIC offers a faster, more reliable, and more secure foundation for modern Internet communication.

The benefits are clear: faster page loads, smoother video streaming, more responsive applications, and seamless transitions between networks. This translates to a better user experience, increased engagement, and ultimately, a more vibrant and dynamic internet.

But QUIC’s potential extends far beyond the web. Its ability to handle real-time communication, coupled with its built-in security features, makes it an attractive option for applications like online gaming, video conferencing, and more. There are already some big projects built on QUIC:

While the transition to HTTP/3 is well underway, it’s far from complete. There are still challenges to overcome, such as improving tooling, expanding language support, and ensuring compatibility with existing infrastructure. And it remains to be seen if HTTP/3 is ready for mass adoption.

However, the momentum is undeniable. With major browsers, cloud providers, and load balancers embracing QUIC, the future is bright. As developers and engineers continue to innovate and build on this foundation, we can expect to see even more exciting possibilities emerge.

So, what are you waiting for? If you’re a web developer or tech enthusiast, now is the time to dive into QUIC and start exploring its potential. Experiment with new applications, build innovative tools and share your findings with the community. I’ve started doing this by trying HTTP/3 with gRPC. The future of the internet is being written in QUIC, and you can be a part of it.

If you want to learn more, here are a lot of resources that I used to research for this article:

]]>HTTP/0.9 From Scratchhttps://kmcd.dev/posts/http0.9-from-scratch/Tue, 30 Jul 2024 00:00:00 +0000https://kmcd.dev/posts/http0.9-from-scratch/ <p> <img hspace="5" src="https://kmcd.dev/posts/http0.9-from-scratch/cover_hu_72f2d6eeec83ba36.webp" /> </p> Building the foundation with HTTP/0.9 Introduction

Welcome to the first installment of our “HTTP from Scratch” blog series! In this series, we’ll embark on a journey through the evolution of the Hypertext Transfer Protocol (HTTP), the backbone of the World Wide Web. By building simple implementations of each major HTTP version in Go, we’ll gain a deep understanding of how this essential protocol has shaped the internet we use every day and how it has evolved to what we have now. Be warned that none of the code will be the most performant, secure or featureful.

In this post, we’ll travel back to the early days of the web (1991) and explore HTTP/0.9, HTTP’s initial incarnation. At the time, HTTP/0.9 was a groundbreaking technology that enabled the first web browsers and servers to communicate, laying the foundation for the World Wide Web that we know today. But HTTP/0.9 had its limitations. We’ll discuss these shortcomings, which ultimately paved the way for subsequent versions of HTTP that introduced features like headers, status codes, and support for additional HTTP methods, connection reuse, binary framing and, eventually, abandoning TCP for UDP for better reliability and performance. For a more formal description of the HTTP/0.9 specification, you can reference http.dev or w3.org.

To get a hands-on understanding of HTTP/0.9, we’ll take a practical approach. Since no modern web servers support this early version, we’ll create our own HTTP/0.9 server from scratch using Go. This will allow us to experiment with the protocol and gain valuable insights into its inner workings.

By the end of this post, you’ll have a solid grasp of HTTP/0.9 and the foundation it laid for the modern web. You will be well-equipped to continue our journey through the evolution of HTTP in the upcoming articles.

Understanding HTTP/0.9

HTTP/0.9, the inaugural version of the Hypertext Transfer Protocol, laid the groundwork for the web as we know it today. It introduced a simple request-response model that facilitated communication between web clients and servers.

The Request

In HTTP/0.9, a client sends a single-line request to the server. This request line consists of only two components:

  1. GET Method: The only method supported in HTTP/0.9. It instructs the server to retrieve the specified resource.
  2. Resource Path: The path to the resource on the server that the client wants to access.

For example, to request the index.html file from the server’s root directory, the client would send:

GET /index.html

That’s… it. It can’t get much simpler than that. Note that there are no headers to indicate content size, compression, content type, etc. This isn’t a part of HTTP/0.9 as all of that was introduced in HTTP/1.0.

The Response

Upon receiving a request, the server responds with the contents of the requested resource. This response is a simple stream of bytes, usually representing an HTML document. Notably, there are no headers in an HTTP/0.9 response or a status code that tells us if we’re receiving an error page or the resource that we asked for. Since there were no status codes, there was no way to indicate if a requested resource was not found — a concept that would later be introduced with the 404 status code.

Full example

> GET /index.html
< <html>
< Hello World!
< </html>

Again, this is an extremely simple protocol at this point. Request a resource and get the data. There’s no place to put metadata when requesting or responding. By the way, lines prefixed with > are from the client to the server and < are from the server to the client. Remember this, because this is important for understanding verbose curl output, which we’ll use a good amount in the future.

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: TCP SYN
        Server ->> Client: TCP SYN-ACK
        Client ->> Server: TCP ACK
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end

    rect rgb(47,75,124)
        Server ->> Client: TCP FIN
		Client ->> Server: TCP ACK
    end

Limitations

While revolutionary for its time, HTTP/0.9 had some significant limitations that paved the way for future improvements:

  • No Headers: The absence of headers meant that the server could not convey any additional information about the response, such as content type or length.
  • Only GET: The only supported method was GET, which meant that clients could only request resources and not submit data to the server.
  • No Status Codes: There was no mechanism to indicate errors or other status information.

Implementing an HTTP/0.9 Server

Okay, now let’s implement the server.

type Server struct {
	Addr    string
	Handler http.Handler
}


func (s *Server) ListenAndServe() error {
	if s.Handler == nil {
		panic("http server started without a handler")
	}
	l, err := net.Listen("tcp", s.Addr)
	if err != nil {
		return err
	}
	defer l.Close()

	for {
		conn, err := l.Accept()
		if err != nil {
			log.Fatal(err)
		}

		go s.handleConnection(conn)
	}
}

You can see here that our ListenAndServe() listens on the configured TCP port and then loops forever, accepting and handling new connections in separate goroutines. Now let’s look at s.handleConnection()

func (s *Server) handleConnection(conn net.Conn) {
	defer conn.Close()

	reader := bufio.NewReader(conn)
	line, _, err := reader.ReadLine()
	if err != nil {
		return
	}

	fields := strings.Fields(string(line))
	if len(fields) < 2 {
		return
	}
	r := &http.Request{
		Method:     fields[0],
		URL:        &url.URL{Scheme: "http", Path: fields[1]},
		Proto:      "HTTP/0.9",
		ProtoMajor: 0,
		ProtoMinor: 9,
		RemoteAddr: conn.RemoteAddr().String(),
	}

	s.Handler.ServeHTTP(newWriter(conn), r)
}

Handling a connection is very simple in HTTP/0.9 because clients can only create a new connection, send a single request and then the connection is closed. This code reads the first line given by the user and then calls strings.Fields to split the different parts of the request up. As a reminder, this is what the request looks like in HTTP/0.9:

GET /path/to/resource

Okay, now let’s look at what newWriter does. ServeHTTP expects a http.ResponseWriter, which looks like this:

type ResponseWriter interface {
	Header() Header
	Write([]byte) (int, error)
	WriteHeader(statusCode int)
}

Here’s what our HTTP/0.9 looks like:

type responseBodyWriter struct {
	conn net.Conn
}

func (r *responseBodyWriter) Header() http.Header {
	// unsupported with HTTP/0.9
	return nil
}

func (r *responseBodyWriter) Write(b []byte) (int, error) {
	return r.conn.Write(b)
}

func (r *responseBodyWriter) WriteHeader(statusCode int) {
	// unsupported with HTTP/0.9
}

func newWriter(c net.Conn) http.ResponseWriter {
	return &responseBodyWriter{
		conn: c,
	}
}

The important thing to note here is that HTTP/0.9 doesn’t support status codes or headers so we don’t need to do anything in Header() and WriteHeader(statusCode)

Now let’s put it together in a main function:

func main() {
    addr := "127.0.0.1:9000"
	s := Server{
		Addr: addr,
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			w.Write([]byte("Hello World!"))
		}),
	}
    log.Printf("Listening on %s", addr)
	if err := s.ListenAndServe(); err != nil {
		log.Fatal(err)
	}
}

Note that the HTTP handler is a normal-looking http.Handler so this server could work with any existing HTTP router or framework.

See the full source at Github: main.go.

Testing the server

Now we just need to run the server:

$ go run server/main.go
2024/07/27 21:55:28 Listening on 127.0.0.1:9000

Now that we have an HTTP/0.9 server, how do we test it?? Since HTTP/0.9 pretty much isn’t used anywhere, how do find a client to test this server? Luckily, curl supports HTTP/0.9, so let’s try that!

$ curl --http0.9 http://127.0.0.1:9000/this/is/a/test
Hello World!

The –http0.9 flag instructs curl to accept the headerless responses of HTTP/0.9.

You should note that curl is sending a request as if it were HTTP/1.1 but is configured to accept the headerless responses of HTTP/0.9. Here’s what the curl manpage says about the flag:

--http0.9
       (HTTP) Tells curl to be fine with HTTP version 0.9 response.

       HTTP/0.9 is a completely headerless response and therefore you can also connect with this to non-HTTP servers and still get a response since curl will simply transparently downgrade - if allowed.

I verified this behavior a bit more when I mixed --http1.0 and --http0.9. Instead of using HTTP/1.1 in the first line of the request, curl declares that it is using HTTP/1.0 while still treating the body correctly (not expecting a status code or headers):

$ curl --http1.0 --http0.9 http://127.0.0.1:9000/this/is/a/test
Hello World!

By the way, if you don’t use the --http0.9 (or if your server returns the status code/headers in a format that doesn’t make sense) you will receive an error message, “Received HTTP/0.9 when not allowed”:

$ curl http://127.0.0.1:9000/this/is/a/test
curl: (1) Received HTTP/0.9 when not allowed

You can test this server with simpler tools. For example, netcat (ncat), can be used to make this same request. This will be a theme for the text-based protocols where often it’s simpler just to write text out directly to netcat than it is to use other kinds of tooling:

$ echo GET this/is/a/test | ncat 127.0.0.1 9000
Hello World!

After writing most of this article, I finally realized that I never even attempted to test my web server using a real web browser. It seems like the only major browser that still supports HTTP/0.9 is Firefox, so let’s see it!

Yes, it works! Success!

Implementing an HTTP/0.9 Client

Now that we’ve made a server and used existing clients, we might as well make a client in Go. Don’t worry, this one is super simple:

conn, err := net.Dial("tcp", "127.0.0.1:9000")
if err != nil {
	log.Fatalf("err: %s", err)
}

if _, err := conn.Write([]byte("GET /this/is/a/test\r\n")); err != nil {
	log.Fatalf("err: %s", err)
}

body, err := io.ReadAll(conn)
if err != nil {
	log.Fatalf("err: %s", err)
}

fmt.Println(string(body))

See the full source at Github: main.go.

This code does the following:

  • Establishes a TCP connection to the server.
  • Sends the HTTP/0.9 request.
  • Receives the response and displays it to the user.

Simple, right? Almost too simple.

$ go run client/main.go
Hello World!

Conclusion

In this post, we delved into the origins of the web by exploring HTTP/0.9. While simple in its design, HTTP/0.9 was a groundbreaking protocol that enabled the first web browsers and servers to communicate.

We explored the basic request-response model of HTTP/0.9, understanding how clients could request resources and servers could deliver them. We also acknowledged the limitations of this early version, including the lack of headers, status codes, and support for other HTTP methods besides GET.

By building a rudimentary HTTP/0.9 server and client in Go, we gained hands-on experience with the protocol’s core concepts. We learned how to handle TCP connections, parse requests, and send responses, laying a solid foundation for understanding more advanced HTTP versions.

In the upcoming parts of this series, we’ll delve into how HTTP evolved to overcome the limitations of HTTP/0.9 and address the needs of the evolving World Wide Web. We’ll explore the introduction of headers, status codes, and additional methods, which enabled more robust and feature-rich communication between web clients and servers. Stay tuned for the next part of our series, where we’ll dive into HTTP/1.0 and its significant enhancements over HTTP/0.9. As a sneak peek, these are the major features added to each version:

  • HTTP/0.9: First attempt to transfer generic resources by paths
  • HTTP/1.0: Adds status codes, headers, verbs
  • HTTP/1.1: Adds connection re-use so connections can be reused
  • HTTP/2: Switches from a text-based protocol to binary
  • HTTP/3: Built on QUIC instead of TCP
]]>What version of HTTP are you using?https://kmcd.dev/posts/http-tool/Tue, 23 Jul 2024 10:00:00 +0000https://kmcd.dev/posts/http-tool/ <p> <img hspace="5" src="https://kmcd.dev/posts/http-tool/cover_hu_ae04e3ed28fd1220.webp" /> </p> Find out if your browser using the latest and greatest. Click here to see what version of HTTP your browser is using to load this website.

HTTP/3

HTTP/3 is the newest version of the protocol for powering the web. It offers the same features as HTTP/1.1 and HTTP/2 but has a drastically different foundation: using a protocol called QUIC that is built on UDP instead of using the good-ol TCP protocol.

Recently I’ve been talking about HTTP/3 and how it might be good to use it with gRPC. That’s a good idea, but based on many of the comments that I saw in response to that post I feel like many people don’t know how much work has been done for HTTP/3 already. The HTTP/3 spec was published by the IETF and has since been adopted by so many companies. Here’s just a few:

  • Cloudflare
  • Google Cloud CDN and Load Balancer
  • AWS CloudFront
  • Akamai CDN
  • Azure CDN
  • CDN77
  • Fastly CDN
  • Azure Application Gateway
  • nginx
  • Caddy
  • HAProxy
  • and tons more

So as you can see, the server side of HTTP/3 is getting more and more ready for prime time. And the browser support is also pretty much there. You can see on caniuse.com that HTTP/3 is supported on all major browsers including all of the familiar names: Chrome, Firefox, Safari, and Edge.

So because web browsers and many web load balancer services already have support for HTTP/3 it shouldn’t be surprising to know that 30% of websites now support HTTP/3. This is an impressive figure, given how different the foundations of HTTP/3 are compared to HTTP/2 and HTTP/1.1.

As a reminder, here are the benefits of HTTP/3:

  • Faster connection establishment, which is especially useful with slower, unstable or congested connections.
  • Completely avoids the head-of-line blocking problem, allowing browsers to use a single connection to load everything at full speed.

The tool

So, are you curious to know if your browser is leveraging the latest web technology? I built a simple tool that will reveal the HTTP version you’re using to access this website. It’ll even let you know if you’re enjoying the full benefits of HTTP/3’s speed and efficiency. Give it a try and see how your browser stacks up!

How It Is Made

This is a static blog, which obviously makes it a little harder to make a tool like this. In addition to that, the protocol used to connect from the load balancer isn’t necessarily the protocol used by clients. In my case, I use Cloudflare as a host and load balancer, so I configured the endpoint to add a new HTTP header that contains the HTTP version used between the client and load balancer. It might look like:

x-kmcd-http-request-version: HTTP/3

Javascript/Browser Limitations

So, I have this new header being returned, but I need to do something to display the value. One thing that’s a little crazy to me is that Javascript on a page cannot read the headers that were returned for the current page. So… When you load the page, Javascript runs and calls the Fetch API to retrieve the page again. The response object that is returned does contain the full list of headers, which is what I used to determine the output. By the way, you might ask “doesn’t the response object also contain the version of HTTP used?” and the answer is, unfortunately, no. It seems like it would be simple to add but it’s just not there.

Check it out

So go check out the tool. If you don’t get HTTP/3 at first you may have to refresh a few times. There are a few different ways to hint to browsers that HTTP/3 is supported and I use all of them… but many browsers will avoid attempting HTTP/3 for the first few requests to a website until it trusts that HTTP/3 is available. So go on and try it and let me know if it’s useful at all:

Click here to go to the tool.

]]>Texans in Denmarkhttps://kmcd.dev/posts/texans-in-denmark/Tue, 16 Jul 2024 10:00:00 +0000https://kmcd.dev/posts/texans-in-denmark/ <p> <img hspace="5" src="https://kmcd.dev/posts/texans-in-denmark/cover_hu_5f6abe097a46c21.webp" /> </p> Why I decided to go from BBQ to Smørrebrød. In my last post, I talked about the reasons that me and my family decided to leave Texas. Today I’m going to talk about why we decided Denmark as our new home. Why Denmark? I get this question often, especially from Danes. I hope to answer how we came up with Denmark as our family’s new home and how it has been in the past two and a half years. I do not intend for this to be a guide for others since the details of bureaucracy change quite often and I think everyone’s experience will be very different from ours. My colleague has actually written a Developers Guide to Moving to Denmark which is very informative on this front. My of his experiences echo mine. Instead, I want to describe “the vibe” of the last few years in Denmark, so others might have a general sense of what it might be like to make the same life-changing move.

Why Denmark?

To give you a better idea of why we chose Denmark, let’s dive into the specific factors that influenced our decision.

Beautiful cities and landscapes

I love the design of many European cities. The historic buildings, the walkable city centers, the metro and train systems, the bus systems that get to most places, etc. So Europe was already very high on our list. However, there are many places to live in Europe, so let’s continue.

English Usage

We did realize how hard it would be to live in a country that doesn’t have a high proficiency in English, so we naturally gravitated towards places that do use English a good amount. With this criteria, we identified Ireland, England, The Netherlands, Denmark, Sweden and Norway as notable choices.

Women also work

Denmark has a very good record for women being able to have careers. The gap between the percentage of women and men working is much lower than in most other countries. Women are very much expected to be a part of society, and men are expected to do their share of housework and childcare.

Work-life Balance

Denmark has a legal requirement for employees to have five weeks of paid vacation time per year. Many jobs offer a week or two more than that. Non-salary employees also get “vacation pay” which can be used to keep your pay consistent while you take time off for you and your family. Furthermore, the amount of hours for full-time work in Denmark is 37, which is slightly better than the typical 40. People shouldn’t be working their lives away in hopes that one day they will be able to retire and finally do the things that they want to do when their health is likely the worst.

But how does Denmark make this work? How is it possible for so many women to work when there are children to raise? In addition to generous leave policies, Denmark also provides very generous maternity and paternity leave with up to a year off in total. When the child is one year old, subsidized child care starts and is extremely cheap for the quality of care that you get. And from then on, both parents have time to work full-time jobs and continue their careers. It just makes sense. Most citizens here are happy to pay more tax to live in a society where this is how it works. This very much matches my values.

Public health care

Health care is a touchy subject for everyone, but I firmly believe that making it a public service is the right choice. I’m tired of seeing fundraising campaigns to cover basic medical expenses. Your level of care shouldn’t be decided on how good of a sob story your relatives can give on GoFundMe. I would rather have a giant fundraiser that everyone, by default, pays into and everyone can tap into if they need. Health care in our new country should be mostly a public service. This is the case for many countries so this criterion didn’t actually narrow our decision down very much, but this is a notable difference from The United States where health care is an abomination of private hospitals negotiating with private insurance companies to which you only get access to by being a productive worker at a (usually) private company.

Happy Citizens

It feels silly, but the fact that Denmark consistently ranks near the top of the happiest citizens tells me something about the society. Yes, this is likely a flawed study but there seems to be some truth to it. People are generally happy and the poverty rate is low. This matches my values.

Weather

You may find it surprising that I would list the weather as a positive for Denmark. In contrast to Texas, it’s nowhere near as extreme. There are few-to-no tornados, earthquakes or hurricanes. Occasionally there is a storm with strong winds with flooding but it is nowhere near the amount of destruction of tornados. Why do I care about tornados so much? My house was hit by one in 2019. I was alone with my 6-month-old daughter and it was the scariest few minutes of my life that shook up our lives for the next year. I talked more about this in my last post.

Biking culture

Denmark has a strong bicycle culture. What I like most about Danish cycling culture is that it’s not just the super extreme cyclers who have insanely expensive bikes and full-body suits. It’s mostly just normal people who use a bike to get to work because it’s the cheapest, fastest and usually the most pleasant way to get places.

And more

  • Literally Vikings
  • Our daughter will be bilingual, which is super cool
  • World leaders in wind energy
  • Wonderful design aesthetic
  • Delicious pastries
  • Low crime rate
  • Very progressive social policies
  • Probably a lot more things that I am just forgetting to list

All in all, these factors combined painted a picture of Denmark that aligned with our values and aspirations, making it the ideal place for our family to put down roots. So we took a leap of faith and decided to jump into the unknown.

Preparing for the move

We decided to NOT do a “scouting” vacation, so the first time we were to visit Denmark, we were moving there. This seems like a strange decision to most people that we talk to, but I firmly think we did the right thing for us. We reasoned that we weren’t going to learn anything from a vacation that would stop us from at least trying to live there for a year. Vacationing is such a different experience than actually living somewhere, and it really is hard to imagine what it is like living somewhere unless you just do it. So we did. The timing of our move was also a crucial factor.

You may ask why we moved when we did. This was the second half of 2021. COVID was still a big deal. Our daughter was two years old at the time, and we considered that it would get harder and harder for her to learn Danish like a native Danish kid the longer we waited. This was the biggest reason we pulled the trigger as fast as we could.

Three months before the move

Before our big move, we had a lot of things to settle. We rented an Airbnb for three months so that we could have time to sell almost everything that we owned. Our house. Our cars. And quite literally almost everything else. We took a total of 8 large suitcases, 3 carry-on bags and a stroller (which was destroyed in transit). We packed some clothing, practical items and a few keepsakes. That is all we had to start our new life.

During these three months, I also interviewed for jobs in Denmark. This could be a post all of its own, but when I decided to start applying to jobs in earnest was July of 2021. One thing I didn’t know about Denmark at the time is that the entire month of July is typically when a lot of Danes take off for their summer vacations. This is because most schools have three weeks in July designated for vacations. I wasn’t getting a lot of responses simply because the country was quite literally closed for business. But I stuck to it and eventually got traction on a few jobs which eventually ended up in a pretty good offer that I signed. I quite literally had one of my interviews with someone from their summerhouse. After the offer was agreed to, I started the process for my permit. This required that I travel to Houston to officially submit my paperwork and to have my fingerprint scanned and picture taken. Luckily the closest Danish embassy was relatively close.

At this time I was still working at my dream job at Apple. I only left the job so we could expedite the move to Denmark. I did try searching for openings in Denmark within Apple but there really aren’t a lot of Apple jobs in Denmark that match my skills. I was interviewing for jobs in The Netherlands as well since it was a very close 2nd place on our short list of places to move to. I did find several opportunities in Amsterdam and I was close to accepting one of them whenever a few people in Denmark decided to check their email while on vacation, thankfully. If things were slightly different, I could be learning Dutch instead of Danish right now.

Onward!

With a job offer secured, the work permit process underway, and plane tickets booked, it felt like our plans were finally coming together. As we prepared to leave everything familiar behind, we knew that this was just the beginning of our adventure.

The day finally arrived, and we embarked on our journey to Denmark.

Arrival in Denmark

Day 1

Oddly, the first day was the hardest for me. I can’t sleep very well on airplanes so it felt like the longest day on earth. We had two layovers and by the time we got on the third plane, my two-and-a-half-year-old daughter was COMPLETELY over the experience. We were the kind of parents that severely limited our daughter’s television-watching time. This has been great so far… Except when you need to entertain a toddler who isn’t interested in TV shows for 14 fucking hours… then it’s not so great. She also was far less interested in the experience of being in an airplane than we expected.

After our plane had finally landed at Copenhagen Airport, we had to get to our Airbnb (we had one rented for a month while we got our rental house settled) to check in and get our keys. Remember that we had several large and heavy suitcases to contend with and this is when we learned that the “1st floor” is up one and a half flights of stairs because in most of Europe the “first floor” is one level above the “ground floor”. We picked the most fun way to discover this. We also were scheduled to get the keys for the rental house soon after we arrived. This schedule was a mistake. Everything took much longer than expected so I was in a rush to get all of our luggage up to the Airbnb in time to get to our appointment. I moved everything we owned up a few flights of stairs before quickly running/walking to the nearby rental house. We ended up making it only a few minutes late. Everything went well with the walkthrough and we headed back to the AirBnB where we slept the first night in our new home country.

I have some pictures of this day but we look absolutely exhausted so I will not be sharing those!

The First Month

On the second day, we bought some bikes, determined to learn how to ride them. Yes, it was pouring rain, so we got the proper Danish welcoming but we didn’t let a little rain get in the way of getting things done.

We mostly treated the first few weeks, or so, like a vacation. We walked around and explored our new city like tourists. Then we would come home and build Ikea furniture in our rental house. We ended up only sleeping at the Airbnb for a few days. Once we had a mattress in our rental house there was no way we weren’t going to sleep there. We should have realized this but we added a lot of buffer into our plans because the nightmare scenario for us was being stuck without a place to sleep.

We ended up walking a lot. It’s quite insane how much more walking you end up doing whenever most places that you want to go are a walkable distance away or there’s transit available that’s a walkable distance from your destination. I mentioned cycling culture as a positive for the country, but I did not know how to ride a bicycle at this point. So we ended up learning how to use the bus and metro system. We opted to NOT purchase a car. This ended up being an amazing decision for us because Copenhagen is perfectly navigable without needing a car.

Bureaucracy is a little difficult when you don’t speak the language. In Denmark there is a specific order of when you can do certain things… and some steps took a surprisingly large amount of time, like onboarding to a Danish bank. It was a ridiculous amount of time before I was able to receive my first paycheck (3 months). We luckily planned on much worse things and had plenty of savings to use in the meantime.

As the initial excitement of arrival settled, we began to navigate the realities of daily life in Denmark. The first year was a whirlwind of learning and adjusting.

The first year

The first year went by so quickly. I was super excited about every little cultural difference. It sounds very strange but grocery shopping took a while to adjust to. Not only was everything in a language we didn’t understand (we expected that) but it’s strange how much you rely on familiar types of packaging and brands when shopping. It’s funny because now I can’t even sympathize too much with my past self because I no longer have any issues with grocery shopping. I don’t even remember the specifics behind what was so difficult… But I do remember having a pasta dish using extremely spicy marinara sauce on accident; the dish ended up being inedible.

I started my new job. We figured out how to sign my daughter up for kindergarten. My wife got a part-time job near home. We rebuilt our wardrobe since we weren’t able to take a lot of clothes over with us. I took a stab at learning Danish but I did this with a fully online class and none of the lessons “stuck” for me, and life ended up getting in the way so I would consider this attempt as a failure.

The weather did take some adjustment. We intellectually knew that the winters would be dark but there’s nothing that really prepares for so many dark days. There was a time soon after New Year’s when I just wanted it to stop being so dammed dark all the time. But other seasons eventually came. This is when I realized that Denmark may have the perfect spring, summer and autumn weather for me so maybe I can tolerate the dark winters.

Overall, the first year was a period of rapid adaptation and cultural immersion. While we faced challenges, we also embraced new experiences and began to feel a sense of belonging in our adopted home. Throughout the year, we encountered several surprising differences compared to life in Texas.

Surprising differences that I have noticed during the first year:

  • Very rare small talk with strangers. To me, it seems like the people who do strike up small talk are not from Denmark.
  • Americans are loud. I thought of myself as a quiet person but I often felt like I was scream-talking compared to those around me. I have (hopefully) adjusted my speaking volume lower.
  • Transit etiquette: everyone is extremely quiet.
  • Copenhagen can be incredibly quiet in general. People talk at low volumes, cars don’t go incredibly fast and there are no air conditioning units everywhere that pump out large amounts of background noise. It’s often quiet. And pleasant.
  • No tipping. I don’t miss tipping culture.
  • Fireworks: Danes are absolutely insane about fireworks. They fire them off at random times and places in the city in the weeks leading up to New Year’s. It’s absolutely crazy and the smoke and noise can make an otherwise quiet night look like a war zone. It all builds up in a crescendo on New Year’s evening to what can only be described as the noise of World War III. Dogs don’t have a good time in Danish cities during this time.
  • Feeling of safety; my wife often has to work late nights and has never felt unsafe walking home alone. This is a dramatic difference from where we come from.
  • Children with cigarettes and walking beer. I will never get used to this. The lack of open carry laws and the drinking age for beer is age 16.
  • Danes don’t seem to like using weed killer to kill grass that sprouts up between paving stones. Instead, you will see old people with a hand-held frame thrower burning off the unwanted plant matter.
  • No car; I haven’t missed having a car. It’s so freeing to not have one in a city with many alternative modes of transportation.
  • Weird traditions: During a holiday called Fastelavn there’s a tradition of getting a barrel with a black cat in it and beating the barrel until the cat gets out. They no longer use a cat but the barrel remains, usually filled with candy instead. There’s also a Midsummer holiday called Sankt Hans Aften where Danes gather to burn witches. Again, they no longer use real witches but the symbolism is kind of nuts when you first hear about it.
  • We would come across pikes of cinnamon on the ground close to poles. We asked around to see what the hell was happening here and discovered a fun tradition: If you aren’t married by the time you are 25 years old your “friends” will tie you to a pole and throw cinnamon at you to make you sweeter so you’ll have better luck finding a mate. It’s, of course, voluntary and all in good fun.

Despite these cultural adjustments, the second year brought a sense of belonging and a deeper understanding of Danish life.

The second year

As we settled into our second year, we began to feel more at home in Denmark. Many aspects of our lives fell into place, giving us a sense of stability and belonging.

Permanent home

Our lease was running out for our rental house so we decided to look for a more permanent place to stay. We ended up finding an apartment that we loved and we went through the process to purchase it. We’ve learned that there are a lot of caveats when purchasing real estate in Denmark as a non-permanent resident: financing is harder to get and you will likely be offered a worse rate unless you put a larger down payment (up to 40%), the Danish government can just say “no” if you haven’t lived in Denmark for more than 5 years, and you can’t rent out or sublet the place until you’ve lived here for five years. Despite these potential hurdles, the process went surprisingly smoothly for us, thanks in part to the streamlined Danish system and the assistance of a knowledgeable lawyer. We have found that the home-buying process is far more streamlined and easier than the process in the United States. I don’t think I physically signed a single thing during the entire process and there were no random $500 charges that popped up during the process.

Obviously, we had another round of Ikea orders to complete our new home.

Obviously, we had another round of Ikea orders to complete our new home.

Learning Danish

I’m going to Danish class in person now and it has been working a lot better for me compared to online classes. I’m at a weird skill level now where I can understand a good number of basic conversations but it’s still challenging for me to form sentences on the fly and all of it is still exhausting. I think this is the point where I can start saturating my mind with Danish shows, newspapers, magazines, books and music so my brain starts thinking in Danish more naturally. But it’s going well. My daughter is 5 years old now and is speaking Danish fluently. Our plan of bringing her over early enough to grasp the language like a native speaker has worked exceedingly well.

Unions

I haven’t mentioned unions yet. Denmark has unions for most industries and several general-purpose unions. They appear to operate a bit differently than what we see in the US but the principal is the same. Unions negotiate with employers for better working conditions and pay. They will also review employment contracts for you and if there is ever a disagreement between you and your employer you can get legal assistance from them. This system appears to work well in Denmark. There’s no government-enforced minimum wage, but there are often industry-specific minimums that have been agreed upon.

Work

I also switched jobs. After living in Denmark for a year I’ve gotten a better grasp of the market and made a move to get closer to the amount I should be getting paid. Part of this was due to data released by my union. As an ex-Apple software engineer from the United States, I took a deep pay cut coming to Denmark but it works out well because of our mix of significant savings and our severely downscaled lifestyle. Furthermore, the Danish work culture places a high value on work-life balance and employee well-being, which more than compensates for the lower salary.

Cycling

The second year is when I started reliably riding my bike in earnest. As my legs got stronger I transitioned from a heavy e-bike to a normal city bike. It was stolen once (I recovered it thanks to a hidden AirTag). I got tired of carrying around the battery. I cycle 2-3 times a week to work and Danish class. It is still tiring but that is because I can’t seem to force myself to go slower than my “max speed” when cycling. I always seem to want to go at a pace that pushes my limits. I also continued to cycle throughout all of winter, which was a big accomplishment for me; only opting to take the metro if there was ice on the roads or there were particularly windy days. You can see what my commute looked like at the time by checking on the video I made while commuting to work. I’ve been intending to make a new video because my office location has changed.

Youtube Link (if embed doesn’t work)

Cultural Differences

I listed a bunch of cultural differences in the last section. Well, there’s not much to note here because I don’t come across them often anymore… or at least not many of these things seem as strange anymore.

What now?

Our family is set on staying here for the long haul. There are some big dates to keep an eye on: passport renewals, visa extensions for me and my family, and applying for permanent residency.

Other than waiting, I have two requirements that I need to actively work on for permanent residency: pass a higher level of Danish and pass an active citizen exam. Then I wait for my 4 years of employment to pass and I can apply. For citizenship, there’s a much more difficult exam and I have to wait 9 years before I can apply. These requirements can and often do change.

As we look ahead, I have a lot of hope that Copenhagen will continue to be an amazing place to live. While the path to permanent residency and eventual citizenship may be long and winding, we are confident in our decision to make Denmark our home. We eagerly await the milestones ahead, embracing the opportunities for growth and the chance to become even more integrated into this vibrant and welcoming society. While we may always be Texans at heart, we are excited to continue to forge a new life in Denmark filled with new experiences, new friendships, and a newfound appreciation for the simple joys of Danish living.

]]>gRPC Over HTTP/3https://kmcd.dev/posts/grpc-over-http3/Tue, 09 Jul 2024 10:00:00 +0000https://kmcd.dev/posts/grpc-over-http3/ <p> <img hspace="5" src="https://kmcd.dev/posts/grpc-over-http3/cover_hu_6bb864e66717cc70.webp" /> </p> Turbocharging gRPC with HTTP/3 Introduction

At the time of writing, HTTP/3 is supported by 30.4% of the top 10 million websites. This market penetration is astounding, but it seems like all of this progress has been possible almost exclusively by work on browsers, load balancers and CDN providers. What about the backend? How’s HTTP/3 doing there? The answer, sadly, is not as incredible.

Because of this, I have been very interested in HTTP/3 in the context of gRPC. While gRPC has been instrumental in driving the adoption of HTTP/2, the same isn’t true for HTTP/3 even though HTTP/3 promises several benefits that all seem to apply exceptionally well to gRPC services.

In this post, we’ll dive into what HTTP/3 is and explore the compelling reasons why it’s an ideal fit for gRPC applications. We’ll uncover the technical advancements that promise to make HTTP/3 faster, more reliable, and more secure. But we won’t stop at theory; we’ll get our hands dirty with practical examples in Go, demonstrating how to set up and test gRPC servers and clients over HTTP/3.

By the end of this journey, you’ll have a solid understanding of the benefits HTTP/3 brings to gRPC, the tools available to start using it today, and the potential it holds for the future of API development. So, fasten your seatbelts and get ready to experience the next generation of network protocols!

All code examples are available at sudorandom/example-connect-http3 on GitHub.

Why HTTP/3

gRPC has had a lot of success pushing the world into HTTP/2 but there are some advantages to pushing even further and adopting the new major version of HTTP, HTTP/3. Let’s discuss these advantages before delving into code examples. So why should we use HTTP/3?

Faster Connection Establishment

HTTP/3 is built on QUIC (Quick UDP Internet Connections). You can think of it as a similar protocol to TCP but as the name suggests, it is built on top of UDP. While TCP has historically been the foundation of web communication, offering sequential packet ordering, congestion control and retransmissions, the needs of web browsers have evolved to require some different features and tradeoffs. All three main features of TCP—sequential packet ordering, congestion control, and retransmissions—can sometimes hinder the performance of modern web applications. Acknowledging these trade-offs led to the development of QUIC and HTTP/3, which both leverage UDP.

As the internet evolved better security practices, we’ve layered TLS on top of TCP, so a TCP connection gets established first and then an HTTP/gRPC/etc. request can be made. This layering made it much easier to slowly adopt TLS but it has also made it slower to establish connections that need TLS because it added more round trips between the server and client before a connection can be “established”. Each round trip introduces latency, delaying the time it takes for a request to reach the server and receive a response. Here’s what it looks like with HTTP/1.1 and HTTP/2:

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: TCP SYN
        Server ->> Client: TCP SYN-ACK
        Client ->> Server: TCP ACK
    end

    rect rgb(102,81,145)
        Client ->> Server: TLS Client Hello
        Server ->> Client: TLS Server Hello
        Client ->> Server: TLS Server Finished
        Server ->> Client: TLS Client Finished
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end

Yes, this process involves three round trips before the client even sends a request. Viewed in this way, this seems shockingly slow to me. By combining TLS 1.3, QUIC and HTTP/3, we can do away with several of those round trips. So with HTTP/3 it typically looks like this with only a single round trip before we can issue our request:

sequenceDiagram
	    actor Client

    rect rgb(47,75,124)
        Client ->> Server: QUIC
        Server ->> Client: QUIC
        Client ->> Server: QUIC
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end

But wait, we can actually go further. With a feature called 0-RTT, the number of round trips for a new connection can actually be ZERO. The idea is that the client can presume that a connection will be established and send the request based on that presumption. 0-RTT quite literally removes the need to do a round trip before making the first request. It looks like this:

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: QUIC
    end
    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
    end

    rect rgb(47,75,124)
        Server ->> Client: QUIC
    end
    rect rgb(200,80,96)
        Server ->> Client: HTTP Response
    end

Note that 0-RTT requires the client to have some cached information about the server so it’s not always possible to use. Also, there are still some security concerns, mostly focused on the potential for denial of service attacks.

Head-of-line Blocking

HTTP/2 has allowed gRPC to be quite good at multiplexing multiple requests and streams onto a single connection. This wasn’t possible with HTTP/1.1. However, there is an issue that can arise due to TCP’s guarantee of delivering packets in order even if they arrive out of order. This is an issue because packets for a request could be waiting for retransmissions from another request that is using the same connection. From gRPC’s point of view, these requests are independent things, so there’s no need to wait but TCP does not know about these separate streams, so we end up experiencing a so-called head-of-line blocking issue.

HTTP/3 solves the head-of-line blocking issue by avoiding TCP altogether. Instead, it is built on top of a protocol called QUIC which is built on top of UDP. QUIC is aware of multiple streams so it knows when it is appropriate to deliver packets without having this head-of-line blocking behavior. This makes HTTP/3 much better when dealing with unreliable networks. With its heavy use of streams, gRPC in particular would greatly benefit from the elimination of the head-of-line blocking issue.

Encryption is Required

When creating HTTP/2 there was a lot of disagreement over if it should require TLS. Many have argued that this requirement would hurt the adoption of HTTP/2. Further, they argue that HTTP/2 offers a lot of benefits that can be attained without the use of TLS. As a result, we now have h2c, which is a way to use HTTP/2 without encryption.

These arguments might have worked for HTTP/2 but they didn’t hold up for HTTP/3. HTTP/3 requires encryption and usage of the TLS 1.3 protocol. There are several reasons for this, but the most compelling one is that TLS 1.3 working with QUIC allows for fewer (and sometimes zero) round trips to negotiate a new connection before making a request, which is even faster than HTTP/1.1 without TLS. It’s a double win by being both more secure and, theoretically, much faster.

Experimentation

When researching this topic, I discovered that HTTP/3 with gRPC is a bit of a complicated story. I covered this a bit in my gRPC: The Good Parts post but the gist is that no decision has been made to officially support HTTP/3 throughout the entire gRPC ecosystem. There is an open issue to discuss this and there is an official proposal also discussing the idea. Some projects are closer to HTTP/3 than others, so here are the implementations that I have found where you can make gRPC over HTTP/3 work:

  • C#: grpc-dotnet is the pioneer here, already containing an implementation of an HTTP/3 transport for gRPC.
  • Rust: Tonic with the Hyper transport appears to be able to support this, although I’m not sure if there are good examples of this in the wild yet.
  • Go: ConnectRPC for Go uses the standard library http.Handlers, so any http server implementation can be used, including the transport available in quic-go.
  • Cronet: Cronet, designed primarily for mobile clients (Android and iOS), provides a way to utilize Chrome’s network stack, including its QUIC and HTTP/3 support. This can be particularly useful for building gRPC clients with HTTP/3 capabilities in mobile environments.

If you know about other gRPC implementations that can work with HTTP/3 (client or server), let me know. These are only the ones I know about.

When learning something new I always like getting my hands dirty by using it. I feel like this is the best way to learn. Since Go is my current working language, I decided to explore HTTP/3 using ConnectRPC in Go. I’m including the full examples in this article because I’ve looked and haven’t really been able to find good examples for this. I hope it’s helpful for others. The full working examples are on a git repo that I made for this post. Examples will have things like imports omitted for brevity but the full source is linked under each example.

Example server in Go

Let’s explore how easy it is to set up a gRPC server with HTTP/3 support using Go. The key players here are:

  • quic-go: This library provides a robust implementation of the QUIC protocol in Go, enabling HTTP/3 communication.
  • ConnectRPC: connect-go, the Go implementation of ConnectRPC allows us to define gRPC services using familiar Go HTTP handlers (http.Handler), which greatly simplifies the integration process.

I am using the Eliza service as the service to implement with ConnectRPC. This is a simple service intended to help demonstrate ConnectRPC. The implementation is un-important, so I omitted it but my implementation just echos back whatever the user sends.

func main() {
	mux := http.NewServeMux()
	// Implementation is only in the full source
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}
	if err := h3srv.ListenAndServeTLS("cert.crt", "cert.key"); err != nil {
		log.Fatalf("error: %s", err)
	}
}

That’s it! With this minimal setup, we now have a gRPC server that supports HTTP/3! I just used an http3.Server instance instead of http.Server. http3.Server does have different options related to the differences between QUIC and TCP. Note that it is possible to run an http.Server alongside http3.Server using the same address with the same port. How is this possible? Because HTTP/3 uses UDP and the ports are completely separate from TCP ports. Here’s an example:

func main() {
	mux := http.NewServeMux()
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}

	srv := http.Server{
		Addr:    addr,
		Handler: h2c.NewHandler(mux, &http2.Server{}),
	}

	eg, _ := errgroup.WithContext(context.Background())
	eg.Go(func() error {
		return h3srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	eg.Go(func() error {
		return srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	if err := eg.Wait(); err != nil {
		log.Fatalf("error: %s", err)
	}
}

This code demonstrates running multiple HTTP servers concurrently, providing support for HTTP/1.1, HTTP/2, and HTTP/3 on the same port by leveraging TCP for HTTP/1.1 and HTTP/2 and UDP for HTTP/3.

Oh, and note that this requires a certificate and a key because HTTP/3 requires TLS. Here’s the command that I used to create a self-signed cert for testing:

openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 \
    -subj "/C=DK/L=Copenhagen/O=kmcd/CN=local.kmcd.dev" \
    -keyout cert.key  -out cert.crt

That’s it! With just a few lines of code, we now have a gRPC server that supports HTTP/3!

Testing with an example client in Go

Now that we have a server running we need some way to test it, so let’s use the standard library http.Client with some quic-go magic sprinkled on top:

const (
	url        = "https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say"
	reqBody = `{"sentence": "Hello World!"}`
)

func main() {
	roundTripper := &http3.RoundTripper{
		TLSClientConfig: &tls.Config{
			// we need this because our certificate is self signed
			InsecureSkipVerify: true,
		},
	}
	defer roundTripper.Close()
	client := &http.Client{
		Transport: roundTripper,
	}

	log.Println("connect: ", url)
	log.Println("send: ", reqBody)
	req, err := http.NewRequest("POST", url, strings.NewReader(reqBody))
	if err != nil {
		log.Fatalf("error: %s", err)
	}
	req.Header.Add("Content-Type", "application/json")

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("error: %s", err)
	}

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("error: %s", err)
	}
	defer resp.Body.Close()

	log.Println("recv: ", string(body))
}

In the case of the client, we only need to define a http3.RoundTripper instance and pass that into a completely normal http.Client instance. That’s… quite literally it. Everything else should be the same.

2024/07/06 12:57:24 connect:  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
2024/07/06 12:57:24 send:  {"sentence": "Hello World!"}
2024/07/06 12:57:24 recv:  {"sentence":"Hello World!"}

This works as expected! In this next example, I’m calling the service using a client built with ConnectRPC, demonstrating how it can work seamlessly over HTTP/3 as well, just by configuring http.Client a little differently.

const url = "https://127.0.0.1:6660"
var reqBody = &elizav1.SayRequest{Sentence: "Hello World!"}

func main() {
	roundTripper := &http3.RoundTripper{
		TLSClientConfig: &tls.Config{
			InsecureSkipVerify: true,
		},
	}
	defer roundTripper.Close()
	client := &http.Client{
		Transport: roundTripper,
	}

	svcClient := elizav1connect.NewElizaServiceClient(client, url, connect.WithGRPC())

	log.Println("connect: ", url)
	log.Println("send: ", reqBody)
	resp, err := svcClient.Say(context.Background(), connect.NewRequest(reqBody))
	if err != nil {
		log.Fatalf("error: %s", err)
	}

	log.Println("recv: ", resp.Msg)
}

It works the same, but the output is a bit different since protobuf types print slightly differently from JSON:

2024/07/06 12:57:12 connect:  https://127.0.0.1:6660
2024/07/06 12:57:12 send:  sentence:"Hello World!"
2024/07/06 12:57:14 recv:  sentence:"Hello World!"

Now we have a server and a client that talks HTTP/3! Amazing!

Testing HTTP/3 servers with curl

I wanted to test this with a “real” HTTP/3 client because maybe there are some weird issues where quic-go servers only work against quic-go clients. To be extra sure, I wanted to validate this further with a well-established HTTP/3 client. So I reached for curl, the de-facto tool for calling HTTP APIs from the command line. So I immediately tried to just run curl --http3:

$ curl \
  --json '{"sentence": "Hello World!"}' \
  --http3 -k -v \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
curl: option --http3: the installed libcurl version doesn't support this
curl: try 'curl --help' or 'curl --manual' for more information

Wait, what? What’s happening?? I thought curl supported HTTP/3! What gives? Well, the story isn’t that simple. The curl CLI might support HTTP/3 but the libraries that it uses also need to be on the correct version to make it all work. To get around that, I installed curl using Cloudflare’s homebrew formula which will install everything needed to get curl to work with HTTP/3. So let’s try it out:

$ curl \
  --json '{"sentence": "Hello World!"}' \
  --http3 -k -v \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
*   Trying 127.0.0.1:6660...
* Server certificate:
*  subject: C=DK; L=Copenhagen; O=kmcd; CN=local.kmcd.dev
*  start date: Jul  6 06:14:48 2024 GMT
*  expire date: Jul  6 06:14:48 2025 GMT
*  issuer: C=DK; L=Copenhagen; O=kmcd; CN=local.kmcd.dev
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Connected to 127.0.0.1 (127.0.0.1) port 6660
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
* [HTTP/3] [0] [:method: POST]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: 127.0.0.1:6660]
* [HTTP/3] [0] [:path: /connectrpc.eliza.v1.ElizaService/Say]
* [HTTP/3] [0] [user-agent: curl/8.9.0-DEV]
* [HTTP/3] [0] [content-type: application/json]
* [HTTP/3] [0] [accept: application/json]
* [HTTP/3] [0] [content-length: 28]
> POST /connectrpc.eliza.v1.ElizaService/Say HTTP/3
> Host: 127.0.0.1:6660
> User-Agent: curl/8.9.0-DEV
> Content-Type: application/json
> Accept: application/json
> Content-Length: 28
>
* upload completely sent off: 28 bytes
< HTTP/3 200
< content-type: application/json
< accept-encoding: gzip
< date: Sat, 06 Jul 2024 13:17:12 GMT
< content-length: 27
<
* Connection #0 to host 127.0.0.1 left intact
{"sentence":"Hello World!"}

Success! You can see that HTTP/3 is being used by inspecting the verbose logging. This flexes our server with HTTP/3 but you may be wondering what this has to do with gRPC because I’m only using basic HTTP requests with JSON and you’re totally right. The previous examples only leverage one of the three protocols that ConnectRPC provides by default, the connect protocol. Thus far, I haven’t validated if the other two protocols, gRPC or gRPC-Web, really work using this transport. To test that, we’ll need more gRPC-centric tooling.

Adding HTTP/3 to the Buf CLI

I also wanted to test with gRPC-specific tooling to ensure gRPC and gRPC-Web worked as expected. So I added support for HTTP/3 with the buf CLI because it supports calling services using Connect, gRPC and gRPC-Web. This ended up being pretty easy and looks similar to the ConnectRPC example client above. I have an open PR here and if you want a sneak peak you can build the Buf CLI from my branch. Below, I show how I tested this new feature with ConnectRPC’s demo website:

$ buf curl \
    --http3 \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello...I'm glad you could drop by today."
}

Success! HTTP/3 works now! When I tested this, I was surprised that demo.connectrpc.com actually supported HTTP/3, but it does! By default, buf curl will use the connect protocol, so we didn’t test anything new. So let’s try gRPC-Web next:

$buf curl \
    --http3 \
    --protocol=grpcweb \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, how are you feeling today?"
}

And now the classic gRPC:

$ buf curl \
    --http3 \
    --protocol=grpc \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
   "code": "internal",
   "message": "protocol error: no Grpc-Status trailer: unexpected EOF"
}

Wait, what? Why is it complaining about missing HTTP trailers? What gives?

HTTP trailers are a recurring issue for gRPC. HTTP trailers are a special type of HTTP header that is sent at the very end of a message body after all the data has been transmitted. They are useful for sending metadata that cannot be determined until the entire message is known, such as checksums, signatures, or other end-of-message signals. gRPC relies on trailers to return status codes and error details.

To explain why I am not receiving the HTTP trailers, I had to dig into quic-go’s HTTP/3 implementation.

Adding trailer support to quic-go

When I looked into quic-go, I discovered that it doesn’t support HTTP trailers yet. There is an issue and a related pull request. But the issue is four years old, the PR is two years old, and both are still open after all of this time. This seemed quite crazy to me at first, but, on reflection, I realized that gRPC is the most popular thing that currently uses HTTP trailers. However, grpc-go directly uses HTTP/2 support from golang.org/x/net/http2 instead of using net/http so it would require a good amount of work to get HTTP/3 support through quic-go. Therefore, this issue probably isn’t on the radar of anyone working on grpc-go. However, it is trivial (as seen above) to use quic-go with ConnectRPC, so I think this is a unique situation where progress can be made quickly.

So I ended up implementing trailer support for clients and I submitted my own PR. I don’t think it is quite ready yet but when I use my branch of quic-go with my branch of the buf CLI, gRPC actually works with HTTP/3!

$ buf curl \
    --http3 \
    --protocol=grpc \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello there...how are you today?"
}

There are a few issues that I have with my PR. I based it off of the earlier PR but many things have changed with the codebase that actually make it harder to implement this feature. However, it’s super encouraging that this code seems to work without too much fuss.

Experiment Results

My experimentation shows that while Go doesn’t yet have full, native support for gRPC over HTTP/3, there are practical workarounds available today. Both the gRPC-Web and Connect protocols function seamlessly over HTTP/3, and in fact, ConnectRPC may already be leveraging HTTP/3 in production environments where infrastructure allows. I discovered this firsthand with the ConnectRPC demo website, which will connect using HTTP/3 from the browser to the load balancer (after a few requests so the browser knows HTTP/3 is available).

Conclusion

In this post, we’ve explored the exciting potential of HTTP/3 to supercharge gRPC performance. We dove into the key advantages of HTTP/3, such as faster connection establishment, elimination of head-of-line blocking, and mandatory encryption. By getting our hands dirty with practical examples in Go, we’ve seen firsthand how HTTP/3 can be seamlessly integrated into gRPC services using tools like ConnectRPC and Buf.

While the gRPC ecosystem’s full adoption of HTTP/3 is in its early stages, the benefits are clear, and the tools are already available in some libraries and tools. As developers, we have the opportunity to push this technology forward and shape the future of high-performance, secure communication.

I encourage you to experiment with HTTP/3 and gRPC in your own projects. Explore different implementations, measure the performance gains, and don’t be afraid to dive into the code if you run into issues. Your active engagement with this evolving technology can directly contribute to the ongoing development of gRPC and HTTP/3. Although widespread adoption of HTTP/3 for gRPC on the backend is still in its early stages, if you have the flexibility to control both server and client components, or are working with browser-based clients, you might find compelling use cases for it even today.

I’d love to hear about your experiences with HTTP/3 and gRPC. Have you seen significant performance improvements? Or perhaps you’ve found that QUIC is slower without the kernel-level optimizations that TCP can take advantage of? What challenges have you encountered while experimenting with this exciting technology? Let’s share our experiences and findings because even though HTTP/3 is still finding its footing in this context, there’s a lot we can learn from each other.

]]>gRPC: The Good Partshttps://kmcd.dev/posts/grpc-the-good-parts/Tue, 02 Jul 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-the-good-parts/ <p> <img hspace="5" src="https://kmcd.dev/posts/grpc-the-good-parts/cover_hu_208016cb095a54b9.webp" /> </p> Not perfect, but still pretty awesome. While REST APIs remain a popular choice for building web services, gRPC is increasingly being adopted for its unique advantages in performance, efficiency, and developer experience. You may have seen my post, gRPC: The Bad Parts, where I talk about some of my issues with gRPC. Based on the many comments about that article, I could easily write a sequel with even more complaints. However, today I’m going to focus on the good parts of gRPC. It has become obvious to me that many people didn’t read the ending of the last post which attempted to outline how many of the points that I made are no longer true. So I figured that I need to give the positive aspects of gRPC a dedicated post.

Let’s dive into the key advantages that make gRPC a powerful tool for modern web development.

Performance

This one might be a little controversial, but Protocol Buffers is, indeed, faster than JSON and XML. This continues to be demonstrated over and over again. Protobuf is able to be faster for these reasons:

  • Field names are not included in the message. Instead, protobuf uses numbers to distinguish fields. In most cases you’ll see field numbers take one or two bytes on the wire when it can be much more than that depending on your JSON field names.
  • Protobuf’s VARINT type allows for small scale integers to take up a single byte, even if it’s an int64. Realistically we really don’t use that many large numbers so these savings can add up. Again, it’s much better than ASCII-encoded numbers being used for each digit.
  • There’s no real winning with strings or byte arrays but compression is still supported with gRPC so at worst this aspect is even with HTTP/JSON.

I’ve personally seen 50% data transfer savings by switching to protobuf encoding with realistic payloads.

There are some haters of protobuf encoding, and that’s perfectly fine. The only “fatal flaw” that I’ve actually been annoyed with is “map values cannot be other maps.” It does seem like that should be possible, even when you consider the internal representation of a map:

map<key_type, value_type> map_field = N;

transforms into:

message MapFieldEntry {
  key_type key = 1;
  value_type value = 2;
}

repeated MapFieldEntry map_field = N;

It’s super frustrating because I don’t understand why value_type can’t be a map. The solution to this problem is just to make your own wrapper type to use as the value that contains a map. It is kind of annoying and this does come up semi-often. Crap, this was supposed to be a positive article. Let’s get back on track.

I think the protobuf encoding is better than JSON in many ways. However, I understand that sometimes you just want JSON, and with gRPC you absolutely can just use JSON. gRPC still has a few binary framing bytes before each message that won’t be human readable but if you’re really concerned with those check out the ConnectRPC section below.

Most gRPC implementations also let you define your own encoding, so it may possible to insert your own favorite encoding if you want to push the limits.

Strongly Typed Contracts

Say goodbye to the guesswork of loosely typed APIs. gRPC’s protobuf definitions create rock-solid contracts between client and server. This translates to:

  • Fewer errors: Clear expectations for data types reduce the chance of mismatched data.
  • Better code generation: Automatic generation of client and server code in various languages saves time and effort.
  • Smoother development cycles: Consistent contracts make it easier to evolve your API without breaking existing clients.
  • Generated Documentation: Automatic generation of documentation means that your documentation will never be out of sync with your API.

API contracts are very powerful. For more on this topic, I’ve written an article discussing API contracts called Building APIs with Contracts.

Streaming Support

Streaming support is arguably the best and most unique feature for gRPC. It does away with needing to frequently poll for updates in many scenarios which make it a good candidate for:

  • Chat applications: Seamlessly handle messages flowing back and forth.
  • Live updates: Push updates to clients as soon as they happen.
  • Any scenario where constant communication is key: From gaming to financial data, gRPC’s streaming capabilities open up a world of possibilities.

If you come from the networking world, you might know that gNMI (which is based on gRPC) is the replacement for SNMP. Instead of polling network devices for the same data every minute, you can now use gNMI to subscribe to counters. I’ve written more about this in a post called Why you should use gNMI over SNMP in 2024.

Cross-Language Support

gRPC doesn’t care what programming language you prefer. Thanks to code generation tools, you can seamlessly work with gRPC in a wide range of languages, including:

  • Go
  • Rust
  • Java
  • Python
  • C#
  • Node.js
  • Ruby
  • …and many more!

This promotes flexibility, collaboration, and the ability to choose the right tool for the job. Currently, I believe gRPC has such a large momentum with language support that it’s hard for me to consider an alternative that doesn’t also speak gRPC. Many alternatives that have similar benefits to gRPC have mediocre support for a handful of languages at best.

Pioneered HTTP/2

gRPC was a driving force behind the adoption of HTTP/2, a major upgrade to the web’s underlying protocol. This means you get all the benefits of HTTP/2’s:

  • Multiplexing: Multiple requests and responses can share a single connection, improving efficiency.
  • Header compression: Smaller headers mean faster transmission.
  • Overall performance improvements: HTTP/2 is simply a faster, more efficient way to communicate over the web.

HTTP/3

There’s some movement on HTTP/3 support for gRPC. There is an open proposal created by the dotnet gRPC library maintainers and there is an open issue to discuss actually adding HTTP/3 to the gRPC spec. Frustratingly, there hasn’t been a lot of movement on the official gRPC repo to add support directly to any of their implementations, but as you can see from the thread, there’s a lot of interest and a lot of people making prototypes that prove the concept.

This is likely an incomplete list but here are the packages that you can likely use HTTP/3 with today:

  • The standard grpc library for C#, dotnet-grpc (ref)
  • It may already be possible in rust with Tonic with the Hyper HTTP transport (ref)
  • It’s possible in Go if you use ConnectRPC with quic-go - I don’t have a link for this, but I’ve tested this out myself. This is a topic for a future post!
  • This is untested but I believe many gRPC-Web implementations in the browser might “just work” with HTTP/3 as well as long as the browsers are informed of the support via the ALT-SVC header and the servers support it.

As more servers and clients support HTTP/3 they should see faster connection establishment times, complete removal of the head-of-line blocking problem and much better recovery from packet loss. There’s a long way to go here, but there is progress.

Bridging the Gap

If you’re looking to gradually adopt gRPC or need to support existing REST clients, there are several options available today!

JSON/HTTP Transcoding

Tools like gRPC-Gateway, Google Cloud Endpoints and Envoy can expose REST-like interfaces while still reaping the benefits of gRPC on the backend. You can define a service that looks like this:

syntax = "proto3";
package your.service.v1;
option go_package = "github.com/yourorg/yourprotos/gen/go/your/service/v1";

import "google/api/annotations.proto";

message StringMessage {
  string value = 1;
}

service YourService {
  rpc Echo(StringMessage) returns (StringMessage) {
    option (google.api.http) = {
      post: "/v1/example/echo"
      body: "*"
    };
  }
}

And get a REST-like endpoint where you can make this request:

curl -XPOST '{"value": "my value!"}' http://localhost:8000/v1/example/echo

This is pretty amazing because it’s doing a lot of the hard work for you and you can now support many different REST APIs without writing any additional code. This is a simple example here but there are many options, like being able to populate message fields from components of the path.

gRPC-Web

One of the big limitations of gRPC is that it doesn’t work on the web with web browsers due to limited support of HTTP trailers. Browsers support receiving trailers but there isn’t yet a way to retrieve those trailers from javascript. Yes, this is incredibly frustrating, especially since there are many small use cases where trailer support would be amazing to have.

The gRPC-Web protocol gives browsers the ability to use gRPC, which drastically improves the story of contract-based services in gRPC. It also allows for HTTP/1.1 clients to work with gRPC. Some platforms (I’m looking at you, Unity) still don’t support HTTP/2, even though it’s 2024 and the HTTP/2 spec was created nearly a decade ago.

ConnectRPC

ConnectRPC automatically generates JSON/HTTP APIs from your gRPC definitions while also maintaining compatibility with gRPC and gRPC-Web. This HTTP protocol, called Connect, follows HTTP standards more closely. For example, the Content-Coding header, Content-Length header, HTTP status codes, etc. all work as expected for unary RPC calls. That means you can run this normal-looking curl command and talk to a gRPC service:

curl --header "Content-Type: application/json" \
    --data '{"sentence": "I feel happy."}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say

Twirp

Twirp is very similar to ConnectRPC. It was developed by Twitch, and is another framework that can help bridge the gap between gRPC and REST. Twirp’s approach is to use protobufs to generate an alternative protocol that also aligns more with HTTP conventions. It doesn’t also support gRPC and gRPC-Web. Implementing those alongside twirp is left as an exercise for the user if you want to interoperate with other gRPC tooling.

Tooling

I have mentioned that gRPC tooling isn’t that great. I still agree with that if we’re talking about the “out of the box” tooling from the gRPC project. However, the community is much bigger than the gRPC Authors and someone finally made the protobuf code generation a lot better.

Buf CLI

Buf (the company) has made a client called Buf CLI, which I’m going to refer to as just “buf” from here on out.

protoc is the official compiler for protobufs, which has plugins for many languages, frameworks, documentation and other kinds of outputs. Buf completely replaces protoc by using the same protoc plugins that protoc uses. How is it better? It adds a set of config files for defining the structure of your protobuf files, including external protobuf dependencies and external plugins using the Buf Schema Registry. Instead of random makefile directives or bash scripts, we now have a well-defined config file for defining how the protobuf is built, which is amazing.

Similarly, buf curl provides a convenient way to interact with gRPC services, much like the popular tool grpcurl.

In addition to replacing existing tooling with easier-to-use versions, buf also implements some extremely useful functions. I first started using buf by using buf lint, which helps enforce some common rules and practices that developers should follow when making their protobuf files. Soon after, I started using buf breaking which will report on breaking changes being made to protobuf files that may break clients. Both have easy-to-use Github actions and were pretty painless to set up.

Adding buf into the mix can greatly improve your workflow with protobufs, especially when working in a larger team or working with other teams.

Third-party protoc plugins, libraries and tools

There’s so many plugins now. I even made one and to be honest, plugins aren’t that hard to make. I think this is the way that API development should work where you base API services off of a contract and generate everything from that same contract. No typos. No confusion over what methods exist. No arguing over REST semantics that has never been clear to anyone.

  • protoc-gen-doc: Builds gRPC documentation in several formats. The default styling honestly doesn’t look the prettiest but it does allow you to specify a custom template which has been amazing for me to generate something custom without requiring an entire plugin.
  • protoc-gen-connect-openapi: This is my plugin. It generates OpenAPIv3 specs for your ConnectRPC services and has support for protovalidate, gnostic OpenAPIv3 annotations, and gRPC-Gateway annotations.
  • protovalidate: Protovalidate allows you to embed validation rules in your protobuf files which can be used by an associated library to enforce those rules. A large complaint that people have with gRPC is that it’s hard to use whenever every single field is optional. Now you can replace a lot of validation code, including required fields, with protobuf options. I’m anxiously awaiting typescript support so validation logic can be shared on web frontends and backends, which, to me, is the “holy grail” feature of a contract-driven service.

In addition to these libraries and plugins, more tools that you know and love from HTTP are supporting gRPC like Postman, Insomnia and k6.

The availability of numerous third-party plugins underscores the fact that gRPC is more than just a framework – it’s a dynamic ecosystem that fosters innovation and empowers developers to customize their workflows to meet their specific requirements.

Conclusion

gRPC offers a compelling set of advantages for modern web development.

Its performance, strong typing, streaming capabilities, cross-language support, and HTTP/2 foundation make it a powerful tool for building efficient and scalable APIs. With various adoption options available, you can gradually incorporate gRPC into your projects and experience its benefits firsthand.

The growing community and active development around gRPC suggest a bright future for this technology. If you’re looking to build fast, reliable, and future-proof APIs, gRPC is a tool that deserves a serious look. Dive in, explore the ecosystem, and discover how gRPC can revolutionize the way you approach API development.

]]>Leaving Texas for Greener Pastureshttps://kmcd.dev/posts/leaving-texas/Tue, 25 Jun 2024 00:00:00 +0000https://kmcd.dev/posts/leaving-texas/ <p> <img hspace="5" src="https://kmcd.dev/posts/leaving-texas/cover_hu_c36372a20c1b28a0.webp" /> </p> Why I left my life behind. Today, I’m going to talk a bit about a topic that I haven’t talked about much on this blog. I have crossed the two-and-a-half-year mark of living in Denmark and I figured it was time to reflect on how it is living in the little Nordic country after living my entire life living around Dallas, Texas. First, though, I want to outline why I decided to leave Texas. In a follow-up post, I will discuss “Why Denmark?” and reflect on how the last few years have been.

Why Leave Texas?

Most people aren’t surprised at why anyone would leave the United States, but I think it’s useful to list some of the big reasons. Because I have a terrible memory I will be referencing notes that I made in May 2021. Be warned, many of these notes are political but I just can’t avoid political topics because it does have a large influence on my decision to leave.

Education

I don’t have a high opinion of public education in Texas. Guess which kinds of schools are WAY less likely to have a school shooting? Private Schools. While I very much had the means to send my child to private school since I had a very high-paying job, the prospect of raising my child in a drastically different environment than the majority of her peers felt extremely gross and classist. And having to pay to feel like your child is relatively safe just seems crazy. It seems broken. This isn’t even considering the extremely low resources that many public schools have to do their job of educating children. This doesn’t match my values.

Health Care

Even for wealthier families, a serious health condition can lead to financial ruin. The United States ties health insurance with your job. So you have to be working to get good health insurance. Many medical conditions can prevent you from working and it’s all too common for companies to shove you to the door when your health is declining. Some people might think that the system in the US is fine because Americans don’t pay as much taxes because of it. However, the United States spends significantly more than other nations on health care, many of which have public health care. This doesn’t match my values.

Abortion and Contraceptives

Laws against abortion were starting to take shape when we were making our decision. I feel like women and girls deserve health care and autonomy over one of the most impactful processes that happens to their bodies. I was set on leaving Texas for this reason because I have a daughter and I feel strongly that raising a daughter with these kinds of laws is wrong. More and more states have added their own laws against abortion after the overturning of Roe v. Wade and now it looks like it has become a national issue… This doesn’t match my values.

Religion

I am not religious and I feel like I don’t share the same values and beliefs as most people in Texas or the United States. While there is a growing “non-religious” demographic in the United States I still feel surrounded by those who don’t share my values.

Source: Gallup

Source: Gallup

Work-life balance

Work-life balance is generally awful in the States. Even for high-paying jobs, it is nearly impossible to get the same amount of paid time off as you get by law in many European countries. It is very common to be expected to work way more than is reasonable and work even when you’re “a little sick”. Pregnant women are often expected to work until the day they give birth and return to work soon after birth. And it’s absolutely crazy that we solve the wrong problems. For example, many “lactation room” where working women who have recently been pregnant have a place to pump milk, out of view from others. This outlines so many societal issues for me.

While I do have a high work ethic, life is more than work. The work culture of the United States in general doesn’t match my values.

Poverty

The United States has a lot of homeless people. There’s an adaptation that many people develop when living in a large city in the US where you just stop seeing homeless people. They’re still there, but your subconscious mind will actively prevent you from registering that the person is there. This appears to be the only way for people to get through their day. It’s sad. This doesn’t match my values.

Here’s some related data provided by OECD. According to this data, the United States has a poverty rate of 18%. With this data, Denmark is very close to the best with 4.8%.

Source: OECD

Source: OECD

Car Culture

How most cities are designed in the US is insane. Cars are a required item that everyone has to own. This has a lot of impacts on day-to-day life but there are two I want to focus on here. The age that a kid feels truly autonomous in the United States is age 16 when you can get a driver’s license and start driving yourself places. Let’s ignore the classism where I assume that every 16-year-old has parents who can afford to buy each of their kids a car. It is also so easy to not think of other cars around you in traffic as people. Everyone is separated by glass and metal. I feel like this subtracts from social cohesion and may be a much more causal part of many other societal ills that the US has.

I am, in fact, a fan of this newer wave of urban planners like notjustbikes and Strong Towns.

Anyway, just listing all the ways that car culture harms people can be the topic of several posts. But for now, let’s just say that the over-reliance on cars doesn’t match my values.

Gun control

Gun ownership is deeply ingrained in American culture. I think that’s actually fine. Guns can be fun and in some cases, useful. But having NO control over who gets to access deadly firearms is lunacy. And I don’t think Americans can fully understand how off-base Americans are on this until they’ve lived in a country with strong gun control. Danish people will randomly set off fireworks. Sometimes in the middle of the day when there’s nothing to see. They do it as part of protests and leading up to New Year’s you will constantly hear random fireworks being set off all over the city. And then you realize that NO ONE hears these noises and immediately think “Active shooter, get to cover!” I no longer react to random fireworks this way because I have healed from this trauma that I never knew I had. The gun laws in the United States don’t match my values.

Weather

I hate the Texas heat. It’s only bearable due to massive amounts of air conditioning. Due to unstable arctic weather, Texas can also get quite cold, causing some notable problems in the state.

That said, I do have one more issue with extreme Texan weather…

My house was hit by a tornado in 2019. Since my wife was traveling out-of-state visiting relatives at the time I was home alone with our six-month-old daughter. It was a terrifying experience. The house was damaged but was repairable so our family lived out of an apartment for half a year while the house was repaired. It was a hard time for me and I have since experienced extreme anxiety when severe storms happen, which is pretty often in Texas. I was close to seeking a therapist to try to help with anxiety, but, luckily, the reaction has faded in time. I know it’s impossible to avoid all kinds of extreme weather but I am done living in tornado alley.

Climate Change

I’m not sure how much I need to talk about here. Many Americans just refuse to accept that their lives may have to change to reduce their CO2 emissions. I think it takes a well-informed population to hold companies accountable for their externalities and to make a series of small sacrifices that will end up doing less harm to our environment. I think Americans are actively misinformed in this regard. Some effects of climate change are happening now. It’s making some areas of the United States impractical to live in already and this is only the beginning. I want to live in a society that cares about the world they live in. Again, it’s absolutely insane that I have to say this.

Summing Up

Leaving Texas was a deeply personal decision, driven by a desire for a society that better aligned with my values and aspirations for my family’s future. While the Lone Star State holds cherished memories, family and countless opportunities, the challenges I’ve outlined ultimately outweighed the benefits.

But amidst the disappointment and frustration, a glimmer of hope emerged – the prospect of a fresh start in a land known for its progressive policies, strong social safety net, and emphasis on well-being.

Next week you can join me in the next installment of this series, where I’ll delve into the factors that drew me to Copenhagen, Denmark, and share the joys, struggles, and unexpected lessons learned during my first two years abroad. Will Denmark prove to be the “greener pastures” I was looking for? Stay tuned to find out.

]]>gRPC: The Bad Partshttps://kmcd.dev/posts/grpc-the-bad-parts/Tue, 18 Jun 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-the-bad-parts/

gRPC isn't perfect but who is? ]]>gRPC, the high-performance RPC framework, has been super successful (if you work for Google) and has drastically changed the way we all deploy APIs (if you work for Google). gRPC and protobuf is an extremely performant contract-focused framework with extremely wide language support. But it’s not without its downsides. Making an RPC framework that requires code generation and support in many programming languages is sure to get some things wrong. As gRPC approaches a decade of usage, it is important to reflect on what could have been better.

Learning Curve

Let’s start out extremely nit-picky. So-called unary RPCs are calls where the client sends a single request to the server and gets a single response back. Why does gRPC have to use such a non-standard term for this that only mathematicians have an intuitive understanding of? I have to explain the term every time I use it. And I’m a little tired of it.

Speaking of unary RPCs, the implementation is more complicated than it needs to be. While gRPC’s streaming capabilities are powerful, they have introduced complexity for simple RPC calls that don’t require streaming. This hurts the ability to inspect gRPC calls because now there is framing on every unary RPC which only makes sense for streaming. Protobuf encoding is complicated enough so let’s not add extra gRPC framing where it isn’t needed. Also, it doesn’t pass my “send a friend a cURL example” test for any web API. It’s just super annoying to explain to someone how to use gRPC. I’ve said “okay, but is server reflection enabled?” so many times. I’m just tired of it.

This complexity also bleeds into the tooling with the mandatory code generation step. This can be a hurdle, especially for dynamic languages where runtime flexibility is valued. Additionally, some developers might be hesitant to adopt a technology that necessitates an extra build step. We already need 20 build steps for modern web development, so it’s sometimes hard to justify one more.

Compatibility with the Web

The reliance on HTTP/2 initially limited gRPC’s reach, as not all platforms and browsers fully supported it. This has improved over time, but it still poses a challenge in some environments. But even with HTTP/2 support, browsers have avoided adding a way to process HTTP trailers so browsers today still cannot use “original” gRPC. gRPC-Web has acted as a plaster for this issue by avoiding the use of trailers, but it often requires “extra stuff” like running a proxy that supports gRPC-Web. Which is annoying.

Late Adoption of HTTP/3: The delay in embracing HTTP/3 might have hindered gRPC’s ability to take full advantage of the protocol’s performance and efficiency benefits. I have personally been affected by the head-of-line blocking issue that can happen when using gRPC with HTTP/2 and it would be so nice to be able to completely do away with this issue by being able to use HTTP/3 with gRPC. It’s strange to see a framework that pushed many languages to support HTTP/2 struggling to do the same thing with HTTP/3.

JSON Mapping and Prototext

Another area where the “timing” was wrong was the lack of a standardized JSON mapping early on. It has made gRPC less accessible for developers accustomed to JSON-based APIs and I don’t think it ever recovered from that stigma. Having a mapping between protobuf types and JSON simplifies integration and interoperability with existing tools and systems. You would not believe how happy web developers can get when you say “yeah, this is a super-efficient binary format… but you can set this flag and get JSON back if you want to debug.” They get unreasonably excited. Unreasonably. excited. Anyway, now that protobuf has standard rules for mapping protobuf types to JSON (and the other way) I feel like the protobuf text format is an unnecessary complexity. I don’t see a use-case for the text format now that we have JSON. So let’s throw the text format away. We don’t need it and I’m down to pretend like it never existed if everyone else is. Cool?

Finite Message Sizes

Most Protobuf encoders/decoders expect to fully parse an entire message and give the full response to the consumer but memory is finite and sometimes you might want larger messages. Sometimes you want to stream parts of these larger messages somewhere else and not keep the entire message in memory. Therefore, if you want to, for example, upload large files you’re going to need to implement some kind of chunking. While chunking is a reasonable solution for handling large files, the absence of a standardized approach within gRPC might lead to inconsistent implementations and increased development effort.

As a demonstration, here’s what it may look like to upload a file with gRPC:

syntax = "proto3";

package file_service;

service FileService {
   rpc Upload(stream UploadRequest) returns(UploadResponse);
}

message UploadRequest {
    string file_name = 1;
    bytes chunk = 2;
}

message UploadResponse {
  string etag = 1;
}

This is both a strength and weakness of protobufs. This concept is super easy to define in protobuf but in practice, the code to properly implement this can be cumbersome and error-prone. And while Google, the creator of gRPC, has figured out solutions for their APIs, the lack of a standardized approach leaves others to reinvent the wheel.

You might be thinking “Google uses gRPC in most of their APIs so obviously they’ve done this” and you’d be right. They actually have a gRPC and HTTP version for downloading (potentially large) files. We can compare the gRPC and HTTP versions directly and gRPC is BY FAR more complex. Go ahead and compare the linked code. I’ll wait.

ded internet theory

I see a lot of gRPC/protobuf communities that are devoid of activity. The lack of visible activity on some websites might create the impression that gRPC is stagnant or less actively maintained. This could deter potential adopters and contribute to slower community growth. This might be a case of too many options, making it difficult to find someone to nerd out about gRPC outside of GitHub issues where such enthusiasm might be perceived as annoying.

Bad tooling

For the longest time, when I saw that a codebase uses protobuf I found a weird script that downloads random protobuf files in super custom ways and places them in random paths and then makes a series of super complex calls to protoc. Only google would think not solving dependency management is the solution to dependency management. Google has its own extremely Google-y way of managing dependencies that we peasants can only dream of using.

It can be (and is) better

While I’ve been critical of gRPC, I hope my comments come across as constructive. Those who have read this far down in the article get to know that many of these issues are already fixed or at least on the way to being fixed!

  • Several gRPC implementations already support HTTP/3. ConnectRPC makes it pretty easy to use HTTP/3 with gRPC (I’ll follow up on this in a future post).
  • Since the protobuf spec has a canonical mapping to/from JSON I no longer have to worry about the text format. I really do hope that everyone forgets that it exists. There’s only room for so many text-based formats. I wasn’t joking about that. This is the last time I’m acknowledging its existence.
  • The gRPC community is actually alive and well if you know where to look. For example, the buf slack has been a great resource for me. You may find me hanging out and answering questions fairly often.
  • The Buf CLI is an amazing tool for gRPC. It completely replaces protoc but also adds linting, breaking change detection, curl for gRPC, integration with the Buf Schema Registry (wow, real dependency management!) and more! In addition, more tools that you know and love from HTTP support gRPC like Postman, Insomnia and k6.

Despite gRPC’s undeniable successes, it’s important to acknowledge the framework’s shortcomings to ensure its continued evolution and improvement. By addressing its learning curve, compatibility issues, lack of standardization, and community engagement, we can unlock gRPC’s full potential and make it a more accessible and user-friendly tool for all developers.

]]>Unit Testing ConnectRPC Servershttps://kmcd.dev/posts/connectrpc-unittests/Tue, 11 Jun 2024 11:00:00 +0000https://kmcd.dev/posts/connectrpc-unittests/ <p> <img hspace="5" src="https://kmcd.dev/posts/connectrpc-unittests/cover_hu_15a2717b59d9d13e.webp" /> </p> Learn how to test your ConnectRPC services. If you’ve embarked on the journey of building efficient and scalable RPC systems with ConnectRPC, you might be pondering the best way to ensure the reliability and correctness of your services. Unit testing is the obvious tool for this, providing a safety net that catches bugs early and empowers you to refactor code fearlessly. In the ConnectRPC world, unit testing can be daunting due to its integration with Protocol Buffers and the client-server architecture. In this guide, we’ll unravel the mysteries of unit testing ConnectRPC services, while arming you with practical examples and advanced techniques to fortify your codebase.

First off, the full source code can be found on github. If it helps, feel free to download, run, and modify as you see fit!

Why Unit Test?

Before we dive in, let’s address the “why.” Unit testing your ConnectRPC servers brings a multitude of benefits:

  • Isolation: Focus on testing individual components in isolation, making it easier to pinpoint and fix issues.
  • Speed: Unit tests execute quickly, providing fast feedback during development.
  • Refactoring Confidence: When you have solid unit tests, you can refactor your code with confidence, knowing that the tests will catch any unintended consequences.
  • Documentation: Well-written unit tests can serve as living documentation, illustrating how your code is meant to be used.
  • Bug Prevention: A good suite of unit tests can help you catch bugs early on, before they become harder and more expensive to fix.

Testing Strategies with ConnectRPC

ConnectRPC, built upon the Protocol Buffers ecosystem, offers a couple of primary approaches to unit testing:

  1. Direct Service Testing: This is ideal for unit testing but it’s not always possible. You directly call the methods of your service implementation (typically a struct in Go), bypassing any client and server networking.
  2. Server Testing: This approach creates an actual ConnectRPC server with net/http/httptest. It’s helpful when you want to test the interactions between your client and server code but is usually “overkill” unless you’re wanting to test interceptors or HTTP middleware.

Hands-On: Our example service

Here is the protobuf file that we’re using for our example:

syntax = "proto3";

package greet.v1;

option go_package = "example/gen/greet/v1;greetv1";

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

And here is the resulting server implementation:

type greeterService struct{}

var _ greetv1connect.GreetServiceHandler = (*greeterService)(nil)

func (g *greeterService) Greet(ctx context.Context, req *connect.Request[greetv1.GreetRequest]) (*connect.Response[greetv1.GreetResponse], error) {
	if req.Msg.Name == "" {
		return nil, errors.New("missing name")
	}

	// Simulate some network call that takes 10ms
	select {
	case <-ctx.Done():
		return nil, ctx.Err()
	case <-time.After(10 * time.Millisecond):
	}
	return connect.NewResponse(&greetv1.GreetResponse{Greeting: "Hello, " + req.Msg.Name}), nil
}

Here we defined our greetv1connect.GreetServiceHandler implementation. It implements the Greet method defined in the protobuf file alove. Since this is for demonstration purposes, all we do is check to see if the given name is empty, sleep for 10 milliseconds to simulate a network call and returns the greeting as "Hello, {name}".

A keep observer might notice that this file contains our first “test”. The line var _ greetv1connect.GreetServiceHandler = (*greeterService)(nil) is a way of doing a type assertion in Go. It ensures that your greeterService struct correctly implements the GreeterService interface defined by the protobuf file above. This relies on a trick of the Go syntax that will try to bind a variable _ using the greetv1connect.GreetServiceHandler type. If the given greeterService pointer doesn’t implement the interface then the compiler should complain about what specific methods are missing and which method signatures don’t match.

Hands-On: Direct Service Testing Example

Let’s write some unit tests for a simple ConnectRPC service:

func TestGreet(t *testing.T) {
	service := &greeterService{}
	response, err := service.Greet(context.Background(), connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}))
	if err != nil {
		t.Fatalf("Greet failed: %v", err)
	}
	if response.Msg.Greeting != "Hello, Alice" {
		t.Errorf("Unexpected greeting: got %q, want %q", response.Msg.Greeting, "Hello, Alice")
	}
}

Explanation: The TestGreet function creates an instance of your greeterService and directly calls its Greet method. We then assert that the response matches our expectations. This is, by far, the simplest method for testing a ConnectRPC service.

Hands-On: Table-Driven Tests with Testify

Now that we wrote a single unit test, the next example will show you how to utilize table tests in order to easily write more test cases. You will see code that looks like this in well-tested Go repositories.

func TestGreetTable(t *testing.T) {
	service := &greeterService{}
	cancelledCtx, cancel := context.WithCancel(context.Background())
	cancel()

	testCases := []struct {
		name    string
		ctx     context.Context
		req     *connect.Request[greetv1.GreetRequest]
		want    *connect.Response[greetv1.GreetResponse]
		wantErr string
	}{
		{
			name:    "Success",
			req:     connect.NewRequest(&greetv1.GreetRequest{Name: "Bob"}),
			want:    connect.NewResponse(&greetv1.GreetResponse{Greeting: "Hello, Bob"}),
			wantErr: "",
		},
		{
			name:    "Empty Name",
			req:     connect.NewRequest(&greetv1.GreetRequest{}),
			want:    nil, // Expecting an error
			wantErr: "missing name",
		},
		{
			name:    "Context Cancelled",
			ctx:     cancelledCtx,
			req:     connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}),
			want:    nil,
			wantErr: "context canceled",
		},
	}

	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			ctx := tc.ctx
			if ctx == nil {
				ctx = context.Background()
			}

			got, err := service.Greet(ctx, tc.req)
			if tc.wantErr != "" {
				require.Error(t, err)
				assert.Contains(t, err.Error(), tc.wantErr)
			} else {
				require.NoError(t, err)
				assert.Equal(t, tc.want, got)
			}
		})
	}
}

Explanation:

  1. Table Setup: A testCases slice defines scenarios with varying inputs (req), expected outputs (want), and potential errors (wantErr).
  2. Context Cancellation: The test case “Context Cancelled” simulates a cancelled context by creating a context with context.WithCancel and immediately calling cancel().
  3. Testify Assertions: The require package is used for assertions that should stop the test if they fail (e.g., requiring an error). The assert package is used for assertions that are not critical for continuing the test. Typically, errors during test setup use require and assertions on the results of the test use assert.

Hands-On: Server Testing Example

Here’s how you can test the same service using net/http/httptest server:

func TestGreetWithServer(t *testing.T) {
	mux := http.NewServeMux()
	mux.Handle(greetv1connect.NewGreetServiceHandler(&greeterService{}))

	server := httptest.NewServer(mux)
	t.Cleanup(func() { server.Close() })

	client := greetv1connect.NewGreetServiceClient(http.DefaultClient, server.URL)

	response, err := client.Greet(context.Background(), connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}))
	if err != nil {
		t.Fatalf("Greet failed: %v", err)
	}
	if response.Msg.Greeting != "Hello, Alice" {
		t.Errorf("Unexpected greeting: got %q, want %q", response.Msg.Greeting, "Hello, Alice")
	}
}
  • Server Setup: We create a ConnectRPC handler and start it with httptest.NewServer(mux).
  • Client Setup: We create a ConnectRPC client that connects to the server that we just created.
  • Test Interaction: We use the client to call the Greet method and assert the response, just like in the direct service testing example.

Conclusion: Test with Confidence

In this guide, we’ve explored the “why” and “how” of unit testing your ConnectRPC services. By embracing unit testing as a core part of your development workflow, you’ll create more robust, reliable, and maintainable RPC systems. Remember, effective testing isn’t just about fixing bugs – it’s about building confidence in your codebase and enabling you to iterate and evolve your services with ease.

The full source code can be found on github.

Next Steps:

  • Go Beyond the Basics: Explore more advanced testing techniques, such as mocking dependencies for more complex scenarios.
  • Integrate with Your CI/CD: Automate your unit tests to run as part of your continuous integration and continuous delivery (CI/CD) pipeline for immediate feedback on code changes.
  • Share Your Knowledge: Help the ConnectRPC community grow by sharing your own testing strategies and experiences!

Ready to put your newfound knowledge into action? Start writing those unit tests and watch your ConnectRPC projects thrive!

]]>Daily Promptshttps://kmcd.dev/posts/daily-prompts/Tue, 04 Jun 2024 10:00:00 +0000https://kmcd.dev/posts/daily-prompts/ <p> <img hspace="5" src="https://kmcd.dev/posts/daily-prompts/cover_hu_9224a1a8ebaf8ea.webp" /> </p> Kickstart your creative juices with a daily writing prompt. You may have noticed from my mastodon feed or from a link on the website’s navigation that I have added daily prompts. Each day is a new question to answer. You can respond to the post on Mastodon and it will appear (after some minutes) on this website. I’ve been doing it for the entire month of May and I intend to keep doing it to spur my thoughts and maybe others. I feel like having a question to answer each day helps give others context on who I am, what I’m doing and how I see the world. I have also found that it acts as a good way to reflect on things. I’ve wanted some answers to be different… so I now have some motivation to make them different.

So go and check out the prompts page.

The rest of this article talks about all of the pieces that went into making this happen behind the scenes, besides actually writing the prompts.

I wanted to show a banner image for the prompts… but using the same one every day seemed boring… So now I have a set of 6 banner images that I rotate through. Here’s how I did that with Hugo:

layouts/partials/prompts/load-cover.html

{{- $images := (resources.Match "images/prompt_covers/*") -}}
{{- return (index $images (mod .Date.Day (len $images))) -}}

This piece of code picks a new cover based on the day of the month, so the first of every month will have the same cover, for example. As I add more covers, the covers might shift over some number of days but I hope to eventually make 31 covers so each day of the month is unique. Notice that I used the return statement. This is a special Hugo-specific statement that allows you to return values from partials.

Here’s an example how how I use this partial in another template:

layouts/prompts/single.html

{{- $cover := (partial "prompts/load-cover.html" .).Fill "1520x400 Center webp q100" -}}
<figure>
    <img src="{{ $cover.RelPermalink }}" alt="{{ $cover.Title }}" />
</figure>

Mastodon “Comments”

I’ve changed a lot about how this website is deployed lately and hopefully, no one noticed… The biggest change was switching from GitHub Pages to Cloudflare Pages. At first, this was just a temporary change, because Cloudflare was going to make me wait an entire week before allowing me to use kmcd.dev with the pages website, so I adjusted the Github Action to publish to Cloudflare Pages instead and everything went pretty great with that. That got me thinking… I was scheduling GitHub Actions to post daily updates and I also had a separate posting schedule for Tuesday to publish a new post. That seemed both wasteful and limiting at the same time. So I came up with a plan.

Instead of using Github Actions or Cloudflare Workers to deploy my static website, I’m just going to do it myself, from my Synology NAS. This allowed me to have a much more frequent update schedule for my website if there were changes. You may ask: what does it mean to “have changes” for a static website? Changes can be one of the following:

  • I committed some changes that affected the website output
  • A scheduled post becomes live
  • A new mastodon post happened on my feed or there are other changes with the post (likes, favorites, boosts, edits etc.)
graph TD
    deployment[Deployment Script]
    cloudflarepages[fa:fa-cloud Cloudflare Pages]
    hugo[Hugo]
    deployment -- Git --> commits[Git Commits]
    deployment -- Git --> schedule[Scheduled Content]
    deployment -- Mastodon API --> mastodon[Mastodon Updates]
    commits --> hugo
    schedule --> hugo
    mastodon --> hugo
    hugo -- Wrangler --> cloudflarepages

I have a simple script that checks for new changes in my Github repo periodically and does a rebuild to see if there are any scheduled changes. However, you may be wondering how I add mastodon posts to my “static” website. If you experiment enough, you may see that replies get added to my website within 10 minutes (if posted during the day in the European CEST timezone, otherwise it waits until the morning). I made this happen by writing some code with Go and a library called go-mastodon that scans through my previous mastodon posts and puts the updated statuses into a Github repo which is updated for each build of my website. If I want to block content from appearing on my website, I can block the user or mute the reply.

The system works out pretty well. I can improve this, however. I could use a webhook to trigger builds more quickly when there are changes to my Github repo or if there are new posts on mastodon. As it stands, I have to wait up to 10 minutes to see the changes reflected… which isn’t bad, it could just be better. This doesn’t help with scheduled posts, but it would reduce the frequency that I have to check for updates on a timer.

I intend to polish the script that I use to pull down Mastodon updates and publish it as a tool that others can use. I may try to also add support for Blue Sky as well but I make no promises!

Enjoy

I do hope that others find the prompts useful. If you have a suggestion for a prompt or any questions about my setup, please let me know in the comments for this post!

]]>Adding chart.js to Hugohttps://kmcd.dev/posts/hugo-chartjs/Tue, 28 May 2024 00:00:00 +0000https://kmcd.dev/posts/hugo-chartjs/

Let's find out how to add chart.js to your static website built with Hugo. ]]>I recently wanted to add some charts to a blog post and mermaid just wasn’t cutting it. Mermaid didn’t support the options I wanted to use and ultimately wasn’t flexible enough to show a horizontal bar chart with the customization options I wanted. So I went looking for alternatives… and that’s when I found this shen-yu/hugo-chart… layout? for hugo that adds Chart.js support. Chart.js is a great javascript library for creating many kinds of charts with many customization options. Perfect, I thought. As I started using shen/yu-hugo-chart to add chart.js to my site, a few things stood out to me:

  • I felt like adding a “layout” and a new git submodule just for this was fairly extreme just to add support for a single javascript library.
  • The documentation on the chart.js website didn’t match up with what was supported. I then realized that the hugo-chart plugin was using Chart.js version 2 when the latest released version was v4… a whole two major versions behind. Yikes. So not only would this add a new git submodule, which is annoying, but it wasn’t even up-to-date.

At this point, I was ready to throw the entire project aside and do my own thing. This is what I ended up with. It only requires a single file to add the new shortcode. So instead of having a git dependency, just add this file to your shortcodes.

Installation

Add this file at the path layouts/shortcodes/chart.html:

{{- if not (.Page.Scratch.Get "hasChartJS") -}}
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script> Chart.defaults.color = '#fff'; </script>
{{- .Page.Scratch.Set "hasChartJS" true -}}
{{- end -}}

{{- $id := substr (md5 .Inner) 0 16 -}}
<div class="chart">
    <canvas id="{{ $id }}"></canvas>
</div>
<script>
    document.addEventListener('DOMContentLoaded', () => {
        var ctx = document.getElementById('{{ $id }}')
        var options = {{ .Inner | chomp | safeJS }};
        new Chart(ctx, options);
    });
</script>

The .Page.Scratch.Set and .Page.Scratch.Get calls allows me to only have the <script> tag that imports chart.js a single time, which seems cleaner to me than importing it for each call to the chart shortcode.

Once this file is in place, you can now call the chart shortcode like this:

{{< chart >}}
{
  type: 'type',
  data: {...},
  options: {...}
}
{{< /chart >}}

and a chart will be generated using chart.js. A few full examples are below:

Line Chart

{{< chart >}}
{
  type: 'line',
  data: {
      labels: [
        'Jan',
        'Feb',
        'Mar',
        'Apr',
        'May',
        'June',
        'July'
      ],
    datasets: [{
      label: 'My First Dataset',
      data: [65, 59, 80, 81, 56, 55, 40],
      fill: false,
      borderColor: 'rgb(75, 192, 192)',
      tension: 0.1
    }]
  },
}
{{< /chart >}}

See more options on line charts here.

Bar Chart

{{< chart >}}
{
    type: 'bar',
    data: {
        labels: [
          'Dataset 1',
          'Dataset 2',
          'Dataset 3',
          'Dataset 4',
          'Dataset 5',
          'Dataset 6'
        ],
        datasets: [{
            label: 'units',
            data: [36, 50, 3, 36, 63, 79]
        }]
    },
    options: {
        indexAxis: 'y',
        plugins: {
            legend: {
                display: false
            },
            title: {
                display: true,
                text: 'Bar Chart title'
            }
        }
    }
}
{{< /chart >}}

See more options on bar charts here.

Polar Area Chart

You can use any chart available in chart.js. Here’s a polar area chart:

{{< chart >}}
{
  type: 'polarArea',
  data: {
    labels: [
      'Red',
      'Green',
      'Yellow',
      'Grey',
      'Blue'
    ],
    datasets: [{
      label: 'My First Dataset',
      data: [11, 16, 7, 3, 14],
      backgroundColor: [
        'rgb(255, 99, 132)',
        'rgb(75, 192, 192)',
        'rgb(255, 205, 86)',
        'rgb(201, 203, 207)',
        'rgb(54, 162, 235)'
      ]
    }]
  },
  options: {}
}
{{< /chart >}}

See more options on bar charts here.

Some other kinds

I’m going to spare you from giving an example of every type of chart. Please reference the official chart.js documentation to see all available options.

End

I introduced a way to add a fully up-to-date chart.js to your hugo website that avoids many of the downsides of existing solutions. If you liked this post, feel free to send me any comments or questions on mastodon!

]]>Why I'm Rebrandinghttps://kmcd.dev/posts/why-im-rebranding/Thu, 23 May 2024 00:00:00 +0000https://kmcd.dev/posts/why-im-rebranding/ <p> <img hspace="5" src="https://kmcd.dev/posts/why-im-rebranding/cover_hu_56bfd0b4654a8def.webp" /> </p> Find out why the name of this website changed For the longest time, I’ve had the alias sudorandom. At this point, it feels like just another name. So why did I rebrand this website from sudorandom.dev to kmcd.dev? Let’s dig into it.

I am me

My name is Kevin McDonald. The name isn’t going away any time soon, so it simply makes sense to use my name in my personal branding online. And while I think sudorandom is a brilliant pun, I don’t want to immediately drive away those who don’t immediately understand the pun or, worse, think that I just can’t spell ‘pseudorandom’.

I am not currently looking to reach a broader audience, but I think it’s useful to have a branding that isn’t appealing to only a small subset of people. kmcd.dev, since it’s a shortening of my name, transcends any niche linux/computer science reference.

professionalism

I still want to keep a casual tone to the website for now, but I think my website fails the simple test of being able to verbally explain the name of your website without having to awkwardly guess how proficient in the tech world the other person is. I think kmcd.dev easily passes this test and is frankly very nice to type. I’ve typed it a few times already in this post and it just feels snappy.

You should see how it feels by sending my website to a few friends. (hey! it’s worth a shot!)

SEO

Initially, when I searched for “sudorandom” on google, it would redirect me to “pseudorandom”. It was literally impossible to search for my website using the exact name. This appears to have been changed but I think it does signal a problem with the name from an SEO standpoint.

I know my name isn’t the most unique. The more famous of the men sharing my name is a Canadian actor. But he’s quite a bit older than me so what I should really be fighting for is to beat the Scottish footballer, who was born in the same year I was. But my point here is that using an alias like sudorandom doesn’t help my name SEO at all.

Finally, it’s four letters long. With a name this short it must have some SEO magic.

longevity

I’m not sure what I will want to focus on in the future. I’ve had periods where I was obsessed with a space-based MMO and another period where I loved learning about optical networking. Who knows, I may take up teaching people cringy Danish/English puns after my wife and daughter ban me from the practice around them.

The beauty of kmcd.dev is that it doesn’t restrict me to a specific topic. As my interests evolve, the website name remains relevant, allowing me to explore new technologies and share my knowledge with you.

skål!

Overall, rebranding to kmcd.dev offers a clearer, shorter, and more professional website name that strengthens my personal brand in the long run.

iuuoekifqsusemreo
]]>Benchmarking gRPC (golang)https://kmcd.dev/posts/benchmarking-go-grpc/Tue, 21 May 2024 00:00:00 +0000https://kmcd.dev/posts/benchmarking-go-grpc/

Let's so how fast gRPC can go in Go. ]]>Hey everyone, as you know from my previous posts I’m a big fan of gRPC especially when working with Go. It’s been my go-to tool for building remote procedure calls (RPCs) for a while now. It has been extremely reliable for providing high-performance RPC systems. Originally, there was only one choice: Google’s grpc-go library. It came before HTTP/2 support landed in the Go standard library, and being from Google, the creators of gRPC, it seemed like the natural fit. But is it actually the best gRPC library for Go?

As time marches on, and with it, some concerns about grpc-go have emerged:

  • Experimental Features: Many important features are labeled experimental, meaning they could change or even disappear in future updates. This can cause compatibility issues down the road.
  • Design Quirks: Some design choices feel a bit odd. Because it doesn’t integrate well with any standard HTTP middleware or tooling then it makes for a more fragmented series of libraries that are specific to gRPC. Furthermore, widely used features are often deprecated, moved and removed, breaking compatibility for existing projects. There’s also been a few missteps around API management.
  • Separate Server Dependency: grpc-go requires a separate server instead of leveraging the standard Go HTTP server. This might not be ideal if you need to serve other functionalities alongside your gRPC endpoints, like OAuth or large file uploads.

The lineup

Because of these reasons, I decided to explore some alternatives to grpc-go and if I require similar performance, I should make sure that the alternatives are in the same ballpark of performance. This post dives into benchmarks comparing the performance of three different ways to run a gRPC server in Go:

  • grpc-go: The official library from Google. See my implementation here.
  • grpc-go-servehttp: A variant of grpc-go that allows using the standard Go HTTP server. This is useful when you want to add extra HTTP routes alongside the gRPC ones without having to manage another HTTP server instance on a different port. See my implementation here.
  • connectrpc: A third-party gRPC library that supports the Go HTTP server (and gRPC-Web and the Connect protocol). I wrote about this previously. See my implementation here.

Before diving into writing my own benchmarks, I wanted to see what exists today:

  • The gRPC authors have published their own benchmark results. In these results, it looks like grpc-go is among the top performers.
  • LesnyRumcajs/grpc_bench has a very good suite of gRPC frameworks in many different languages. The latest “official” results are here. This paints a significantly different picture where grpc-go is in the “middle of the pack”.

I wanted to reproduce what happens in grpc_bench but with Go’s CPU profiling enabled so that I can dig in and see what parts of the code are taking longer. After trying to bootstrap it to grpc_bench for a while I ended up just making my own repo with benchmark tests just for Go libraries.

Aside: Don’t worry, before I did this I contributed back a change that updates all of the relevant libraries and the version of Go being used. I guessed that this might be some of the reason for the low ConnectRPC numbers because it was using a super old version from before they moved to connectrpc.com/connect as the canonical import.

In my new benchmark repo, I used ghz to run the benchmarks. I added the ability to capture pprof profiles while running the benchmark.

Test One: The Empty Message Test

This scenario uses an empty message to gauge the overall performance of handling requests, without any of the overhead of protobuf parsing.

{
    "proto": "proto/flex.proto",
    "call": "flex.FlexService/NormalRPC",
    "total": 101000,
    "skipFirst": 1000,
    "concurrency": 50,
    "connections": 20,
    "data": {},
    "max-duration": "300s",
    "host": "0.0.0.0:6660",
    "insecure": true
}

(source)

Results

The results are measured in Requests Per Second (RPS) which means that higher is better. This was run on an Intel NUC:

  • Intel(R) Core(TM) i7-8705G CPU @ 3.10GHz
  • 8GB of DDR4 RAM

This is a pretty outdated machine so if you run these benchmarks locally, you should see better performance.

This benchmark measured the performance of handling requests with an empty message. This means no actual data is being exchanged, so it focuses on the overhead of handling gRPC requests themselves.

  • grpc-go was the clear winner, processing over 20,000 requests per second.
  • connectrpc followed closely behind at around 16,000 requests per second.
  • grpc-go (with ServeHTTP) lagged behind at roughly 14,000 requests per second.

Test Two: Something slightly more realistic

This scenario uses a larger message with most of the protobuf types to exercise all of the different code paths when marshaling and unmarshalling protobuf messages.

{
    "proto": "proto/flex.proto",
    "call": "flex.FlexService/NormalRPC",
    "total": 101000,
    "skipFirst": 1000,
    "concurrency": 50,
    "connections": 20,
    "data": {
        "msg": {
            "doubleField": 123.4567,
            "floatField": 123.4567,
            "int32Field": 1234,
            "int64Field": 1234567,
            "uint32Field": 1234,
            "uint64Field": 1234567,
            "sint32Field": 1234,
            "sint64Field": 1234567,
            "fixed32Field": 1234,
            "fixed64Field": 1234567,
            "sfixed32Field": 1234,
            "sfixed64Field": 1234567,
            "boolField": true,
            "stringField": "hello world",
            "msgField": {},
            "repeatedMsgField": [{}, {}],
            "optionalMsgField": {}
        }
    },
    "max-duration": "300s",
    "host": "0.0.0.0:6660",
    "insecure": true
}

(source)

In this scenario, I also tested with and without an alternative marshal/unmarshal implementation provided by vtprotobuf, which can further optimize performance. This was omitted from the last test because there shouldn’t really be much performance difference when you’re parsing an empty message.

Results

Now here are the results of a test that uses a relatively complex message so we can see how protobuf parsing factors into the benchmarks:

The goal of this benchmark was to introduce a larger message with various data types to better simulate real-world gRPC communication. It examines how well each library handles message marshaling and unmarshaling on top of the request processing.

  • Again, grpc-go came out on top, processing over 16,000 requests per second.
  • Even with a complex message, connectrpc stayed competitive at around 14,000 requests per second.
  • grpc-go (with ServeHTTP) stayed in last place with nearly 13,000.
  • Note that vtprotobuf does help out a decent amount. vtprotobuf is being used in a mode that has zero extra hand-written code. The plugin also allows another optimization using pools which does require code changes. I may add that setup to these benchmarks at some point.

The Takeaway

If performance is the most important aspect of your application, stick with grpc-go. But if you want to add extra HTTP endpoints, add gRPC-Web support without an extra proxy, support for HTTP/1.1 or be able to just use curl your gRPC endpoints you might want to look into ConnectRPC.

I wouldn’t recommend using grpc-go with ServeHTTP because it’s slower than ConnectRPC while offering fewer features. Maybe only if you’re really entrenched into the grpc-go ecosystem should you consider using it.

Open Questions and Next Steps

I am curious about exactly why supporting the ServeHTTP interface causes such a drastic difference in performance for grpc-go. There must be some cost to supporting the ServeHTTP interface that you don’t incur when implementing the HTTP/2 server or some kind of optimization that you can’t do in a general case that grpc-go can do. If you have any ideas about this, I’m happy to hear them.

I do have to give the standard disclaimer about benchmarks. This is an artificial benchmark that was run on my underpowered, hobbyist Intel NUC and while grpc-go performs well, the difference between it and the other methods is probably negligible for use cases that don’t require peak performance.

The approach that ConnectRPC has with using ServeHTTP with a normal http.Server provided by the standard library means that it’s likely to “just work” with http/3 when it lands in the Go standard library. This is exciting for these artificial benchmarks (and some real use cases) where creating new connections is a significant part of the overhead.

See the full benchmark source here at github.com/sudorandom/go-grpc-bench. The repo does contain CPU profile captures alongside the results which I was attempting to use to narrow down the performance differences, but I haven’t found anything interesting there yet. If I succeed at finding anything interesting, I may post another update here!

]]>Blog Updatehttps://kmcd.dev/posts/blog-update/Tue, 14 May 2024 00:00:00 +0000https://kmcd.dev/posts/blog-update/ <p> <img hspace="5" src="https://kmcd.dev/posts/blog-update/cover_hu_b2f16ab6ff96474e.webp" /> </p> An update about the state of the blog. Hello! This will be a lighter update for this blog. I’m going to recap some of the changes I’ve made to it over the last couple of months, the technologies that power it, and my plans for the future.

Schedule

I’ve been keeping up a one-post-a-week cadence. This has started pretty well because I was in a hyper-productive mode and built up a fairly sizable backlog but I’m now in a less productive phase. Therefore, some weeks in the future may be “softer” than others and I may resort to link posts, commenting about recent events, and other lower-effort ways of getting more words onto the website. This post is one of the lower-effort posts since it doesn’t require research!

Importing mastodon posts

Mastodon posts are automatically downloaded and re-hosted on this website. This serves a few purposes:

  • At the bottom of every blog post, the corresponding mastodon post is referenced. I hope this will be a good way to point readers who want to make a comment on the article or to give me a thumbs up.
  • Allows me to reference Mastodon posts inside of blog posts (without relying on an external service or iframes). I haven’t used this yet, but it may come in handy.
  • It also just acts as my personal mastodon archive. Because I have this history, I don’t need some other mechanism to back up my public Mastodon posts. I strongly believe in content ownership, which is the whole reason I don’t use many of the existing blog publishing platforms except for syndication.

Prompt of the day

I’ve started posting a daily writing prompt every day on my Mastodon account. This is automatically posted from a combination of future-dated pages in Hugo, scheduled Github Actions and Echofeed. I feel like these prompts help me write SOMETHING every day. I typically just respond to the mastodon post. There’s been an encouraging amount of participation from others with this. It was fun to come up with the first month’s worth of questions. Follow me on Mastodon to participate in these prompts!

RSS

I’ve made some improvements to how RSS feeds are generated so that links and images use absolute paths in all cases, which allows you to see these images in your RSS feed if your reader supports that. It’s strange how awful the “out of the box” experience is with Hugo and RSS. I feel like at least some of my issues should be handled automatically.

Small Refinements

I originally used the Hello Friend 4s3ti theme for my blog. I’ve slowly made small adjustments, moving more and more layouts away from the theme into the main blog section. At some point, I may copy over the rest of the resources and use a completely custom theme which will allow me to delete features I don’t ever plan on using and more easily customize. But anyway, I like the style that I’ve settled on. It fits me.

Projects Page

I updated the projects page to be more… clear. The image gallery format just wasn’t working to show what I worked on. I’m not an artist and I should come to terms with that! See the page here.

Links Page

I have become a fan of the “small web” and I feel like link pages bring some of that small web feeling back. See mine, here.

Technologies

Some may be interested to know how this blog is built. The source code is available here on GitHub at sudorandom/kmcd.dev. An enterprising reader could get sneak-peaks by watching this repo 👀

Cloudflare

As you may be able to tell from DNS, I use Cloudflare as a CDN, DNS, and firewall. Cloudflare has an incredibly generous free tier, which I use extensively.

GitHub Pages

The actual website is hosted on Github Pages. GitHub also has a generous free tier here, but because of Cloudflare, I’m not sure it actually gets a lot of non-cached traffic. I think I could easily switch to using Cloudflare R2 if I needed to. The killer feature for GitHub pages is having easy integration with GitHub Actions, which I use to build and deploy the website. I simply push to my main branch and the process is handled from there. I also rely on GitHub Actions to publish scheduled content for me and to import Mastodon post updates for me.

Hugo

I use Hugo for my static website generator. I prefer it due to the Go templating style that I was already very familiar with along with the ease of use. With Hugo, I don’t need to download 100 different npm packages to build my website. It does have its quirks, though and some things I do get frustrated about.

EchoFeed

EchoFeed is a new but welcome addition to my tech stack. This service consumes my RSS/Atom/jsonfeed feeds and makes corresponding posts to… somewhere. Currently, it supports Mastodon, bluesky, discord, Github, webmentions, webhooks, and a few others. Here’s how I use the service:

  • updating mastodon when there is a new post
  • updating mastodon with a new prompt of the day
  • updating the github repo with new mastodon posts from myself

Until Next Time

I’m always looking for ways to improve this blog, so let me know on Mastodon what you think! In the meantime, I’m planning to work on the gRPC From Scratch series in the coming weeks. Thanks for reading!

01110011 01100101 01100101 00100000
01111001 01101111 01110101 00100000
01110011 01110000 01100001 01100011 01100101 00100000
01100011 01101111 01110111 01100010 01101111 01111001
]]>gRPC From Scratch: Part 3 - Protobuf Encodinghttps://kmcd.dev/posts/grpc-from-scratch-part-3/Tue, 07 May 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-from-scratch-part-3/

Let's look under the hood of gRPC by getting into the weeds of protocol buffers. ]]>In the last two parts, I showed how to make an extremely simple gRPC client and server that… kind-of works. But I punted on a topic last time that is pretty important: I used generated protobuf types and the Go protobuf library to do all of the heavy lifting of encoding and decoding protobufs for me. That ends today. I’ll start by looking at the protowire library directly, which is a bit closer to what is actually happening on the wire. The library includes a fun disclaimer:

For marshaling and unmarshaling entire protobuf messages, use the google.golang.org/protobuf/proto package instead.

Am I going to listen to this solid advice? No! I want to know how this works! No reflection and no reliance on generated code. All of the code in this post are taken from unit tests available here so feel free to download the tests and play around with it locally. Now, let’s get started.

Wire Types

I discuss in my Inspecting Protobuf Messages post that protobuf only has a small handful of types. Here they are again:

ID Name Used for
0 VARINT int32, int64, uint32, uint64, sint32, sint64, bool, enum
1 I64 fixed64, sfixed64, double
2 LEN string, bytes, embedded messages, packed repeated fields
3 SGROUP group start (deprecated)
4 EGROUP group end (deprecated)
5 I32 fixed32, sfixed32, float

For today we’re only going to implement some of the LEN and VARINT wire types. Protobuf messages are a series of key-value pairs. Keys are “field numbers” and values are one of the types in the table above. To save space, protobuf decided to encode both the field number and wire type into a single byte. The lower three bits are the wire type and the other 5 (and maybe more, more on that later) are used for the field number. This is what the “Tag-Length” part looks like for field number 1 with the VARINT wire type:

0000 1000

You can shift the three least significant bits off to get the protobuf wire type and the rest is used for the field number. It is at this point that the bitwise operators start becoming useful:

(field_number << 3) | wire_type

So… 3 bits for the wire type leaves room for 8 different options so protobuf has room for two more wire types before they have to break compatibility with older versions or add another byte to describe more types. And 5 bits for the field number which leaves room for…… 32 different fields??? What?! You can’t have a message that has more than 32 fields?! Why is no one talking about this glaring limitation where the number of fields is limited to a four-year-old’s counting ability?! Well, obviously this is not true and, at this point, I am now forced to explain what protobuf refers to as Base 128 Varints.

Big numbers

In the previous example, we saw VARINT take a single byte. What does it look like when your number is too big? Where does the variableness part of VARINT come in? The protobuf encoding uses what it calls Base-128 Variable Integers in several places. “Base 128” means you can count to 127 before rolling over to the next “digit” (or in this case, byte). The most significant bit is used as a continuation bit, which is a signal that there’s at least one more byte worth of data to complete this integer. Let’s decode one for practice:

11000000 11000100 00000111   // Original inputs.
 1000000  1000100  0000111   // Drop continuation bits.
 0000111  1000100  1000000   // Convert to big-endian.
 000011110001001000000       // Concatenate.
 (1 × 2^16) + (1 × 2^15) +   // Convert binary to decimal
 (1 × 2^14) + (1 × 2^13) +   // because we're not a computer.
 (1 × 2^9) + (1 × 2^6)
 = 123456                    // Interpret as an unsigned 64-bit integer.

Next, I will show you some go code that can do this VARINT encoding process for us. Note that this code is actually in Go’s standard library and makes reference to protocol-buffers directly in the documentation; (source).

// AppendUvarint appends the varint-encoded form of x,
// as generated by [PutUvarint], to buf and returns the extended buffer.
func AppendUvarint(buf []byte, x uint64) []byte {
	for x >= 0x80 {
		buf = append(buf, byte(x)|0x80)
		x >>= 7
	}
	return append(buf, byte(x))
}

What’s happening here? Let’s break it down:

  • The core logic is in a for loop that continues as long as x is greater than or equal to 128 (represented by 0x80 in hexadecimal). Why 128? That’s the highest number (in hexadecimal) that you can count to only using 7 bits. Remember that the most significant bit (MSB) is used as a continuation bit so we can only use 7 out 8 of the available bits in each byte.
    • The next line appends the uint64 argument truncated to a byte, so the last 8 bits. One of those bits isn’t actually used because of the |0x80 part of the line. This combines our extracted 7 bits with 0x80 to ensure that the continuation bit is set to true.
    • The next line uses the right shift operator to shift 7 bits off of our current uint64 value because we’ve “dealt with” these bits already. Next, we check the for-loop condition again until we get a number lower than 128.
  • Finally, we append the final byte as-is because we know that it’s less than 128 which assures us that the MSB is not set, so we are done building this integer.

Next, we have the decoding code for VARINT. That looks like this (source):

// Uvarint decodes a uint64 from buf and returns that value and the
// number of bytes read (> 0). If an error occurred, the value is 0
// and the number of bytes n is <= 0 meaning:
//
//	n == 0: buf too small
//	n  < 0: value larger than 64 bits (overflow)
//	        and -n is the number of bytes read
func Uvarint(buf []byte) (uint64, int) {
	var x uint64
	var s uint
	for i, b := range buf {
		if i == MaxVarintLen64 {
			// Catch byte reads past MaxVarintLen64.
			// See issue https://golang.org/issues/41185
			return 0, -(i + 1) // overflow
		}
		if b < 0x80 {
			if i == MaxVarintLen64-1 && b > 1 {
				return 0, -(i + 1) // overflow
			}
			return x | uint64(b)<<s, i + 1
		}
		x |= uint64(b&0x7f) << s
		s += 7
	}
	return 0, 0
}
  • It loops through each byte:
    • Checks for overflow (reading past expected max size).
    • Handles the last byte (if the continuation bit is 0):
      • Validates it and returns the decoded value.
    • Processes continuation bytes:
      • Extracts data bits and combines them with the accumulated value, shifting them to their correct position. The expression b&0x7f ensures that the continuation bit is not included in the result by using a bitwise AND with the current byte and 0x7f, which looks like this in binary 01111111.
      • s keeps track of the total bits already processed. It increments by 7 because that’s how many bits we can use for each byte because of the continuation bit.
  • If the loop finishes without a valid ending, it signals an error.

As a short aside, you should probably know that the modern protobuf library doesn’t use these functions from the encoding/binary standard library package. I suspect that they were initially used but the current protowire implementation doesn’t use for loops at all because it has performed an optimization technique called loop unrolling. Here’s AppendVarint and ConsumeVarint that are now used in the modern protobuf library. That code looks insane but it is likely a bit faster than the version of the code that I just showed you.

As a test for comprehension of this last section, you should now understand that smaller numbers take up less room on the wire. That’s true for all numeric types except for the fixed-length types: I64 and I32. Encoding 20 in protobbuf takes a single byte on the wire but 1234 would take two bytes.

Integers

Now that we know how the VARINT wire-type works we now have enough raw material to write some protobuf packets. Again, we need three things to make a message with a single field: Field number, wire type, and the encoded value. As mentioned earlier, the field number and wire type are merged into a single VARINT value using the formula: (field_number << 3) | wire_type. This essentially means that the the least significant bits are reserved for the wire type and the rest are used for the field number, and we expand using the Base-128 varint method above if the field number needs more bits to be represented. Now, let’s write a full field in protobuf! Here’s how we encode a message with a single int32 field:

1: 1234

var buf []byte
buf = protowire.AppendVarint(buf, uint64(1<<3)|uint64(0))
buf = protowire.AppendVarint(buf, 1234)

And… that’s it. We’ve fully encoded probably the simplest (non-empty) protobuf message. We can check that it works by running a test with the “real” protobuf library:

func TestEncodeRaw(t *testing.T) {
	var buf []byte
	buf = protowire.AppendVarint(buf, uint64(1<<3)|uint64(0))
	buf = protowire.AppendVarint(buf, 1234)

	res := gen.TestMessage{}
	require.NoError(t, proto.Unmarshal(buf, &res))
	assert.Equal(t, int32(1234), res.IntValue)
}

By the way, these tests are using a protobuf file that looks like this:

syntax = "proto3";

package example;

message TestMessage {
  // Basic types
  int32 int_value = 1;
  uint64 uint_value = 2;
  bool bool_value = 3;
  string string_value = 4;

  // Repeated fields (varint encoding)
  repeated int32 repeated_int_value = 10;

  // Nested message
  message NestedMessage {
    string nested_string = 1;
  }
  NestedMessage nested_message = 11;
}

All of the field numbers and corresponding types in this article match up to this protobuf file.

Okay, let’s show what it looks like to read this message:

tagNumber, protoType, n := protowire.ConsumeTag(b)
...
b = b[n:]
i, n := protowire.ConsumeVarint(b)

Note that I omitted error handling and in a real scenario we need to check the tagNumber and protoType to decide how to read this field.

Strings/byte arrays

Next, we’re going to start with strings and byte arrays. These use the LEN wire type. This type is a little more complex than VARINT. It’s composed of a VARINT that tells us the size of the LEN field followed by the actual content in bytes. For LEN this content can be a string, a byte array, an embedded message or packed repeated fields. Here’s what a field looks like:

  • Field Tag Byte (field number along with the field type, set to 2 for the LEN type)
  • Byte size of our content as a VARINT
  • The actual content

Encode:

var buf []byte
buf = protowire.AppendTag(buf, protowire.Number(4), protowire.BytesType)
buf = protowire.AppendString(buf, "Hello World!")

Decode:

tagNumber, protoType, n := protowire.ConsumeTag(b)
b = b[n:]
i, n := protowire.ConsumeVarint(b)

Integer Arrays (Packed)

Packed repeated fields are a space-saving optimization for integer arrays. It’s a feature that’s enabled by default for most repeated scalar types when using the proto3 syntax. Instead of encoding each element individually as a separate field/value pair an entire array is packed into a single LEN type. Therefore it looks like the following for repeated int32 types:

  • VARINT of our field number with the three least significant bits being reserved for field type for LEN which is 2. This is the
  • The raw integer data, also encoded with VARINT for our int32 type, one after another

Here’s an example of how to encode a packed repeated integer field:

arr := []int32{100002130, 2, 3, 4, 5}
var buf, buf2 []byte
buf = protowire.AppendVarint(buf, uint64(10<<3)|uint64(2))
for i := 0; i < len(arr); i++ {
	buf2 = protowire.AppendVarint(buf2, uint64(arr[i]))
}
buf = protowire.AppendVarint(buf, uint64(len(buf2)))
buf = append(buf, buf2...)

Notice that we are writing the list of int32 values to a second buffer. I do this so that we can know the size of the encoded/packed int32 values so that I can properly set the size for the encapsulating LEN wire type.

Decoding a packed repeated field is similar. First, you read the field tag (like always). Then we are using protowire.ConsumeBytes that reads the entire LEN value into a byte array. Then you read varint values until you exhaust the buffer. Remember that varint values have that continuation but so the protowire.ConsumeVarint function knows when to finish reading each varint value.

Here’s what that looks like in code:

tagNumber, protoType, n := protowire.ConsumeTag(b)
...
b = b[n:]
int32buf, n := protowire.ConsumeBytes(b)
b = b[n:]
...
res := []int32{}
for len(int32buf) > 0 {
	v, n := protowire.ConsumeVarint(int32buf)
	int32buf = int32buf[n:]
	res = append(res, int32(v))
}

Conclusion

In this part of the series, we’ve taken a deep dive into the world of manual protobuf encoding. We explored a few wire types and the primitives that they are built on, like Base-128 varints, and built functions to encode and decode basic protobuf data types. While for most use cases, you’ll probably want to leverage the efficiency and convenience of generated code and libraries like golang/protobuf, understanding how to manually encode these messages can be a valuable asset for debugging, creating a protobuf library implementation in your favorite new language or for possibly creating your own binary encoding that’s better in some way than protobuf.

In the next part of the series, we will put more pieces together with a layer on top of our encoding code and integrate this protobuf library into the client and server that we made in parts 1 and 2. Build a real gRPC client and server that uses protobufs for data exchange!

]]>Tracking the Winshttps://kmcd.dev/posts/tracking-the-wins/Tue, 30 Apr 2024 00:00:00 +0000https://kmcd.dev/posts/tracking-the-wins/ <p> <img hspace="5" src="https://kmcd.dev/posts/tracking-the-wins/cover_hu_840d211415a7c25.webp" /> </p> Stay motivated and avoid burnout Mental health is hugely important for software engineers. When I first started coding, the thrill of solving problems and getting that hit of dopamine from the instant feedback loop was incredible. It’s easy to forget those initial challenges and accomplishments as you progress, though, and problems that used to look like unscalable walls turn into speed bumps. That’s why I believe regularly reflecting on your work is so valuable. Otherwise, you might find yourself dwelling on the struggles and forgetting the successes you’ve achieved along the way.

Writing down the wins

Here’s how I track my progress: Every week or two, I set aside some time to quickly document my achievements, even small ones. I keep it concise to avoid procrastination. I have some anxiety around tasks with too many unknowns, so a simple, mechanical process works best for me. I’m also a very forgetful person, so I usually have to review artifacts created by the work that I do, like following digital breadcrumbs that I leave around. Here’s a list of where I might find these things:

  • issues I’ve worked on
  • pull requests
  • emails
  • edits to documentation
  • slack messages

The more concrete evidence, the better, but some achievements don’t leave clear evidence so be sure to capture interesting anecdotes as they happen. Anything that has allowed me to progress forward should be highlighted, even if it is a result of compromise or realizing that you made a mistake.

I also apply a similar strategy to my personal life. Did I stay on top of my Danish lessons and language learning apps? Did I have periods of being ill during this period? Was I physically active, by cycling, exercising, or simply taking the stairs? Tracking these things helps me see the broader picture.

Quarterly reviews

To pair with the weekly tracking. I conduct a more in-depth reflection every 3 months. Here, I add my thoughts to these accomplishments. I create a narrative of the past quarter, encompassing both highs and lows, as well as the inevitable “neutral” experiences. Three months seems like a good timeframe to identify any significant shifts in my attitude or goals. It allows me to assess my progress on past goals and set new ones for the future. I believe this time frame is MUCH better than the yearly cycle of New Year resolutions. Twelve months is way too long. One month is usually not enough to make good progress on goals or to establish habits. Three months seems just right to me. So here’s the list of what might appear on a quarterly review:

  • A “narrative” of the last three months using concrete examples from my journal to back it up.
  • Report on the progress of previous goals.
  • Create new goals or revise the older goals if needed.
  • Pictures and video of important moments.
  • Reflect on big family budget items and see if they align with my priorities in life.

Yearly themes

Yes, I am a Cortexan and podcast episodes outlining yearly themes are among my favorites. Yearly themes give a framework for me to push myself to do things I want to do while allowing me to adjust (every quarter) exactly what it means to accomplish my theme. Sometimes life throws additional challenges. For example, it’s silly to fail at achieving a goal of “running a mile every day” if you break your leg.

For me, this year is “The Year of Building”. Here’s what I wrote about my theme shortly before the year began:

I want to build some cool things. I want to make some useful open source projects. I want to implement some neat solutions at work and actually prove them out. My follow-through sometimes lacks when I lose interest in things. I want to make more cool blog posts.

I also want to build on my Danish by continuing my Danish classes, self study with and using it more “in the wild”.

I think I’m doing a good job with this one. You may notice though that the latest batch of blog posts didn’t start in January. I “slacked” on this goal for over a month but in my theme framework, it’s fine because I was ramping up my Danish classes again. My yearly theme allows me a lot of grace on the day-week-month scale while hopefully keeping me honest on the yearly scale and allowing for adjustment of individual goals to match reality.

Try it yourself

Journaling and goal-setting are extremely personal things so your process may vary a lot from mine. But one thing is likely true of everyone: by keeping track of your wins, both big and small, you can maintain motivation and see your progress over time. Give it a try and see how it impacts your software engineering (and life) journey!

]]>Visualizing the Internet (2024)https://kmcd.dev/posts/internet-map-2024/Tue, 23 Apr 2024 00:00:00 +0000https://kmcd.dev/posts/internet-map-2024/ <p> <img hspace="5" src="https://kmcd.dev/posts/internet-map-2024/cover_hu_8c82ca350b1151a0.webp" /> </p> Map of the Internet including undersea cables and internet exchanges. I recently updated my Map of the Internet visualization that shows all of the undersea Internet cables that run along the bottom of the oceans and seas and the Internet exchange points that offer peering. Together these show a pretty good view of the physical aspects of the internet. This time, I generated a video that shows the evolution of our current internet along with some interesting stats from each year. Enjoy:

Video

YouTube: Internet Map (2010-2024)

Still images

I, of course, also generated new versions of the static maps. Below is an overview of the entire world. Click on the image below to zoom/pan the full-resolution version or click the link under the image to download it:

Map of the Internet
Click here for full resolution image (warning, it’s big)

So what are you looking at? The lines are submarine cables and the circles are cities with Internet Exchange Points (IXPs).

Submarine Cables

The squiggly lines on the map are submarine communications cables. Submarine cables are truly a technological wonder. They carry vast amounts of data across the world’s largest oceans. Yes, they just lie at the bottom of our oceans. Yes, it is crazy. From route selection, cable manufacturing, and specialized cable-laying ships; it’s all incredible. Even more normal aspects of fiber optic cables like having in-line amplifiers become far more difficult when you cannot drive a truck out to fix something. You can’t drive a truck to the bottom of the ocean. Well, you can but you won’t be much use once you get there and good luck doing it more than once. Anyway… from amazing robots that can locate and assist in bringing cables to the surface to in-water fiber optic repair habitats every bit of submarine cable deployment and operations is extreme and fascinating.

Laying and operating submarine cables is a complex and expensive undertaking, but it is vital for our interconnected world. Note how few of these cables exist in total. They connect our content digitally and culturally but there are only 549 active submarine cables right now in the world. That truly is an extremely small amount but you should note that many more terrestrial cables connect on land. Too many for us to map or know about. So instead, I used data about internet exchange points to show the scale of the internet on land.

Internet Exchange Points

The circles in the maps above are all buildings called “Internet Exchange Points”. These buildings (and sometimes closets) are specialized data centers that have a uniquely large number of fiber connections from surrounding data centers, other internet exchanges and long-haul cables. These buildings are “meeting points” for Internet service providers (ISPs), cloud providers, content delivery networks (CDNs), and large enterprises like IBM and Apple and many could say that it is “where” the Internet lives. This is where networks interconnect or “peer”. Many of these Internet exchanges charge a monthly fee to participate in the exchange but who everyone peers with and the financial details around the arrangement is up to each company. Some companies like Apple or Google will peer with anyone because it brings Google devices closer to Google services and reduces latency and cost while improving reliability. Traffic served through peering, for Google, is traffic that they do not have to pay backbone providers for.

These exchanges usually have a “gravity” to them. It’s more valuable to be in an exchange where there are already many other networks present. However, we need to remember to avoid putting all of our eggs into a single basket. Exchanges that are too large create a single point of failure, so large internet-connected cities need to have several of these exchanges.

Looking at each region

Okay, let’s look a bit closer at each region. In terms of infrastructure, you can see which areas have more investments.

pie title Peering By Region
    "Europe" : 1390
    "North America" : 371
    "South America" : 236
    "Africa" : 47.3
    "Asia" : 308
    "Oceania" : 44.7

Europe

Europe

Europe reigns supreme in peering traffic, boasting a staggering 1.39 petabits of advertised peering bandwidth. This dominance can be attributed to several factors:

  • European nations tend to have well-developed economies that prioritize high-speed internet infrastructure. This translates to greater investment in network backbones and internet exchange points (IXPs).
  • Europe has a dense network of IXPs, particularly in major cities like Frankfurt, Amsterdam, and London. These hubs facilitate efficient peering between numerous networks, reducing reliance on expensive long-haul transit.
  • Europe has a regulatory environment that encourages open peering practices. This fosters competition and innovation among internet service providers (ISPs), ultimately benefiting end users with lower costs and improved performance.

North America

North America

North America follows closely behind Europe with a respectable 371 terabits of peering traffic. Similar to Europe, factors like strong economies, a well-developed internet infrastructure, and the presence of major IXPs contribute to this robust peering ecosystem. However, North America’s peering traffic might be slightly lower due to a larger geographical footprint compared to Europe.

I find it interesting that there are two submarine cables planned for 2024 (and two more in 2026) going to Myrtle Beach in South Carolina. It might be a decent time to start an internet exchange around that area since there’s a good amount of submarine cable capacity going through that area now. This could improve peering options and potentially lower latency for users in the southeastern United States

Asia

Asia

Asia exhibits a significant peering presence with 308 terabits of traffic. The region’s economic growth and increasing internet penetration are driving forces behind this. But it’s important to note:

  • There’s a significant disparity in development levels across Asian countries. While developed nations like Singapore and Japan boast strong peering infrastructure, others might lag behind.
  • Government Regulations: Some Asian governments might have stricter regulations on internet traffic, potentially impacting peering arrangements between ISPs.

You may be trying to remember what the island is in the center-right of this picture that many things appear to connect to. That, my friends, is Guam, an unincorporated territory of the United States.

Africa/South America

Africa

South America and Africa have the lowest peering traffic at 236 terabits and 47.3 terabits, respectively. This can be attributed to:

  • These regions are still in the development stages when it comes to internet infrastructure. Lower investments in IXPs and network backbones can limit peering opportunities.
  • The number of IXPs in these regions might be lower compared to Europe and North America.
South America

For South America there seems to be a geographically dominant peering spot: Buenos Aires in Argentina.

How it’s made

The images are generated the same way I did in the last version, so refer to that article for more context. Generating the video, however, required a little bit more technology. I primarily used a tool called timecut that records the SVG animations for me and encodes it as an mp4 file using ffmpeg. I used another ffmpeg call to mix in the audio song. By the way, I used some trial and error to figure out that the tempo of the song is 110bpm. To find out how long to sleep before changing to the next frame I did some simple math… but I also double-checked it with this nifty website that shows how long in milliseconds there are between between each beat with different tempos. It may have been easier to do this “manually” with video editing software but I enjoy the fact that I can generate the entire video from the command line. Because the fps is so low the 4k video file that I generate only takes up 18MB.

Here’s the updated pipeline:

graph LR
    submarinecables[Submarine Cable Database] --> golang[Go Script]
    peeringdb[PeeringDB] --> golang[Go Script]
    geocities[Geolocation Database] --> golang[Go Script]
    golang[Go Script] --> node[Node JS Script]
    node[Node JS Script] --> svg[SVG]
    node[Node JS Script] --> animated-svg[Animated SVG]
    svg[SVG] --> convert[ImageMagick Convert] --> png[PNG]
    svg[SVG] --> morgify[ImageMagick Morgify] --> jpg[Displate JPG]
    animated-svg[Animated SVG] --> timecut[Timecut/ffmpeg] --> mp4[MP4]
    style svg stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style animated-svg stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style jpg stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style png stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style mp4 stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;

Closing Thoughts

This visualization provides a fascinating glimpse into the physical infrastructure that underpins the Internet. As our reliance on the internet continues to grow, so too will the need for robust and resilient network infrastructure. By understanding the physical underpinnings of the internet, we can appreciate its ingenuity and work towards ensuring its continued growth and accessibility for everyone.

References:

]]>Making Greppable Codehttps://kmcd.dev/posts/greppable/Tue, 16 Apr 2024 00:00:00 +0000https://kmcd.dev/posts/greppable/

Meaningful names help make code searchable so think twice before naming everything 'entity' ]]>Have you ever felt like you’re sifting through lines of code, desperately searching for that elusive function or variable? Fear not, for there’s a way to make your code more discoverable: greppable naming conventions.

What is Greppability?

Drawing inspiration from the powerful Linux command grep used for text searching, greppable code refers to code that’s easy to search through using clear and meaningful names. Just like grep helps you quickly find specific text within a file, greppable names allow you to efficiently locate relevant code sections by searching for terms that reflect their purpose.

Crafting Greppable Names

Here are some key principles to follow:

  • Specificity: Ditch generic names like processData() or utils. Opt for terms that reflect the specific content (cleanAndValidateOrderData(), orderId).
  • Priority: It is far more important for filenames, class names and methods to have specific names than it is for variables inside of functions. Variables inside of functions already have a lot of implied context, like the filename, class (if your language uses classes) and the function that the variable appears in. All of that extra context should be greppable but the variables probably don’t need to be.
  • Consistency: Maintain a consistent naming style throughout your codebase (e.g. camelCase or snake_case) and always use the ‘standard’ for your language or framework.
  • Verbs when naming methods: Methods describe actions. Use verbs at the beginning of method names to convey their purpose (calculateOrderTotal(), updateCustomerRecord()).
  • Abbreviations with Caution: Use abbreviations sparingly and only for widely recognized terms (e.g. HTTP, XML). Overuse can hinder readability. If your variable names look like 2010s-era tech startups, you’re probably doing it wrong.
  • Dynamic Dispatch: Doing some more complex programming patterns like dynamic dispatching can greatly hurt the ability for the code to be greppable. It can also add indirection so be sure you use these methods sparingly and add more documentation where needed to help lost souls who are tracing code through the codebase.

The Search Advantage of gRPC

Here’s where things get interesting for APIs. Compared to RESTful APIs with their predefined resource structures, gRPC (and other RPC-based APIs) offer greater flexibility in naming methods. This freedom allows you to leverage greppable naming conventions to their full potential.

For instance, in a RESTful API, an endpoint for updating a user profile might be named /users/:id. While functional, this doesn’t explicitly convey the action performed. In a gRPC API, however, you could have a method named UpdateUserProfile(UpdateRequest request), which clearly describes the purpose and is much easier to search for in the code.

The Greppability Benefit

Meaningful names not only make code easier to understand but also significantly reduce maintenance time. When you (or another developer) revisit your codebase later, clear names can make traversing the codebase much nicer. I think the term “greppability” has a clear meaning and can come up in many contexts: in code, data, documentation, etc. Do you think this a good word for a developer to have in their vocabulary?

]]>Building APIs with Contractshttps://kmcd.dev/posts/api-contracts/Tue, 09 Apr 2024 00:00:00 +0000https://kmcd.dev/posts/api-contracts/ <p> <img hspace="5" src="https://kmcd.dev/posts/api-contracts/cover_hu_a5b854d92e996d16.webp" /> </p> Building for Scale: Why contract-based APIs are the future. In today’s interconnected world, APIs (Application Programming Interfaces) are the glue that connects computers. They allow different applications to talk to each other, share data, and perform actions. However, traditional methods of creating APIs can lead to challenges, especially when dealing with versioning changes and integrating complex systems. This is where contract-based APIs come in, offering a more robust and reliable approach and taming some of the wildness that exists on the web.

The Power of Pre-defined API Contracts

Imagine building a house without a blueprint. It would be chaotic and prone to errors. A contract-based API is like a detailed blueprint for communication between applications. It defines exactly what data can be exchanged, in what format, and what actions can be performed. This pre-defined agreement unlocks several advantages for developers and applications:

  • Improved Developer Experience:
    • Developers on both sides (client and server) have a clear understanding of what’s expected, making integration smoother.
    • Automated Documentation: Contracts serve as self-documenting artifacts, reducing the need for manual documentation creation and maintenance. This saves development time and ensures documentation stays in sync with the actual API implementation.
  • Reduced Errors:
    • Mismatched data formats or API changes become less likely, leading to fewer bugs and headaches. Contracts act as a validation layer, catching potential issues early in the development process.
  • Easier Integration:
    • Contracts act as a single source of truth, simplifying the process of connecting different services. Developers can quickly understand how to interact with the API without extensive back-and-forth communication.
  • Streamlined Development:
    • Contract-based APIs often enable tools to automatically generate code for both client and server implementations based on the defined contracts. This eliminates manual coding and boilerplate, allowing developers to focus on core functionalities.

By leveraging pre-defined contracts, you can build more robust, reliable, and maintainable APIs that streamline development efforts and improve overall communication within your application ecosystem.

Protobuf: The Language of APIs

In my world, the foundation of many contract-based APIs lies in Protocol Buffers (protobuf). It’s a language-neutral data format specifically designed for structured messages exchanged between applications. Protobuf offers several advantages:

  • Smaller Message Sizes: Protobuf messages are compact and efficient, leading to faster transmission and reduced bandwidth usage.
  • Faster Parsing: Parsing protobuf messages is significantly faster compared to traditional formats like JSON or XML.
  • Cross-language Compatibility: Protobuf definitions are language-agnostic. Code for interacting with the API can be generated for many programming languages.

Defining service contracts in protobuf involves creating .proto files. These files specify the structure of messages (data fields and types) and define the service methods (requests and responses) that an API offers.

Here’s a basic example of a .proto file defining messages for a user and an address:

syntax = "proto3";

message User {
  string name = 1;
  int32 id = 2;
  string email = 3;
  Address address = 4;
}

message Address {
  string street = 1;
  string city = 2;
  string state = 3;
  string zip = 4;
}

In this example, the User message has fields for name, ID, email, and an Address message. The Address message itself has fields for street, city, state, and zip code. These defined message structures ensure consistent data exchange between applications.

gRPC: Building APIs on a Solid Foundation

gRPC (gRPC Remote Procedure Call) is a high-performance framework that builds upon protobuf’s strengths. It provides a powerful way to implement remote procedure calls, allowing applications to interact using clients generated for each language using (usually) types and semantics that make sense to the language. Here’s how gRPC leverages protobuf:

  • Protobuf Messages for Communication: gRPC uses protobuf messages to define the request and response data exchanged between client and server.
  • Generated Code for Seamless Interaction: gRPC automatically generates server and client stubs (boilerplate code) from the .proto files. This eliminates manual coding and ensures consistency between client and server implementations.

Introducing Services and Request/Response Types with gRPC

Now let’s expand on the concept of services within a .proto file. We can define a service called UserService with methods for user management:

syntax = "proto3";

service UserService {
  rpc CreateUser(CreateUserRequest) returns (User) {}
  rpc GetUser(GetUserRequest) returns (User) {}
}

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

This example defines a UserService with two methods: CreateUser and GetUser. Each method takes a specific request message and returns a response message. The CreateUserRequest message contains a User object to be created, while the GetUserRequest specifies the user ID to retrieve. This clear separation of request and response types further enhances the contract between client and server. You should also notice just how clear the intention is for the methods of UserServer. A reader of this spec doesn’t have to do a mapping of extremely vague words like “POST” to more understandable actions like “create”. Also notice how the CreateUser and GetUser methods are greppable, making it trivial to locate uses of these RPCs, even across several repositories.

Alternatives

While Protobuf and gRPC are a powerful duo, there are other contract-based API solutions to consider:

  • OpenAPI (Swagger): This is a popular specification for defining RESTful APIs, offering a standardized way to document and interact with web services.
  • Thrift: Similar to protobuf, Thrift is a language-neutral protocol for defining service contracts. It supports various RPC protocols beyond gRPC.
  • Avro: This JSON-like data format uses schemas to ensure reliable data exchange. It’s often used with Apache Kafka for streaming data pipelines.

The choice between these options depends on your specific needs. gRPC and protobuf excel for high-performance, RPC-based communication, while OpenAPI is better suited for existing RESTful APIs. Thrift is also there. Avro has some dynamic typing features and has self-describing messages but is also slower for the same reason.

Conclusion

Contract-based APIs offer a significant advantage in building robust and scalable communication between applications. protobuf and gRPC provide a powerful combination for defining clear contracts and generating efficient code. By leveraging these technologies, you can streamline API development, improve developer experience, and ensure seamless integration within.

]]>Dropping Unknown Fields in ConnectRPChttps://kmcd.dev/posts/connectrpc-dropping-unknown-fields/Tue, 02 Apr 2024 00:00:00 +0000https://kmcd.dev/posts/connectrpc-dropping-unknown-fields/ <p> <img hspace="5" src="https://kmcd.dev/posts/connectrpc-dropping-unknown-fields/cover_hu_6b091745410a087d.webp" /> </p> Learn how to drop unknown fields in ConnectRPC to enhance the security of your gRPC services exposed to the internet. gRPC, with its focus on performance and language neutrality, remains a popular choice for building microservices and APIs. But when exposing your gRPC service to the internet, there are a few security considerations to account for. Protobuf, the serialization format often used with gRPC, offers various encoding options that can significantly impact your service’s security posture.

One crucial optimization for internet-facing gRPC services is customizing the behavior towards unknown fields. I’ve talked about unknown fields in a previous post, so read that one if unknown fields are still a mystery to you and then come back here. By default, protobuf messages can contain fields that are not defined in the current version of the proto schema. While convenient for development and can help with forward compatibility, this poses a security risk in a public environment.

Here’s why you should consider dropping unknown fields when exposing gRPC to the internet:

  • Preventing Malicious Data: Unknown fields can be exploited by malicious actors to inject unexpected data into your service. This could lead to potential security vulnerabilities like code injection or unexpected behavior.
  • Ensuring Compatibility: Uncontrolled unknown fields can cause compatibility issues if your clients are using different versions of the proto schema. Dropping them enforces stricter adherence to the defined message format.
  • Improving Performance: Skipping unknown fields during message parsing can lead to performance gains, especially when dealing with large datasets.

How to Drop Unknown Fields

Here is how you can drop unknown fields while using the standard proto.UnmarshalOptions struct provided by the google.golang.org/protobuf/proto package. Here’s how to do it in your Go code:

import (
	"google.golang.org/protobuf/proto"
	...
)

// Configure unmarshalling options to discard unknown fields
opts := proto.UnmarshalOptions{
	DiscardUnknown: true,
}

// Use the options when unmarshalling incoming messages
msg := &MyMessage{}
err := proto.Unmarshal(data, msg, opts)
if err != nil {
	// Handle error
}

By setting the DiscardUnknown field to true in the proto.UnmarshalOptions struct before unmarshalling incoming messages, you ensure that any unknown fields are ignored. This helps mitigate the security risks associated with unknown fields while processing internet-facing gRPC requests.

How to Drop Unknown Fields in Connect RPC Servers

package main

import (
	"log"
	"net/http"

	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"go.akshayshah.org/connectproto"
)

func main() {
	greeter := &GreetServer{}
	mux := http.NewServeMux()
	path, handler := greetv1connect.NewGreetServiceHandler(
		greeter,
		// Add an option that customizes protobuf marshalling/unmarshalling behavior
		connectproto.WithBinary(
			proto.MarshalOptions{},
			proto.UnmarshalOptions{DiscardUnknown: true},
		),
		// Add an option to customize JSON marshalling/unmachalling
		connectproto.WithJSON(
			protojson.MarshalOptions{},
			protojson.UnmarshalOptions{DiscardUnknown: true},
		)
	)
	mux.Handle(path, handler)
	log.Fatal(http.ListenAndServe(
		"localhost:9000",
		h2c.NewHandler(mux, &http2.Server{}),
	))
}

In this example, connectproto.WithBinary ensures only messages with defined fields are processed, enhancing the security of your gRPC service. connectproto.WithJSON does the same thing but with JSON.

Additional Considerations

While dropping unknown fields is a valuable security practice, it’s important to consider potential trade-offs:

  • Backward compatibility: Clients using older versions of the proto schema will encounter errors if they rely on previously defined unknown fields.
  • Logging and Debugging: Dropping unknown fields might make it harder to identify the source of unexpected behavior during development or debugging.

In such cases, it’s recommended to document these trade-offs and have a clear versioning policy for your gRPC service and client applications.

Conclusion

Exposing gRPC services to the internet requires careful security considerations. By customizing protobuf encoding options, specifically by dropping unknown fields using proto.UnmarshalOptions, you can significantly improve the security posture of your service. Remember to weigh the benefits against potential drawbacks and implement a solution that aligns with your specific needs.

]]>RESTless: Web APIs After RESThttps://kmcd.dev/posts/restless/Tue, 26 Mar 2024 00:00:00 +0000https://kmcd.dev/posts/restless/ <p> <img hspace="5" src="https://kmcd.dev/posts/restless/cover_hu_76b7adf3bf2226c0.webp" /> </p> Web APIs are the backbone of the modern web, but the ever-evolving landscape demands a rethink. This article explores alternatives to the traditional REST approach, diving into solutions like GraphQL, gRPC, and WebSockets. Unlock the full potential of your APIs and discover a world beyond REST! The “RESTful API” has been the workhorse of the web for many years. It has been an ever-changing religion with tenants that developers try their hardest to adhere to. But as web applications evolve, user demands grow and our industry experience with API design grows, it’s time to re-evaluate this approach. This article explores the limitations of REST and delves into modern alternatives that can unlock a world of possibilities beyond.

Objects? More like “objnoxious”

Imagine building a social media API endpoint to retrieve a user’s feed. Using a single REST object to represent a feed item can get messy. This object would need to encompass:

  • User information (username, profile picture)
  • Post content (text, images, videos)
  • User interactions (likes, comments, shares) with timestamps
  • Additional data like post visibility or author verification status

This “feed” object becomes bloated, especially if the feed contains many posts. Fetching and updating this complex object for every feed interaction can be inefficient. You can split the object up into many child objects but you are likely creating the need for clients to make more requests and greatly increasing the complexity of the API.

This highlights a limitation of REST: forcing real-world entities (like a social media feed) into rigid object structures with a strict hierarchy can lead to cumbersome data management.

Versioning is weird

Let’s say you introduce a new field to your product data model in a REST API. Versioning in the path (e.g., /api/v2/products) forces you to update every single endpoint URL that uses that data. Versioning each path by adding a query parameter (e.g., /api/products?version=2) or header seems more targeted, but what if only a specific endpoint needs the new version? Do you maintain a list of versions per endpoint? Do you bump the version for the entire API and default to the latest version? This is open to interpretation and every solution seems awkward to me.

Limited Options, Clever Solution

REST only offers a handful of methods (GET, POST, etc.) to handle data. There are 9 methods in total. Let’s talk about each one.

  • CONNECT
  • DELETE
  • GET
  • HEAD
  • OPTIONS
  • PATCH
  • POST
  • PUT
  • TRACE

Most of these are used in niche, hyper-specific ways. I suspect most developers don’t know what HEAD, TRACE, OPTIONS, and CONNECT do. None of these are incredibly useful when developing web APIs. So let’s ditch them. Let’s also ditch PATCH because it’s just PUT in a trenchcoat.

Now we have: GET, POST, DELETE, and PUT. Sweet, we’re left with enough methods to make a CRUD application. Roughly speaking:

  • Create = POST
  • Read = GET
  • Update = PUT
  • Delete = DELETE

Wow! Such simple. So elegant. I’m sure glad that everything web developers do boils down to these 4 simple actions… Oh wait, that’s totally wrong. There are so many more actions you can do to an object. Just think of how crazy it would be if you were using a programming language where you could only have four pre-defined methods in each class. Think of all of the extra container classes and weird abstractions we’d make on top of that. This sounds like utter insanity and is exactly what REST gives us.

Let’s stop pretending that there are only 9 (but actually 4) things you can do to a resource.

Inefficiency of JSON

JSON is slow and inefficient. It’d be insane if we built the internet around this format. Wait, we did? Really? JSON is wasteful in many ways. It’s text-based, which has an inherent cost in payload size and processing. JSON also will include key names over and over again and the length of the keys directly translates to longer payloads. This is not ideal if we’re trying to save on data transfer. Protobuf, on the other hand, is a compact and efficient binary format specifically designed for data serialization. Even factoring in gzip, JSON loses out to encoded protobuf in pretty much every way: CPU usage, memory usage, message size and speed. This just shows that there are better formats than JSON.

OpenAPI?

OpenAPI is a specification for describing RESTful APIs. It acts as a contract between API providers and consumers, defining the available resources, their properties, and the allowed operations (GET, POST, PUT, DELETE) for each. A common way OpenAPI is used is by generating OpenAPI specifications from the source of the backend service. Depending on the level of library integration these tools can automatically discover the HTTP method, route, request and response types, etc. This can help keep documentation up-to-date compared to manually creating OpenAPI spec, which is pretty incredible.

Okay, now here’s where I get philosophical. If we’re going to have a declarative specification for our APIs, I believe the specification should be the source of truth rather than the output. In my mind, OpenAPI specification should be the very first thing you write and agree upon and the servers and clients should be. However, many people don’t do this because the tooling isn’t amazing. Developers have grown fond of specific libraries and frameworks to develop our APIs so the target for generating language/framework/library code is vast and appears to be an incredibly hard problem. OpenAPI tries to solve so many problems at once. It’s coming in after we’ve designed our APIs and is trying to describe what’s already there. It’s an afterthought.

I’ve attempted to go down the route of using OpenAPI to generate server stubs and clients. It didn’t end well. There were too many issues generating clients and servers, even when the OpenAPI specification was valid. So I had to edit our OpenAPI spec to fit the limitations of the code generators just to get code generation to work. And that was just the beginning of my problems. I contend that this is a natural result of the design goals of the project. OpenAPI was not designed for generating code this way as a priority, so with many complex scenarios, it can be unclear how to map the spec to the semantics of the language/framework/library. OpenAPI wasn’t designed for that, it was only designed to describe the API, not the server or client that produces or consumes it. Now consider just how many “targets” for client and server stubs you want. Now consider that target libraries and the OpenAPI spec itself are evolving. This results in a compatibility matrix from hell.

So what other options do we have?

REST has served us well, but the modern web demands more. Let’s explore the exciting alternatives that offer a range of benefits and functionalities:

GraphQL

The “Choose Your Own Adventure” API. Still with the social media app example, imagine fetching only the user’s name and profile picture for the feed view, and then requesting their full profile and friend list separately when a user clicks on their profile. GraphQL allows you to specify exactly the data you need for each view, reducing unnecessary data transfer.

This method also has its downsides like making the backend API extremely complex.

gRPC

The “Cut the Drama” API. Consider a mobile game that communicates with a game server. gRPC allows you to define remote procedures (like attackEnemy or usePowerUp) that the client can call directly on the server. This removes the need for complex REST resource mapping and makes the communication intent clear. There are variants of gRPC like ConnectRPC that allow for leveraging of HTTP GET requests so you can fully leverage browser caching.

WebSockets

Need real-time updates like a live chat or stock ticker? WebSockets offer a persistent two-way communication channel, ideal for constantly flowing data between client and server. This is different from REST’s request-response cycle, allowing for a more dynamic connection.

Server-Sent Events (SSE)

SSE allows the server to push updates to the client without the client needing to constantly ask. Imagine live sports scores or social media notifications. SSE is simpler to implement than WebSockets, but is one-way (server to client).

Here’s a conclusion that summarizes the key points and offers a final thought:

The Verdict: REST vs. the Rest

REST APIs have served us faithfully for years, but as our applications become more complex and data-hungry, it’s worth considering the alternatives. GraphQL offers flexibility in data fetching, gRPC provides clear and efficient communication, WebSockets enable bidirectional real-time data flow and great browser support, and SSE simplifies server-to-client updates.

The choice ultimately depends on your specific needs. But remember, the API landscape is ever-evolving. HTTP/2 and HTTP/3 have opened up some new functionality that we have yet to fully tap into with our API designs. So, keep an open mind, explore the options, and don’t be afraid to break free from the comfy (but maybe slightly threadbare) jeans of REST when a more fitting alternative emerges.

]]>Introducing unknownconnect-gohttps://kmcd.dev/posts/unknownconnect-go/Tue, 19 Mar 2024 00:00:00 +0000https://kmcd.dev/posts/unknownconnect-go/ <p> <img hspace="5" src="https://kmcd.dev/posts/unknownconnect-go/cover_hu_4bccb0bea96d4f9c.webp" /> </p> unknownconnect-go is library that helps developers using gRPC identify compatibility issues caused by mismatched message definitions. gRPC systems can be quite complex. When making additions to protobuf files the server or the client often gets updated at different times. In a perfect world, this would all be synchronized. But we live in reality. Sometimes release schedules differ between components. Sometimes you just forget to update a component. Many times you might be consuming a gRPC service managed by another team and they don’t tell you that they’re changing things. I made something that will bring unique insight into this problem with very little work.

Let’s make things better

unknownconnect-go is an interceptor for ConnectRPC clients and servers that tells you if you are receiving protobuf messages with unknown fields. Now you can know when you should upgrade your gRPC clients or servers to the latest version. Let’s discuss how to use it.

  1. Install the library:
go get -u github.com/sudorandom/unknownconnect-go
  1. Import the library:
import (
    unknownconnect "github.com/sudorandom/unknownconnect-go"
)

Server-side usage

Here are two examples demonstrating how to use unknownconnect-go on the server side:

Short example:

unknownconnect.NewInterceptor(
    unknownconnect.WithCallback(func(ctx context.Context, spec connect.Spec, msg proto.Message) error {
        slog.Warn("received a protobuf message with unknown fields", slog.Any("spec", spec), slog.Any("msg", msg))
        return nil
    }),
)

This example creates a new interceptor using unknownconnect.NewInterceptor. The interceptor function receives an unknownconnect.WithCallback argument that provides a callback function. This function takes three arguments:

In the previous example, when a message with unknown fields is received, the interceptor will log a warning message using the slog.Warn function. It includes information about the message specification and the message itself.

Full example: Here is a full example that shows you how to register the unknownconnect.Interceptor with a ConnectRPC handler:

func main() {
    greeter := &GreetServer{}
    mux := http.NewServeMux()
    path, handler := greetv1connect.NewGreetServiceHandler(greeter, connect.WithInterceptors(
        unknownconnect.NewInterceptor(
            unknownconnect.WithCallback(func(ctx context.Context, spec connect.Spec, msg proto.Message) error {
                return connect.NewError(connect.InvalidArgument, errors.New("protobuf version missmatch; received unknown fields"))
            }),
        ),
    ))
    mux.Handle(path, handler)
    http.ListenAndServe("localhost:8080", h2c.NewHandler(mux, &http2.Server{}))
}

The interceptor function in this example returns an error with the connect.InvalidArgument code, which will cause the server to reject the request if it receives a message with unknown fields.

Customization options:

The two examples above show two ways to handle messages with unknown fields but you can customize the behavior of the interceptor to suit your specific needs. Here are some ideas:

  • Log the event: As shown in the first example, you can simply log a warning message when an unknown field is encountered. This can help debug and monitor the cause.
  • Add to a metric With this approach, you can emit metrics whenever unknown fields are encountered. This can be helpful to give more monitoring insight.
  • Fail the request/response: This approach, demonstrated in the second example, can be useful in pre-production environments to prevent unexpected behavior caused by mismatched message definitions.
  • Add an annotation to the context: This allows you to pass information about the unknown field to your service handler.

Client-side usage

And it works the same for clients, too:

package main

import (
    "context"
    "log/slog"
    "net/http"

    greetv1 "example/gen/greet/v1"
    "example/gen/greet/v1/greetv1connect"

    "connectrpc.com/connect"
)

func main() {
    client := greetv1connect.NewGreetServiceClient(
        http.DefaultClient,
        "http://localhost:8080",
        connect.WithInterceptors(
            unknownconnect.NewInterceptor(
                unknownconnect.WithCallback(func(ctx context.Context, spec connect.Spec, msg proto.Message) error {
                    slog.Warn("received a protobuf message with unknown fields", slog.Any("spec", spec), slog.Any("msg", msg))
                    return nil
                })
            ),
        ),
    )
    res, err := client.Greet(
        context.Background(),
        connect.NewRequest(&greetv1.GreetRequest{Name: "Jane"}),
    )
    if err != nil {
        slog.Error(err.Error())
        return
    }
    slog.Info(res.Msg.Greeting)
}

This example works in a similar way to how the server interceptor. It creates a new gRPC client for the Greet service and adds the unknownconnect interceptor using the connect.WithInterceptors function.

Conclusion

unknownconnect-go provides a simple and effective way to identify potential compatibility issues in your gRPC systems by detecting messages with unknown fields. It offers flexibility in how you handle these situations, allowing you to log warnings, reject requests, or implement custom logic as needed. By integrating unknownconnect-go into your development workflow, you can gain valuable insights into potential version mismatches and ensure smoother operation of your gRPC systems.

]]>Making gRPC more approachable with ConnectRPChttps://kmcd.dev/posts/connectrpc/Tue, 05 Mar 2024 00:00:00 +0000https://kmcd.dev/posts/connectrpc/ <p> <img hspace="5" src="https://kmcd.dev/posts/connectrpc/cover_hu_b94bf17e0642a707.webp" /> </p> Unleash the power of gRPC: ConnectRPC breaks down barriers, enabling frictionless communication between gRPC, gRPC-Web, and any HTTP client. Introduction

gRPC is an open-source framework for building high-performance applications that communicate with each other. gRPC is language-neutral, meaning clients and servers can be written in different programming languages, and it offers features like authentication, streaming, and load balancing. Also, because it is built using protocolbuffers it gives an amazing way to define contracts in a much more clear way than bolting on swagger to your API after the fact. However, gRPC has some limitations that restrict its usage. It requires special gRPC clients and support for HTTP/2 (which still lacking in some areas), and it doesn’t work from Javascript.

Support for Javascript has been solved in a couple of ways. Let’s discuss the two approaches!

gRPC-Web

gRPC-Web is a variant of gRPC and gRPC client for Javascript that avoids HTTP/2-specific features like HTTP trailers. This is an incredibly practical solution to the problem. And it does work really well, but I still have my reservations on some of the implementation details.

gRPC-Web doesn’t fix the “this doesn’t look like any HTTP API I’ve ever seen” issue that I have with gRPC in general. In other words, I want to be able to send a normal cURL example to someone. gRPC-Web doesn’t work for that without special gRPC-specific clients or tooling.

Additionally, I don’t like how gRPC-Web is typically deployed. You usually are forced into a proxy that can convert gRPC into gRPC-Web. Instead, I prefer the gRPC-Web implementation to sit alongside the actual gRPC server. The protocol isn’t so different from the normal gRPC version so it shouldn’t be too much work to add support to existing gRPC server implementations. I know that the popular gRPC library tonic supports gRPC-Web out of the box.

gRPC Transcoding

The idea with transcoding is to annotate your protobuf service methods with HTTP verbs and path patterns that can map a more REST-like API to gRPC. Many solutions allow you to provide a separate config file so you aren’t required to have the HTTP annotations making a mess of your protobuf files. Google has a service that can use this mapping and provide a REST-like API on top of your gRPC service. and several gRPC proxies can do this kind of transcoding as well like gRPC-Gateway and envoy.

Here’s a simple version of what the annotations can look like:

syntax = "proto3";

package status.v1;

import "google/protobuf/empty.proto";
import "google/api/annotations.proto";
import "google/rpc/status.proto";

service Status {
  rpc GetStatus(google.protobuf.Empty) returns (google.rpc.Status) {
    option(google.api.http) = {
        get: "/v1/status"
    };
  }
}

You can see how status.v1.Status.GetStatus maps to GET /v1/status. Thanks, protobuf options!

However, most ways of deploying this in Go weirdly use proxies, creating a new network hop and a decoding/encoding step. Additionally, transcoding ruins a lot of the benefits you have from a contract-based interface that gRPC provides. Generated clients generally can’t interpret the google.api.http options so if you want to keep to a contract-based model you have to rely on converting your protobuf file to OpenAPI and generating clients based on that. I don’t generally prefer this method because it adds extra complexity. However, it can be a really good way to support existing APIs by “swapping out” traditional HTTP handles with gRPC.

ConnectRPC

Let me introduce ConnectRPC. I believe it elegantly solves all of the issues I have with the gRPC ecosystem. ConnectRPC is a series of libraries for building browser and gRPC-compatible APIs. With ConnectRPC as the server, you get support for three protocols: gRPC, gRPC-Web and the so-called “Connect” protocol. These three protocols are all served from a single ConnectRPC server by simply using the HTTP content-type header, which gRPC and gRPC-Web clients already send. Let me break down where you might use each protocol:

  • Microservice communication: Connect or gRPC
  • Publically exposed API: Connect, gRPC or gRPC-Web depending on client language support
  • Clients running in environments without HTTP/2 support: Connect or gRPC-Web
  • Sending simple API examples to colleagues: Connect

Hopefully, I convinced you to at least give ConnectRPC a try. Now, get started with it! Here are the getting started docs for Go. It does a great job at detailing how you make a Go server and client implemented with ConnectRPC.

Multiple Protocol Support

So, ConnectRPC is running three different protocols? Isn’t that overkill? You can make everything work with the Connect protocol, but there are benefits to supporting all three.

We can use the gRPC protocol to call into this ConnectRPC service using normal gRPC tooling like grpcurl:

$ grpcurl -plaintext \
          -proto greet/v1/greet.proto \
          -d '{"name": "Jane"}' \
          127.0.0.1:8080 \
          greet.v1.GreetService.Greet
{
  "greeting": "Hello, Jane!"
}

If you have the reflection API enabled, you can omit the -proto option.

$ grpcurl -plaintext \
          -d '{"name": "Jane"}' \
          127.0.0.1:8080 \
          greet.v1.GreetService.Greet
{
  "greeting": "Hello, Jane!"
}

Now here’s where the magic is. In addition to gRPC-specific tooling, I can also use generic HTTP tools with ConnectRPC servers, like curl:

$ curl -XPOST \
       -H"Content-Type: application/json" \
       -d '{"name": "Jane"}' \
       "http://127.0.0.1:8080/greet.v1.GreetService/Greet"

This shows how the “I want to be able to send a normal cURL example to someone” desire from above is completely fulfilled.

I also want to point out that there is also the buf curl command which is a CLI tool that allows you to call ConnectRPC services using all three protocols (gRPC, gRPC-Web, Connect).

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet
{
  "greeting": "Hello, Jane!"
}

This last command actually uses the Connect protocol. We can pass in --protocol to use the gRPC-Web or the gRPC protocol instead:

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet \
           --protocol=grpcweb
{
  "greeting": "Hello, Jane!"
}
$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet \
           --protocol=grpc
{
  "greeting": "Hello, Jane!"
}

Digging Deeper (optional)

This is an optional section where we will dig deeper into what is happening here with this last command to show what is happening under the hood. We will see how server reflection works with gRPC and a couple uses of it. We can see more details of the previous buf curl command by adding -v at the end:

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet -v
buf: * Using server reflection to resolve "greet.v1.GreetService"
buf: * Dialing (tcp) 127.0.0.1:8080...
buf: * Connected to 127.0.0.1:8080
buf: > (#1) POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
buf: > (#1) Accept-Encoding: identity
buf: > (#1) Content-Type: application/grpc+proto
buf: > (#1) Grpc-Accept-Encoding: gzip
buf: > (#1) Grpc-Timeout: 119999m
buf: > (#1) Te: trailers
buf: > (#1) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#1)
buf: } (#1) [5 bytes data]
buf: } (#1) [23 bytes data]
buf: < (#1) HTTP/2.0 200 OK
buf: < (#1) Content-Type: application/grpc+proto
buf: < (#1) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#1) Grpc-Accept-Encoding: gzip
buf: < (#1) Grpc-Encoding: gzip
buf: < (#1)
buf: { (#1) [5 bytes data]
buf: { (#1) [244 bytes data]
buf: * Server reflection has resolved file "greet/v1/greet.proto"
buf: * Invoking RPC greet.v1.GreetService.Greet
buf: > (#2) POST /greet.v1.GreetService/Greet
buf: > (#2) Accept-Encoding: identity
buf: > (#2) Content-Type: application/grpc+proto
buf: > (#2) Grpc-Accept-Encoding: gzip
buf: > (#2) Grpc-Timeout: 119994m
buf: > (#2) Te: trailers
buf: > (#2) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#2)
buf: } (#2) [5 bytes data]
buf: } (#2) [6 bytes data]
buf: * (#2) Finished upload
buf: < (#2) HTTP/2.0 200 OK
buf: < (#2) Content-Type: application/grpc+proto
buf: < (#2) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#2) Greet-Version: v1
buf: < (#2) Grpc-Accept-Encoding: gzip
buf: < (#2) Grpc-Encoding: gzip
buf: < (#2)
buf: { (#2) [5 bytes data]
buf: { (#2) [38 bytes data]
buf: < (#2)
buf: < (#2) Grpc-Message:
buf: < (#2) Grpc-Status: 0
buf: * (#2) Call complete
{
  "greeting": "Hello, Jane!"
}
buf: < (#1)
buf: < (#1) Grpc-Message:
buf: < (#1) Grpc-Status: 0
buf: * (#1) Call complete

This is a little overwhelming at first so I will break down the two different calls that are being made here. Because we aren’t passing the protobuf file (or descriptors) as the --schema option we are missing some information needed to generate the protobuf messages from the given JSON string. So that’s the first call that buf curl will make, using the Server Reflection API:

buf: > (#1) POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
buf: > (#1) Accept-Encoding: identity
buf: > (#1) Connect-Accept-Encoding: gzip
buf: > (#1) Connect-Protocol-Version: 1
buf: > (#1) Connect-Timeout-Ms: 119999
buf: > (#1) Content-Type: application/connect+proto
buf: > (#1) User-Agent: connect-go/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#1)
buf: } (#1) [5 bytes data]
buf: } (#1) [23 bytes data]
buf: < (#1) HTTP/2.0 200 OK
buf: < (#1) Connect-Accept-Encoding: gzip
buf: < (#1) Connect-Content-Encoding: gzip
buf: < (#1) Content-Type: application/connect+proto
buf: < (#1) Date: Sat, 02 Mar 2024 06:33:57 GMT
buf: < (#1)
buf: { (#1) [5 bytes data]
buf: { (#1) [244 bytes data]
buf: * Server reflection has resolved file "greet/v1/greet.proto"

It acquired a version of the greet.proto file from the server. Now our client has everything it needs to make the actual request where it calls the actual service.

With HTTP, that service lives at POST /greet.v1.GreetService/Greet. Here’s what the call looks like:

buf: * Invoking RPC greet.v1.GreetService.Greet
buf: > (#2) POST /greet.v1.GreetService/Greet
buf: > (#2) Accept-Encoding: identity
buf: > (#2) Content-Type: application/grpc+proto
buf: > (#2) Grpc-Accept-Encoding: gzip
buf: > (#2) Grpc-Timeout: 119994m
buf: > (#2) Te: trailers
buf: > (#2) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#2)
buf: } (#2) [5 bytes data]
buf: } (#2) [6 bytes data]
buf: * (#2) Finished upload
buf: < (#2) HTTP/2.0 200 OK
buf: < (#2) Content-Type: application/grpc+proto
buf: < (#2) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#2) Greet-Version: v1
buf: < (#2) Grpc-Accept-Encoding: gzip
buf: < (#2) Grpc-Encoding: gzip
buf: < (#2)
buf: { (#2) [5 bytes data]
buf: { (#2) [38 bytes data]
buf: < (#2)
buf: < (#2) Grpc-Message:
buf: < (#2) Grpc-Status: 0
buf: * (#2) Call complete
{
  "greeting": "Hello, Jane!"
}
buf: < (#1)
buf: < (#1) Grpc-Message:
buf: < (#1) Grpc-Status: 0
buf: * (#1) Call complete

I wanted to show you how the server reflection works because it is a really good strength of gRPC. You can expose an API and have it be completely discoverable. Tools can automatically use this discovery mechanism… but so can humans. Look at these commands with grpcurl:

$ grpcurl -plaintext 127.0.0.1:8080 list
greet.v1.GreetService

$ grpcurl -plaintext 127.0.0.1:8080 describe
greet.v1.GreetService is a service:
service GreetService {
  rpc Greet ( .greet.v1.GreetRequest ) returns ( .greet.v1.GreetResponse );
}

$ grpcurl -plaintext 127.0.0.1:8080 describe .greet.v1.GreetRequest
greet.v1.GreetRequest is a message:
message GreetRequest {
  string name = 1;
}

We can see all the available services using the list and describe commands. And if you pass an object to the describe command you can dig down into message definitions. Protobuf works well here as the API contract for our services.

Conclusion

ConnectRPC offers a compelling solution for building gRPC servers that support multiple protocols. It seamlessly integrates gRPC, gRPC-Web, and the Connect protocol, allowing clients written in various languages and environments to interact with your service.

Here are the key takeaways:

  • ConnectRPC eliminates the limitations of traditional gRPC by supporting HTTP/1.1 and Javascript environments.
  • It provides a familiar REST-like API through the Connect protocol while still leveraging the benefits of Protobuf for contracts.
  • Multiple protocol support with a single server simplifies deployment and reduces complexity.
  • Existing gRPC tooling can still be used for server reflection and making gRPC requests.
  • The Connect protocol itself can be used with generic tools like curl, making API exploration a breeze.

If you’re looking for a flexible and future-proof way to build gRPC APIs, ConnectRPC is definitely worth considering. I highly recommend checking out the getting started guide for Go for a hands-on approach to using ConnectRPC.

]]>Inspecting Protobuf Messageshttps://kmcd.dev/posts/inspecting-protobuf-messages/Sun, 25 Feb 2024 00:00:00 +0000https://kmcd.dev/posts/inspecting-protobuf-messages/ <p> <img hspace="5" src="https://kmcd.dev/posts/inspecting-protobuf-messages/cover_hu_9163ecae201b3fce.webp" /> </p> Protocol Buffers is an amazing message format. It’s incredibly compact and performant. However, these advantages come at a cost. Since Protobuf is a binary format it lacks a lot in readability compared to text-based formats like JSON or XML. If you look at encoded protobuf data it just looks like meaningless ones and zeros.

However, all hope is not lost. Even if you just have a binary protobuf file with no knowledge of the corresponding protobuf file we can still get some information out of it. Let me introduce a tool called Protoscope. Protoscope is a tool for inspecting protobuf binary. It can do this with or without the protobuf files or the equivalent descriptor set (but it can do a better job with the protobuf data).

Install Protoscope

Okay, let’s hit the ground running. Here’s how to install protoscope (requires go).

go install github.com/protocolbuffers/protoscope/cmd/protoscope@latest

Using Protoscope

If you have a binary protobuf file, here’s what you can run to get protoscope output:

# protoscope [filename]
# variety.pb contains binary protobuf content.
$ protoscope -explicit-wire-types variety.pb
1:LEN {"Hello World!"}
2:VARINT 3
3:VARINT 175
4:LEN {
  `736563726574fa43bcab0ddfd7f3582699331e7ebbe267196804216a885fad5a3b0b01da25577220`
  `f583a5ac18b5c28516de341db3a7b44226e21ed85a6cdb571019fbee016574b8b99cd4ceab728ddd`
  `34a3e0b54605f7c7d1181ee3e13f4d9a07655f6ec843e74a997fd4b8ab87dc61754a60bd513d0121`
  `e4ad1fdc9e07a632`
}
5:LEN {`01020304`}
6:LEN {
  1:LEN {"Fluffy"}
  2:VARINT 1
}
7:VARINT 921
8:VARINT 1
9:I32 1.2345i32   # 0x3f9e0419i32
10:LEN {
  1:LEN {"key"}
  2:LEN {1:LEN {"Fred"}}
}
19:LEN {`ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01`}

# Often times you might see binary files encoded with hexadecimal.
$ cat variety.pb.hex
0a0c48656c6c6f20576f726c6421100318af01228001736563726574fa43bcab0ddfd7f3582699331e7ebbe267196804216a885fad5a3b0b01da25577220f583a5ac18b5c28516de341db3a7b44226e21ed85a6cdb571019fbee016574b8b99cd4ceab728ddd34a3e0b54605f7c7d1181ee3e13f4d9a07655f6ec843e74a997fd4b8ab87dc61754a60bd513d0121e4ad1fdc9e07a6322a0401020304320a0a06466c75666679100138990740014d19049e3f520d0a036b657912060a04467265649a0128ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01

# We can use xxd to convert it to binary then pipe it into protoscope
$ xxd -r -ps variety.pb.hex | protoscope -explicit-wire-types
... [same as above] ...

Note that with the examples above protoscope needs to guess types because we did not pass in the -descriptor-set and -message-type options. Why does protoscope need to guess types? One thing that will help with understanding this topic is knowing that protobuf encoding only has 6 types. And two of them aren’t even used in the latest version. You may be saying to yourself “I remember seeing a table of protobuf types and it had way more than 6!” and you would be correct. Although protobufs support many types they are all serialized into 6 “wire types.”:

ID Name Used for
0 VARINT int32, int64, uint32, uint64, sint32, sint64, bool, enum
1 I64 fixed64, sfixed64, double
2 LEN string, bytes, embedded messages, packed repeated fields
3 SGROUP group start (deprecated)
4 EGROUP group end (deprecated)
5 I32 fixed32, sfixed32, float

How protobuf encodes each protobuf type into a wire type differs depending on the type. The full explanation exists in the programming guide for the protobuf encoding. I recommend reading through it to fully understand the implications of using the protoscope tool.

Strings

Let’s see what it looks like with a trivial example that is sourced from connectrpc.eliza.v1.SayRequest (protoscope -explicit-wire-types eliza.SayRequest.pb):

1:LEN {"Hello World!"}

What does this tell us? Well, it tells us the message has a single field with field number 1. It also tells us the value of the string is "World". That’s pretty incredible compared to the nothing we knew about this collection of bytes a second ago! Let me quickly explain protobuf field numbers. The 1 in this example is a field number and it corresponds to the protobuf field number. For repeated values, you may see this number appear multiple times. But notice how the name is completely missing. That’s because protobuf doesn’t want to waste resources transmitting or storing metadata like that. It would be fair to summarize protobuf as a list of key/value pairs. The key is the field number and the values are one of the basic types in protobuf.

For the record, here’s what connectrpc.eliza.v1.SayRequest looks like:

message SayRequest {
	string sentence = 1;
}

Disclaimer: Protoscope has to guess types in a lot of instances because the protobuf encoding has only 6 wire types (two deprecated): VARINT, I64, LEN, and I32 are used with modern protobuf files. In the example above, field 1 could have been a string, byte array, an embedded message, or packed repeated fields. Protoscope guessed that it was a string and showed it to us as a string. It can guess wrong.

More Strings

Okay, now we’re going to look at a message derived from a different type: connectrpc.eliza.v1.IntroduceRequest (protoscope -explicit-wire-types eliza.IntroduceRequest.pb):

1:LEN {"Hello World!"}

Wait, what? It’s the exact same? Yep. If the field numbers and types match there’s no distinguishable difference when protobuf is encoded into binary. Here’s the protobuf type:

message IntroduceRequest {
	string name = 1;
}

Notice the message contains a single string field which is similar to `SayRequest`` above, but there are a few notable differences. The message name and the field name are different but since those two things are never transmitted over the wire with protobuf we can’t tell the difference between these two message types without prior knowledge. This flexibility allows you to make certain significant changes to your protobuf file without changing what is encoded… but you do have to follow some rules. These rules make more sense with more knowledge of the protobuf encoding.

Bytes

Okay, now let’s look at a new type: bytes. Let’s take a look at what that looks like with a byte array (protoscope -explicit-wire-types bytes.pb)):

4:LEN {
  `7365637265745b060fd327e7efb49cb89b479b65b71043859c2bafbd2d712fbcea2a759b230ceed4`
  `af7177eef821cd43935bdc74b682aa939ad99379b6a0e9c4e156b42691bf5e7cb7c8194eea230de4`
  `8981314872d7286920d6c5d2799546ce6131391ecd75edf27c17f413e257f50f9834454566c3439d`
  `7d2e52204aa57ba7`
}

What you’re seeing here is a hexadecimal representation of the bytes in our field. In this example, we mostly have random data. But if you pass this hexadecimal text through a hex-to-string converter you may notice that the beginning text, 736563726574, decodes to secret. Even though there is ASCII string content in the byte array, protoscope still treats this as a byte array, not a string. Here’s a byte array with a string that says Hello World! as the content (protoscope -explicit-wire-types bytes2.pb):

4:LEN {"Hello World!"}

Wait, what? Protoscope renders the text as text! What gives? Protoscope is, again, guessing the type of the data. It notices that all of the included bytes are in the ASCII range so it renders the content as text. Could this be the wrong thing to do? Maybe!

Numbers

Now let’s at numbers represented in protobuf. You may see some… odd things. We’ll break it down field by field (protoscope -explicit-wire-types numbers.pb).

2:VARINT 3
3:VARINT 175
5:LEN {`01020304`}
7:VARINT 921
8:VARINT 1
9:I32 1.2345i32   # 0x3f9e0419i32
19:LEN {`ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01`}

Refer to this table to see the actual protobuf types and the intended values. You will notice that several of them don’t match up with what protoscope outputs at all.

Field Number Actual Type Actual Value
2 enum 3 (AnEnum.C)
3 uint32 175
5 repeated uint64 1, 2, 3, 4
7 int64 921
8 bool true
9 float 1.2345
19 I32 -1, -2, -3, -4
  • 2: Enum fields are encoded as numbers on the wire. because of this, the name of the enum value may be unknown to you without the protobuf file.
  • 3: uint32 types look like what you’d expect! Nice.
  • 5: This is the first super weird one. Why is it shown as a string? This has to do with packed repeated fields. The protobuf encoding packs repeated primitive types into a single LEN field (instead of using VARINT, I64 or I32 as normal). Therefore, protoscope may simply represent this as a string or byte array because it can’t tell the difference on the wire.
  • 7: int64 types also look like what you’d expect! Nice.
  • 8: Booleans are encoded as false = 0 and true = 1.
  • 9: In this case, protobufs treated the float correctly and we get the correct value.
  • 19: This has to be the weirdest case. This looks so strange because it’s a result of using repeated int32 type, which is packed with negative values. Protoscope thinks this value looks like a LEN wire type with binary data in it. It guessed the type incorrectly this time.

Submessages and Maps

In protobuf you can put messages instead of other messages, so let’s look at what that looks like from protoscope’s perspective (protoscope -explicit-wire-types submessages.pb):

6:LEN {
  1:LEN {"Harey"}
  2:VARINT 1
}
14:LEN {
  1:LEN {"buf.build/connectrpc/eliza/connectrpc.eliza.v1.SayRequest"}
  2:LEN {1:LEN {"Hello World!"}}
}
17:LEN {}
18:LEN {1:LEN {"alpha"}}
18:LEN {1:LEN {"beta"}}

From the protoscope output above you might also notice that we have two field 18 values. That is because submessages cannot be packed as primitive types can. The “unpacked” way of representing a repeated value in the protobuf encoding is to simply write the field multiple times with different values. Simple. So field 18 is likely a repeated submessage field.

Now let’s look at maps: (protoscope -explicit-wire-types maps.pb)

10:LEN {
  1:LEN {"key"}
  2:LEN {
    1:LEN {"Corgi"}
    2:VARINT 1
  }
}
20:LEN {
  1:LEN {"Knock, knock"}
  2:LEN {"who's there?"}
}
20:LEN {
  1:LEN {"Java"}
  2:LEN {"Coffee, not code."}
}

Wait, what? This looks a lot like submessages! There’s a reason for that! Maps ARE submessages in the protobuf encoding. Here’s basically what the encoder is doing.

A map that looks like this:

message TestWithMap {
  map<string, int32> name_to_age = 7;
}

… is converted into a submessage that looks like this:

message TestWithMap {
  message name_to_age_Entry {
    optional string key = 1;
    optional int32 value = 2;
  }
  repeated name_to_age_Entry name_to_age = 7;
}

This shows that protobuf is a very practical encoding that re-uses basic concepts to support more complex structures.

Summary

In this blog post, we delved into the intricacies of inspecting binary Protobuf messages using the Protoscope tool. We highlighted its ability to decipher binary data even without the corresponding Protobuf files. We also covered the six wire types used in Protobuf encoding and explored various scenarios involving strings, bytes, numbers, submessages, and maps.

I didn’t cover all of the weird edge cases. There are features in the, now deprecated, proto2 format that I didn’t show. However, I hope that I’ve shown that you can get something from a binary protobuf file. This, alone, is quite impressive for a binary format. You would usually have a very hard time understanding anything without knowledge of the specific binary protocol. This demonstrates how protobufs takes some of the benefits you might get from text-based encodings (composability, support for “unknown” fields, some amount of discoverability) with the performance of binary formats (speed, reduced size) but protobuf does bring in an extra ingredient: contracts. Because protobuf files are the source of truth for the format and type-safe serialization code, gRPC client code, gRPC server code, and documentation can all be generated from protobuf files this shows the strength of the format… which is why you should try to never be in a situation where you NEED to use protoscope. You should always have a descriptor set or the protobuf files nearby to decode these messages.

For a more extensive overview of the protobuf binary encoding refer to the official documentation.

]]>Introducing protoc-gen-connect-openapihttps://kmcd.dev/posts/protoc-gen-connect-openapi/Tue, 20 Feb 2024 00:00:00 +0000https://kmcd.dev/posts/protoc-gen-connect-openapi/ <p> <img hspace="5" src="https://kmcd.dev/posts/protoc-gen-connect-openapi/cover_hu_666c893d7cc82563.webp" /> </p> ConnectRPC is a fantastic set of libraries that bridge gRPC into the web. gRPC is no longer relegated to the microservice box. Now it can spread its legs into the browser with gRPC-Web or the Connect protocol. The connect protocol, unlike gRPC-Web, allows for many standard web tools to work for non-streaming APIs. For unary RPCs Connect exposes an API that is simply JSON over HTTP like you’ve seen a million times before so tools like curl, postman, the Javascript Fetch API, etc. all work nicely with Connect. ConnectRPC also provides all three protocols (gRPC, gRPC-Web and Connect) using a single port on a single server. That means that you no longer need proxies to enable gRPC-Web and all standard gRPC tools are also at your disposal as well.

Here’s what an HTTP request looks like for a unary RPC with connect:

> POST /connectrpc.greet.v1.GreetService/Greet HTTP/1.1
> Host: demo.connectrpc.com
> Content-Type: application/json
>
> {"name": "Buf"}

< HTTP/1.1 200 OK
< Content-Type: application/json
<
< {"greeting": "Hello, Buf!"}

If you mark the endpoint as idempotency_level=NO_SIDE_EFFECTS then you can also call GET on the endpoint so the request body has to go into the query parameters. Here’s what that looks like:

> GET /connectrpc.greet.v1.GreetService/Greet?encoding=json&message=%7B%22name%22%3A%22Buf%22%7D HTTP/1.1
> Host: demo.connectrpc.com

< HTTP/1.1 200 OK
< Content-Type: application/json
<
< {"greeting": "Hello, Buf!"}

It’s that… simple? Elegant? No? If you’re not sold on ConnectRPC now then this may not be the post for you because the rest will talk about my new tool created for ConnectRPC; protoc-gen-connect-openapi.

Introducing protoc-gen-connect-openapi

protoc-gen-connect-openapi generates OpenAPI v3.1 files from protobuf files that match the API that the Connect protocol exposes.

We can document the Connect API as if it’s a real JSON/HTTP API… because it is, and the gRPC “flavor” isn’t so noticable due to Connect. With protoc-gen-connect-openapi you can declare your API using protobuf, serve it using gRPC/gRPC-WEb/Connect and fully document it without the API consumers ever knowing what protobuf is or how to read it. To me, this is the best of all worlds.

So, specifically, what good are OpenAPI files generated for Connect? What does that do? Well, first of all, it allows you to generate documentation with tools like redocly or elements.

Screenshot of protoc-gen-connect-openapi with redocly

Additionally, with tools like OpenAPI-Generator you can generate API clients for languages that Connect doesn’t support yet (for non-streaming methods).

flowchart LR

protobuf(Protobuf) -->|protoc-gen-connect-openapi| openapi(OpenAPI)
openapi -->|elements| docs(Gorgeous\nAPI Documentation)
openapi -->|redocly| docs
openapi -->|openapi-generator| other-languages(Languages that\nConnect doesn't\n support yet)
openapi -->|other tool| ???(Something equally amazing)
click elements "https://github.com/stoplightio/elements" _blank
click openapi-generator "https://github.com/OpenAPITools/openapi-generator" _blank

In summary, I hope this tool will extend the contract-based usage of protobuf along with ConnectRPC’s RPC-based API into further places with more documentation, more code generation, and additional validation.

For more information, check out the protoc-gen-connect-openapi repo on Github.

]]>gRPC From Scratch: Part 2 - Serverhttps://kmcd.dev/posts/grpc-from-scratch-part-2/Sat, 17 Feb 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-from-scratch-part-2/

Last part we created a simple gRPC client. Let's take it a bit further. Let's implement a simple gRPC server in go. ]]>Last time we made a super simple gRPC client. This time we’re going to make a gRPC server. We are going to completely reuse the writeMessage and readMessage from last time because they work the same on the server. After all, the envelope for servers is the same as the envelope for clients. Sweet!

The Setup

Like last time, we’re going to use ConnectRPC to help us test our implementation. Last time we used the ConnectRPC’s server to test our custom gRPC client so this time we’re going to use the ConnectRPC’s client to test our custom gRPC server. Did I say that right? Yeah, I think so… Let’s move on. Here’s what the full client looks like:

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"log"
	"net"
	"net/http"

	"connectrpc.com/connect"
	greetv1 "github.com/sudorandom/kmcd.dev/grpc-from-scratch-part-2/gen"
	"github.com/sudorandom/kmcd.dev/grpc-from-scratch-part-2/gen/greetv1connect"

	"golang.org/x/net/http2"
)

func main() {
	httpClient := &http.Client{
		Transport: &http2.Transport{
			AllowHTTP: true,
			DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
				return net.Dial(network, addr)
			},
		},
	}
	client := greetv1connect.NewGreetServiceClient(httpClient, "http://127.0.0.1:9000", connect.WithGRPC())
	req := &greetv1.GreetRequest{Name: "World"}
	fmt.Printf("send-> %v\n", req)
	resp, err := client.Greet(context.Background(), connect.NewRequest(req))
	if err != nil {
		log.Fatalf("err: %s", err)
	}

	fmt.Printf("recv<- %v\n", resp.Msg)
}

And as for more setup, here’s some of the more boring parts of the server:

func main() {
	mux := http.NewServeMux()
	mux.Handle("/greet.v1.GreetService/Greet", http.HandlerFunc(greetHandler))
	log.Fatal(http.ListenAndServe(
		"localhost:9000",
		h2c.NewHandler(mux, &http2.Server{}),
	))
}

Here we create an HTTP server with h2c so we aren’t required to use TLS for these examples, mount an HTTP path for our one endpoint and start the server. The real fun happens in greetHandler. But before I show that, I need to talk about HTTP trailers.

HTTP Trailers

Trailers are the same idea as headers but they happen after the response instead of before. Since gRPC is a streaming protocol it uses trailers to report on the overall status of the request instead of the HTTP status code. Go supports sending trailers. I am having a hard time coming up with a good explanation of how it works, so here’s an example:

import (
	"io"
	"net/http"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/sendstrailers", func(w http.ResponseWriter, req *http.Request) {
		w.Header().Set("Trailer", "MessagesSent")
		w.WriteHeader(http.StatusOK)
        io.WriteString(w, "Hello world!")
		w.Header().Set("MessagesSent", "100")
	})
}

In this example, a trailer named MessagesSent will be sent AFTER the Hello world! body has been sent to the client. Notice how we set a header called Trailer at the beginning with MessagesSent as the value? Yeah, that’s just how you do it in Go. It’s super strange. A more complete explanation exists in the Go documentation where there exists a separate equally magical way to add trailers that is less preferred for an unspecified reason:

	// There are two ways to set Trailers. The preferred way is to
	// predeclare in the headers which trailers you will later
	// send by setting the "Trailer" header to the names of the
	// trailer keys which will come later. In this case, those
	// keys of the Header map are treated as if they were
	// trailers. See the example. The second way, for trailer
	// keys not known to the [Handler] until after the first [ResponseWriter.Write],
	// is to prefix the [Header] map keys with the [TrailerPrefix]
	// constant value.

Why doesn’t Go support a more… normal interface for trailers? Who knows.. Trailers are often a bit of an afterthought. That is also proven by the fact that the “Trailer” values available in Go’s http.Request uses the type http.Headers. Oof.

Implementing the Greet Handler

As I did last time, I’m omitting error handling for clarity since readMessage and writeMessage can both return an error. The code that I end up with does handle errors that might happen when reading or writing the HTTP body. However, take a look at the cleaner version first:

func greetHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Trailer", gRPCStatusHeader+", "+gRPCMessageHeader)
	w.Header().Set("Content-Type", "application/grpc+proto")
	w.WriteHeader(http.StatusOK)
	defer r.Body.Close()

	req := &greetv1.GreetRequest{}
	readMessage(r.Body, req)
	writeMessage(w, &greetv1.GreetResponse{
		Greeting: fmt.Sprintf("Hello, %s!", req.Name),
	})
	w.Header().Set(gRPCStatusHeader, "0")
	w.Header().Set(gRPCMessageHeader, "")
}

Putting it all together

Surprisingly, this is pretty much all of the parts that are needed to handle a unary gRPC RPC. You can see the entire working prototype here.

Logs from the client

send-> name:"World"
recv<- greeting:"Hello, World!"

Logs from the server

recv<- name:"World"
send-> greeting:"Hello, World!"

Using buf curl

In addition to using the generating Go code we can also use tools like buf curl to test our server.

$ buf curl -v \
           --protocol=grpc \
           --schema=greet.proto \
           -d '{"name": "World"}' \
           --http2-prior-knowledge \
           http://127.0.0.1:9000/greet.v1.GreetService/Greet
buf: * Invoking RPC greet.v1.GreetService.Greet
buf: * Dialing (tcp) 127.0.0.1:9000...
buf: * Connected to 127.0.0.1:9000
buf: > (#1) POST /greet.v1.GreetService/Greet
buf: > (#1) Accept-Encoding: identity
buf: > (#1) Content-Type: application/grpc+proto
buf: > (#1) Grpc-Accept-Encoding: gzip
buf: > (#1) Grpc-Timeout: 119989m
buf: > (#1) Te: trailers
buf: > (#1) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#1)
buf: } (#1) [5 bytes data]
buf: } (#1) [7 bytes data]
buf: * (#1) Finished upload
buf: < (#1) HTTP/2.0 200 OK
buf: < (#1) Content-Length: 20
buf: < (#1) Content-Type: application/grpc+proto
buf: < (#1) Date: Sat, 17 Feb 2024 07:35:20 GMT
buf: < (#1)
buf: { (#1) [5 bytes data]
buf: { (#1) [15 bytes data]
buf: < (#1)
buf: < (#1) Grpc-Message:
buf: < (#1) Grpc-Status: 0
buf: * (#1) Call complete
{
  "greeting": "Hello, World!"
}

It works! The [5 bytes data] log lines that you see are the gRPC framing that happens before each message. The [7 bytes data] and [15 bytes data] are the request body and response body respectively. Also, notice the trailers at the end; Grpc-Status: 0 indicates that the RPC finished successfully.

How to improve

How can we improve the client and server that we’ve written? MANY details were glossed over when making this client/server. Here are just a small handful:

  • Different encodings (JSON)
  • Compression Support
  • Support for multiple gRPC status codes
  • Respecting the status codes at all in the client
  • Interceptor support
  • Compile code from protobufs
  • TLS support
  • Usage of the Grpc-Timeout header
  • Retries on error if the method is marked as idempotent
  • Fix a lot of implementation details
    • For example, our server does not percent-encode the error messages, which should be done according to the gRPC spec
  • gRPC-Web support
  • ConnectRPC support

There is quite a lot that goes into making a real gRPC client/server and a lot of small details that can have a large impact. Many of those details can be found in the gRPC over HTTP2 spec. Also, if I were making a real client/server I would use a test suite designed for testing a gRPC implementation’s conformity to the gRPC spec. ConnectRPC has one. Using that paired with testing with various gRPC tools and languages while trying to use as many features is probably the best way to validate your client and gRPC server implementations.

See the full prototype from this post here.

]]>gRPC From Scratch: Part 1 - Clienthttps://kmcd.dev/posts/grpc-from-scratch/Thu, 15 Feb 2024 00:00:00 +0000https://kmcd.dev/posts/grpc-from-scratch/

gRPC is an incredibly popular RPC framework that efficiently connects services. But how does it work? Let's dive in! ]]>

Disclaimer: This article is NOT for beginners unfamiliar with gRPC. If you’re looking to use gRPC like a sane individual, look elsewhere. Maybe start with the official gRPC documentation.

In the realm of distributed systems and microservices, gRPC has become the go-to communication protocol, boasting speed and efficiency over other web technologies like JSON-based HTTP APIs. If you’ve ever wondered what’s happening under the hood of gRPC, you’re in luck. Today, we’re embarking on an exciting journey into the intricate world of gRPC, delving deep into the byte-level details that make it tick. While I do this, I hope to convince you that gRPC is a much simpler protocol than you probably think.

Today I’m focusing on the basics of how the gRPC protocol works from a protocol level. But first, let me tell you what this article is not going to cover: protobufs. That is a separate topic that is covered extremely well in the official documentation. I may cover this topic later on in this series but for now, protobufs are being treated as a black box.

The first thing to know is that gRPC is built on top of HTTP. Let’s outline how protobuf service definitions map to HTTP/2 semantics with gRPC… First, let’s take this hello world example.

package helloworld;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

Take these simple observations:

  • The package is named helloworld
  • The service is named Greeter
  • The method is named SayHello

The corresponding HTTP request starts like this:

POST /helloworld.Greeter/SayHello HTTP/2
Content-Type: application/grpc+proto

That covers how to start a request. But what does the body look like? Take a look at this table. Sorry, yes: This is binary data. How do you expect gRPC to get its performance improvements if it didn’t use a binary encoding??:

Byte Offset Content Decoded Description
0 00000000 off Compressed-Flag
1:4 00000000 00000000 00000000 00000111 7 Message-Length (Unsigned 32-bit integer; Big Endian ordering)
5:12 00001010 00000101 01010111 01101111 01110010 01101100 01100100 1:“World” Message content

There are 5 bytes in total as a prefix to each encoded protobuf message. The first byte is for a flag saying if the message is compressed or not. Even though headers might say that compression is supported, servers can make their own decisions on whether or not to compress each message. Some messages are just too small to make compression worth it.

The last 4 bytes of the prefix are an unsigned 32-bit integer (using big-endian byte ordering) which tells the client/server how many bytes the next message will take.

gRPC always returns an HTTP 200 status code. That’s weird, right? gRPC does this because, for streaming RPCs, it’s impossible to know if a request succeeded ahead of time. Therefore, gRPC always returns a 200 as the status code and waits until the very end of the request to report the gRPC status using a lesser-known feature of HTTP called an HTTP trailer. Trailers are exactly like headers but come at the end of a request instead of the beginning.

Did you know? gRPC doesn’t actually require HTTP/2 support. Most HTTP/1.1 servers and proxies lack support for HTTP Trailers even though trailers were in the HTTP spec since 1.1. You can read more about the full story in this blog post.

Okay, let’s code something

For the second half of this article, we’re going to build a very un-featureful gRPC client in Go. It won’t support many features that are expected out of gRPC but it will be able to make RPC calls.

Making a service in protobuf

First, here’s the full protobuf file that I’m going to use for this example. It’s as close to the simplest Hello World in protobufs as you can get.

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

I used a buf.gen.yaml file along with the buf generate command to build this protobuf into Go types.

Making a simple gRPC server

We’re not writing the gRPC server from scratch in this example, just a client (but the principles are the same if you want to do this as an exercise on your own). Additionally, we need a real gRPC server to test our client against so I will use the server handler that ConnectRPC provides for us. Here’s what that looks like:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"connectrpc.com/connect"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"

	greetv1 "github.com/sudorandom/kmcd.dev/grpc-from-scratch/gen"
	"github.com/sudorandom/kmcd.dev/grpc-from-scratch/gen/greetv1connect"
)

type GreetServer struct{}

func (s *GreetServer) Greet(ctx context.Context, req *connect.Request[greetv1.GreetRequest]) (*connect.Response[greetv1.GreetResponse], error) {
	return connect.NewResponse(&greetv1.GreetResponse{
		Greeting: fmt.Sprintf("Hello, %s!", req.Msg.Name),
	}), nil
}

func main() {
	greeter := &GreetServer{}
	mux := http.NewServeMux()
	path, handler := greetv1connect.NewGreetServiceHandler(greeter)
	mux.Handle(path, handler)
	log.Fatal(http.ListenAndServe(
		"localhost:9000",
		h2c.NewHandler(mux, &http2.Server{}),
	))
}

Encoding the Request

Now that the setup is out of the way, let’s build a client that can send a gRPC request and receive a response from the HTTP server. Note that this will only work for a unary RPC call (a call that does not support streaming). First, let’s start with encoding/decoding messages using the format I mentioned above.

Here’s the code to write a request to a gRPC server.

func writeMessage(w io.Writer, msg []byte) {
	prefix := make([]byte, 5)
	binary.BigEndian.PutUint32(prefix[1:], uint32(len(msg)))
	w.Write(prefix)
    w.Write(msg)
}

Decoding the Response

The response is returned using the exact same format as our request. With go, this is what it might look like (without any error handling at all)

func readMessage(body io.Reader) []byte {
	// read the prefix/envelope
	prefixes := [5]byte{}
	io.ReadFull(body, prefixes[:])

	// Using the message size from the prefix, read that many bytes. That's our protobuf message.
	buffer := &bytes.Buffer{}
	msgSize := int64(binary.BigEndian.Uint32(prefixes[1:5]))
	io.CopyN(buffer, body, msgSize)
	return buffer.Bytes()
}

This code reads the prefixes (the compression flag and the message size) and then the message size is used when reading the message from the server. That’s… essentially it.

The rest

We have the foundation of the gRPC protocol completed. Now, the missing part is code that creates the actual HTTP request, encoding/decoding the actual protobuf types (which is a simple call to proto.Marshal and proto.Unmarshal) and doing some error handling that I didn’t do above. To me, none of that is particularly interesting but you can explore the entire working prototype here on your own time.

Here’s what the output looks like:

send-> name:"World"
recv<- greeting:"Hello, World!"

What about gRPC streams?

Streaming requests simply repeat this envelope encoding. gRPC, for better or worse, made the unary (simple request and response) use case a bit harder for the benefit of making the more complex use cases (streams) simple. Plus, you only have to write one encoder/decoder. For a server streaming RPC, you would simply repeat the readMessage() function call until you get an EOF or some other error. This showcases gRPC’s simplicity in handling streaming.

Note that you will have to periodically call Flush() on the handler’s http.ResponseWriter argument to get the bytes flushed to the TCP socket. This normally happens for you automatically after the http.Handler is complete… but with streaming calls we have to do this ourselves! Here’s an example of how to do it:

if f, ok := w.(http.Flusher); ok {
	f.Flush()
}

By the way, ConnectRPC has chosen to “undo” this approach by offering unary RPC without the 5-byte custom envelope. This makes using tools like cURL possible and even pleasant, especially when using the JSON encoding. But it is also friendly with gRPC clients by offering gRPC and gRPC-Web alongside the Connect protocol.

Okay, what was the point?

Hopefully, I was able to shed a little bit of light on how gRPC really works. Binary protocols often have hard-to-understand documentation about each byte in a packet. However, gRPC only has 5 bytes of this weirdness so it’s a perfect protocol to whet your appetite on network protocols.

If you want to look at more details about the gRPC specification, I would refer you to the official gRPC specification on GitHub. See the full prototype from this post here.

]]>Why you should use gNMI over SNMP in 2024https://kmcd.dev/posts/gnmi/Sat, 04 Nov 2023 00:00:00 +0000https://kmcd.dev/posts/gnmi/ <p> <img hspace="5" src="https://kmcd.dev/posts/gnmi/cover_hu_3ad0859e01f5c296.webp" /> </p> gNMI is better than SNMP and more people need to know about it. Network engineers have some unique challenges in monitoring and managing their own network devices. So it may come as a surprise to some people that, by far, the most used protocol for monitoring network devices was created over 30 years ago. However, there is an industry-accepted replacement and it’s time to upgrade.

SNMP has been the go-to protocol for network management for decades, but it has some limitations. It is complex, inefficient, and doesn’t scale well to modern networks.

gNMI is a newer protocol that is designed to address these limitations. It is simpler, more efficient, and more scalable than SNMP. It is also more flexible, giving network administrators more control over what data they collect and how they collect it.

gNMI has the following operations:

  • Get is used to retrieve data from network devices.
  • Set is used to modify data on network devices.
  • Subscribe is used to receive updates from network devices when the data changes.

Why is gNMI better than SNMP?

Here are a few specific reasons:

  • gNMI is model-driven. This means that it uses YANG to define the data that can be collected from network devices. This makes it easier to write and maintain scripts and applications that collect and manage network data.
  • gNMI is bidirectional. This means that it can be used to both collect data from network devices and send data to them. This makes it possible to use gNMI for a wider range of network management tasks, such as configuring devices and troubleshooting problems.
  • gNMI is efficient. gNMI uses a streaming protocol to collect data from network devices. This means that it can collect data more efficiently than SNMP, which uses a polling-based approach.
  • gNMI is scalable. gNMI is designed to scale to large networks with many devices. It can handle a high volume of traffic without sacrificing performance.

Subscriptions

Let me elaborate on the “streaming protocol” comment. gNMI allows for a sustained streaming connection where the server can send updates to the client. This is possible because gNMI uses gRPC. SNMP has no such streaming ability[0] and most SNMP communication happens with a simple request/response pattern. Let’s look at what a typical SNMP-based monitoring setup looks like:

---
title: Polling for interface statistics with SNMP
---
sequenceDiagram
    participant client AS SNMP Client
    participant device AS SNMP Device

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000000

    client->>device: Get Interface Statistics
    device->>client: Interface Statistics: Ethernet1/inOctets = 1000420

Notice that the “Get Interface Statistics” request is repeated over and over again. The SNMP client has to keep asking over and over again for updates and will get back the same data it just got a minute ago. Also, I’m masking some serious ugliness of SNMP for your sake. With SNMP, you have to look at a table to map an index number to the interface name… And if an interface is “too fast” then you may need to poll the ifHCInOctets value instead. Neither of these issues exists in gNMI.

Now let’s see how gNMI handles this situation using so-called subscriptions:

---
title: Subscribing to interface statistics with gNMI
---
sequenceDiagram
    participant client AS gNMI Client
    participant device AS gNMI Device

    client->>device: Subscribe To Interface Statistics
    device->>client: Subscription established

    device->>client: /interfaces/interface[name=Ethernet1]/state/counters/in-octets = 1000000

    device->>client: /interfaces/interface[name=Ethernet1]/state/counters/in-octets = 1000420

In this example, we set up the subscription once and from then on we get updates to these counters. Note that we ONLY get updates. This is a strength that’s possible in gNMI. We can choose to not get updates if nothing has changed with a piece of data, so if an interface counter has not changed then we just won’t get an update for that interface. This shows how gNMI subscriptions reduce the need for a large number of polling requests which has the benefit of reducing the load on network devices. Said differently, instead of polling devices for updates at regular intervals, gNMI subscriptions can allow devices to send updates to clients only when the data changes.

More

In addition to the above, here are some other benefits of using gNMI:

  • gNMI is more secure than SNMP. Since it is based on HTTP/2, it uses TLS to encrypt all traffic, which helps to protect your network from unauthorized access.
  • gNMI is supported by a growing number of network devices and management tools. This makes it easier to find the solutions that you need to meet your specific needs. Tools like gNMIc make it much easier to peer into the state of devices while offering a much better user experience than tools like snmpget or snmpwalk.
  • gNMI is an open standard. This means that it is not controlled by any one vendor, which gives you more flexibility and choice. Even when the data models that are used aren’t OpenConfig, YANG is still almost always used to describe the vendor-specific data model and makes documentation and automation infinitely easier.

Overall, gNMI is a more modern and efficient approach to network management than SNMP. It is a good choice for organizations of all sizes, from small businesses to large enterprises. I have a hypothesis that gNMI could also be good for smaller-scale (maybe homelab?) setups as well, but more on that in a future post. In the future, I also want to dive into some details of gNMI with: different types of subscriptions: (STREAM, POLL, ONCE), updates_only/sync_response, coalescing of duplicated updates, dial out with grpctunnel, and core data types. However, I’m low on time and each of these topics deserves separate posts. Until then, thanks for reading!

Footnotes

  • [0] Yes, I know SNMP traps exist but they’re not applicable in this and many other examples and it’s also a pain to work with, may silently break and is overall awful to work with.

References

  • gNMI on github - has a reference client/server in Go.
  • gNMI specification - the specification documentation for gNMI
  • SNMP on Wikipedia - SNMP has many different RFCs (because it’s old) so Wikipedia is a source for the full list.
  • YANG specification - Networking is such a complex problem they made an entirely new language to describe the data.
  • OpenConfig YANG models - OpenConfig is the “de facto” models for network devices and is a good example of using YANG itself.
  • gNMIc - Commandline tool that supports many gNMI features and supports data transformation, path suggestions based on YANG, dial-out telemetry, multiple gNMI targets, and a lot more.
]]>The Rollercoaster of Productivity in Side Projectshttps://kmcd.dev/posts/productivity/Sun, 10 Sep 2023 00:00:00 +0000https://kmcd.dev/posts/productivity/ Frank talk about the rollercoaster of productivity. Okay, let’s talk.

I had a super productive period on this blog a few weeks ago. I thought nothing could stop me from updating this blog. I had lots of ideas, lots of motivation, and a lot of energy to write these posts. Even so, I’m blocked. I feel like life has gotten in the way of doing what I like to do with this blog. I’ve been sick, I’ve had work travel, I’ve had a minor surgery. I can come up with a lot of reasons for why I’ve dropped off with updating this blog but I can’t answer why I haven’t just started up again. What I’ve learned over time is that my drive to do side projects follows a rollercoaster pattern. For the rest of this post, I’m going to overview some ideas that I have to cope with this. This is mostly reminders for me, but hopefully, you can get some use out of this as well.

Queue Updates

I have come to realize that I am not consistent with content generation. I’m just not. I will go through phases of hyper-productivity and phases of incredible procrastination. The way I can cope with that is to schedule completed posts in the future. If I have a buffer of updates, it gives me time to roll through the unproductive times without feeling incredibly awful about posting a new update. By the way, all of this pressure is completely self-imposed. I’m not sure anyone cares how often I update this thing. I just feel like I should.

Finish Things

Speaking of “completed posts”, I have several open drafts that are just sitting there, unfinished. As drafts stagnate, I have realized that it’s incredibly hard to get myself back into the mindset to pick up a draft and finish it. To deal with this issue, I will try to complete an idea before moving on to a new one. That means that I should either scrap the idea or complete the post. I sometimes hit roadblocks that are entirely dependent on me doing something that I don’t want to do. That is a death sentence for a post… and I need to be more honest with myself in identifying these cases and just dropping the idea. This is a side project. The whole point is that I like doing it… and if it’s pushing me in directions I don’t want to go then I should just drop that idea. This isn’t my job.

To the future

I hope to get better at my side-project productivity in the future. Hopefully, some of that was useful to read. It’s certainly useful for me to type out. Happy coding and may your side projects thrive, no matter how variable the productivity may be!

]]>Lessons from a Decades-Long Projecthttps://kmcd.dev/posts/lessons-from-a-decades-long-project/Fri, 18 Aug 2023 00:00:00 +0000https://kmcd.dev/posts/lessons-from-a-decades-long-project/

Inside Evepraisal's operations: data automation, databases, deployment, and advice for similar projects. ]]>I wrote Evepraisal. Evepraisal is a tool/website that began as a way to more efficiently blow up transport ships in Eve Online and evolved into a trusted pricing estimate authority for the entire (Eve Online) Universe. This article will cover the technical lessons and experiences that I had maintaining this extremely useful tool for a decade.

Table of Contents (click to expand)

Getting Started

I have already told the story of how it started in my Economists with Guns article, but I did want to add that the first version of what became Evepraisal was written in Python and used a static database of pricing data. It also only worked for cargo scan results because it was used for the Burn Jita event in 2012 where players in cheap ships would attack transport ships carrying expensive goods in high-security space knowing that the police would quickly show up and destroy the attackers. I made this tool to help identify which ships were holding expensive goods. The first version had all of the major features: a box you can paste some text into, a listing of all the items and a total price estimate on the top of the page. After using it for a little bit I realized that people didn’t believe my estimates when I just posted them in the scouting chat… So I worked on a way to share the results in the chat with a link, which, I believe, is most of the reason for the success of the website.

After that, life went on. I kind of forgot about the tool. I only intended it to be used for this event so the prices of items never updated. They were all frozen in time. It was a few months later when I noticed that the tool was still being used a good amount and I started getting requests for updated prices and support for other formats. I bought the evepraisal.com domain and started my work.

v1: Getting real with Python

The next version was written in Python and used an API to fetch market data from eve-marketdata.com, which is now sadly no longer running. Also, the mapping of the item name to the so-called “type ID” was done by harvesting the list of all types and type IDs from data files that come along with installing the Eve Online client. These files were SQLite databases that could be easily opened and queried. Thankfully, someone else has already done the reverse engineering work and published a Python library to help with querying this data.

What the appraisal page looked like in 2013

I used PostgreSQL for the database and memcached to cache both appraisal pages and requests to eve-marketdata.com. To get an idea of what the data looks like, here are the tables that this version ended up with.

Appraisals

    Id INTEGER
    Kind TEXT
    RawInput TEXT
    Parsed JSON
    ParsedVersion INT
    Prices JSON
    BadLines JSON
    Market INTEGER
    Created INTEGER
    Public BOOL
    UserId INTEGER

If you’re wondering where the “type” information is, well… it’s in a giant JSON file that is loaded into memory on startup. If you want to see the last version of this file, it exists here.

Evepraisal homepage in 2013

This first version was great but it wasn’t perfect. A lot of memory and disk space was being used for the database. The database needed a good amount of maintenance. After a few years, a lot of these things made the website run fairly slow and I had too many outages for my liking, so I needed to do something to help the situation…

v2: Just Rewrite it in Go

To address some of the issues with v1 I decided to rewrite the project in Go. I wanted to experience writing something “real” in the language so this was it. This effort involved re-writing all of the parsers, all the API handlers, the frontend, and… everything into Go which took a good deal of effort. But in the end it was worth it. The memory usage was WAY down for several reasons, the CPU usage (which was starting to be a problem too) was very minimal and the website was extremely fast. Rewriting the website in a different language wasn’t the only thing that changed though. I changed the database, caching method, how it sourced data, and probably a lot more. All while maintaining a familiar user experience.

PostgreSQL to Bolt

With this rewrite, I also switched from PostgreSQL to an embedded key/value store called Bolt. This means that the only way I have to access data is by the exact primary key or by scanning a range of keys in alphanumeric order. This made some things more difficult but it also taught me a lot about this type of database and it’s fast. Extremely fast. I would post the schema here, but that’s not a thing for Bolt. So instead, I will show you the buckets and the format I used for keys and values for each bucket.

bucket=appraisals
key_format="AppraisalID"
value=JSON Encoded Appraisal Data

bucket=appraisals-last-used
key_format="AppraisalID"
value=timestamp encoded as a uint64

bucket=appraisals-by-user
key_format="CharacterOwnerHash:AppraisalID"
value=AppraisalID

bucket=stats
key=total_appraisals
value=uint64

Creating an appraisal would involve adding an entry to appraisals, appraisals-last-used, appraisals-by-user (if the user was logged in) and stats buckets. Deleting is pretty much the same. It took a while to get these features worked out but once it was done I haven’t needed to adjust this very much at all.

Eve-Marketdata to ESI

I moved from eve-marketdata.com to ESI, which newly supported fetching live(ish) market orders and allowed Evepraisal to have more reliable pricing. ESI also guesses the market price which was a useful value to fall back on but sometimes it was ridiculously out of whack with the market.

Other Changes

Around this time I also fixed some usability issues. I tackled pricing for items that had a super low volume or couldn’t be sold on the market by summing up the cost of the components that were needed to build that item. I added a percentage upcharge to that to factor in for build time and effort. In the end, it made for a decent price after some tweaking.

In addition to re-implementing the existing parsers in Go, I also added support for a few more formats and improved on the “heuristic” parser that would try VERY HARD to find Eve Online types from the input.

The data migration was fairly extensive. I had a process that would re-parse every appraisal again to ensure that it could. I noted differences and eventually decided that it was good enough (or better enough) to do a final migration and flip over to the new site. Because this process was pretty slow I also optimized the parsers a bit at this time so this would go faster. I tested enough to ensure that when the day came that I swapped the website over to the new version most users didn’t notice. I made some UI adjustments, which people did notice, but the fact that the entire thing was rewritten was unseen to most users.

Data Flow

So let’s put all of this together. Here’s the basic dataflow diagram for Evepraisal:

flowchart TD
    esi["ESI\nEvery 6 minutes"]
    pricedb[Prices DB]
    static-data-export["Static Data Export\nEvery 6 hours"]
    typedb[Types DB]

    parser[Parser]    
    pasted-text["Posted Appraisal Text\nFrom a user"]
    appraisal[Appraisal Logic]
    appraisaldb[Appraisal DB]

    appraisal-page[Render and return appraisal page]

    esi --> pricedb
    static-data-export --> typedb

    typedb --> parser
    pasted-text --> parser 
    parser --> appraisal
    pricedb --> appraisal
    appraisal --> appraisaldb
    appraisaldb --> appraisal-page

And here’s what the new version looked like:

Evepraisal homepage in 2023

Appraisal page in 2023

All-in-all, I think this rewrite went well. It probably took more effort than I expected, but it paid off over the next 6 years from 2017 to a couple of weeks ago. My primary goal was to get a functioning knowledge of operating a project using the Go programming language, and that happened.

Project Recap

So now that I described the evolution of the project and the two major versions, let’s recap some major issues and non-issues that I had while maintaining the project.

What went wrong

I encountered hurdles such as unreliable game data, naming inconsistencies, missing data, and disk constraints. Overcoming these challenges shaped how I approach these problems in my other work.

Unreliable game data

When CCP standardized the names of modules, they never bothered to update the type database with the new names. Also, volumes and packaged volumes are not consistent and not in the game data, so I had to maintain my mapping, which sucked.

Eve-marketdata had a good number of outages and eventually just stopped working for a long period. So to get the website back up I quickly made a few integrations with other services (who I think had less reliable data).

CPU/Memory/Disk Constraints

I already mentioned CPU/memory constraints in the motivations to re-write the project using Go. However, v2 still had issues with disk space. Because appraisals were never deleted I was slowly running low on disk space. I ended up fixing this in two ways:

  • changing the default for the API to not store appraisals
  • purging data more aggressively by using the “last seen” time to decide which appraisals are unlikely to be looked at again

What didn’t go wrong

Certain aspects, like deployment, OS upgrades, and backup/restore, flowed smoothly. Rigorous testing and proactive data management bolstered overall stability.

Deployment

You may be surprised to see my deployment script. You can read it in the deploy.sh file. It simply copies up a systemd unit file and the Go binary. Then it ensures that the service is enabled. That’s basically it. It’s incredible that the deployment only involves updating a single binary file.

OS Upgrades and Migrations

Throughout the past decade, there were instances of image upgrades and migrations. Thankfully, these processes transpired without major issues, resulting in only a few minutes of downtime once or twice a year. In the cloud environment, periodic migrations were required to address security vulnerabilities. Notably, cloud providers have improved their strategies, making migrations to new VM host more seamless. An example of a migration that wasn’t seamless occurred when the VM hosting Evepraisal was moved to a different data center, requiring some downtime of approximately an hour or two.

Backup/restore

I never had to restore from backup… and, to be honest, it’s not the worst thing in the world that the database gets wiped. Sure, it would be annoying to users that have links to their appraisals but a lot of them can be recreated pretty easily.

Testing

There’s a lot of table-based testing in this project, which allowed me to add new test cases without needing to write a lot of code. This is especially useful in the parser part of the app. Here’s an example of a couple of test cases for the “listing” parser, which handles lines that simply have a quantity and an item name delimited by some kind of whitespace:

{
    "quantities with a decimal, for some reason",
    `123.12	Griffin
456.3	Maulus`,
    &Listing{
        Items: []ListingItem{
            {Name: "Griffin", Quantity: 123},
            {Name: "Maulus", Quantity: 456},
        },
        lines: []int{0, 1},
    },
    Input{},
    false,
}, {
    "with ending whitespace",
    `Compressed Iridescent Gneiss x 109 `,
    &Listing{
        Items: []ListingItem{
            {Name: "Compressed Iridescent Gneiss", Quantity: 109},
        },
        lines: []int{0},
    },
    Input{},
    false,
}, {
    "with beginning whitespace",
    `1865 Compressed Glossy Scordite
 105 Compressed Brilliant Gneiss
  27 Compressed Jet Ochre`,
    &Listing{
        Items: []ListingItem{
            {Name: "Compressed Brilliant Gneiss", Quantity: 105},
            {Name: "Compressed Glossy Scordite", Quantity: 1865},
            {Name: "Compressed Jet Ochre", Quantity: 27},
        },
        lines: []int{0, 1, 2},
    },
    Input{},
    false,
}

All of the test cases for all parsers were run by the same code, so adding more tests is just adding a new Case object. Whenever I fixed a reported bug I would create a new test case so that the same bug couldn’t happen again. This has prevented an incredible number of potential regressions over the years.

Stats for nerds

Metric Refs
Apprisals Performed 150 Million
First Commit Dec 17, 2012 3a24930
Last Commit (Python Version) Nov 27, 2017 bb949e2
First Commit (Go Version) Apr 25, 2017 55f6794
Lines of code (Python) 1283+1051=2334 python-evepraisal + eve-paste
Lines of code (Go) 8022 go-evepraisal
Number of Commits 450+234+130=814 go-evepraisal + python-evepraisal + eve-paste
Total Number of Users 3,162,712 Google Analytics (private)
Total Number of Browser Sessions 25,251,120 Google Analytics (private)
Peak Active Users (single day) 11,485 Google Analytics (private)
Top Countries United States, Russia, United Kingdom, Germany, Canada Google Analytics (private)
Gender Breakdown 93.48% Male, 6.52% Female Google Analytics (private)
Browser Breakdown 69.31% Chrome, 13.98% Firefox, 6.28% Edge Google Analytics (private)

Note that this doesn’t cover API access to Evepraisal, so a lot of those usage metrics aren’t captured here.

Commit history for the Python version of Evepraisal
Commit history for the Go version of Evepraisal

Advice

Now here’s some advice for running a similar project. I feel like most of this applies to many kinds of side-projects that you intend to have people use.

Identify data dependencies and find good sources

Evepraisal suffered a lot whenever the source of data wasn’t reliable. Eve-marketdata served as the data source for market data for Evepraisal for a long time. I ended up integrating with a similar service as a backup before settling on using CCP’s ESI market API to fetch market data from the original source.

It’s a similar story with the type database, first, it was gathered using dubious methods that could break often until it was replaced with a more official and reliable data source.

Automate the rotation of data dependencies

I initially had a script that would scrape Eve Online’s type data from the game client data files. This was super manual and it involved me updating the game (which took a lot of my disk storage for a game I didn’t play much), running a script, and checking in the new giant JSON file with all of the types in it. The process was “fine” but it was very reactionary. I had to do this for every major game release and that’s pretty often. I would normally get messaged about the game data needing an update from a user noticing a new item didn’t get recognized. That’s kind of embarrassing, so I needed to improve this.

I was so happy when CCP started regularly releasing the Static Data Export (SDE) for Eve Online. It had (almost) everything I needed. After this change, Evepraisal would periodically check for a new SDE release and, if there was one, would download the new package, process it, make a new type database and switch to using the new one. It was pretty slick once I got this process working reliably.

You may notice a pattern here: moving from scrappy 3rd party APIs that are maintained by individuals to using first-party data sources like SDE and the ESI API. I think CCP did a lot of good work to make it reasonable to create tools like this without a lot of effort.

Have a backup/restore plan

I, luckily, never really had an issue with backup and restore, but I do think you need some kind of plan. Users may forgive you if you lose a day’s worth of data but they’ll be pretty irritated if you lose all of it… So have a backup. And test the restore process every once in a while. My backup strategy changed a good amount for Evepraisal. I initially took SQL dumps and put them into my Google Drive using a script. However, after switching to Bolt for the database I had to come up with a different strategy. I used a combination of image-level backups along with a script that could export recent appraisals to Google Drive. Later on, I relied solely on the VM image backups.

Cleanup stale data automatically

After you’ve gotten the project to a stable state the things that will break are the log files that are never truncated or database tables that grow forever. Come up with a strategy to deal with this. I let systemd manage log files for me. By default, systemd will keep 4GB of log files for you, which was enough for me. I had a subprocess in the Evepraisal server that would clean up old appraisals automatically.

Get alerts

You need alerts, especially if your interest in the project varies over time. You may have an issue for days before someone tries to tell you so you should be proactive.

  • Log-based alerting
    • I used Papertrail’s free plan to handle logging and alerting for me. I filtered a lot of irrelevant things from the log so Papertrail would only receive actionable logs.
  • Uptime monitoring alerts
    • There are a lot of free services for this but I used Uptime Robot.
  • Google Alerts
    • Hopefully, the name of your service is unique enough that you can make a Google alert for the name. It will show Reddit threads, blog articles, forum posts, etc. that mention this name. It’s great, but it can also be a little slow.
  • Subscribe to email lists that will tell you if major changes are happening with the APIs that you’re using.
  • Be available on a social platform like Mastodon and list your handle on the website. If people are using your website someone will find your user and tell you if something is wrong if it’s broken for long enough.

Monetization

I used Google AdSense to generate revenue from Evepraisal. This revenue completely covered operational costs but it didn’t make too much beyond that. The only ongoing costs for this project were hosting and domain name registration. Monetization ensured that the project lived for as long as it did.

If I were to change things…

You should keep a log and make an entry for every time your attention is required, you can use that log to figure out what to focus on the next time you feel like working on the project.

I’ve always wanted to do more with the tool but life always got in the way. There are so many cool ways to show and process this kind of data. At various times I saw orders that made no sense pop up; orders that I don’t think were just margin scans. There would be sell-orders way below market value. Buy orders way above market value. It just made no sense to me because it would happen with high-volume items that had a fairly fixed market price. It didn’t happen super often, but you might be able to make some money off of those… as long as it isn’t an obvious scam.

Other than that… I don’t know. I think this was a massively successful project. I’m extremely proud to have contributed something so significant to the game.

gf in local

This all started from wanting the Burn Jita event to be more streamlined. I saw how chaotic it was when identifying targets and I felt like having a tool to quickly guess the value of the target was very much needed. By the time the next Burn Jita event came around Evepraisal’s usage was incredible. Once the event concluded, I came across a post listing all the killmails (records of ships being destroyed) associated with the event. I wrote a script to identify cargo scan appraisals containing items that match those found in the hull of the ships for each of the killmails. I ensured these appraisals were generated before the corresponding kills, suggesting that Evepraisal played a role in these victories. To my immense satisfaction, this script successfully pinpointed every single kill on the list-a truly gratifying accomplishment and a testament to how critical my project has become to the game.

In conclusion, Evepraisal’s journey underscores the importance of data reliability, automation, strong testing techniques, and monetization. I thank everyone who has used the tool in their everyday space lives. You kept me going for so long. See this post to see more about the reasoning behind shutting the website down.

o7

]]>How I learned to codehttps://kmcd.dev/posts/how-i-learned-to-code/Sat, 12 Aug 2023 00:00:00 +0000https://kmcd.dev/posts/how-i-learned-to-code/ <p> <img hspace="5" src="https://kmcd.dev/posts/how-i-learned-to-code/thumbnail_hu_404d0db577e880ce.webp" /> </p> This is a small post that discusses how I learned computer programming, from cheating at calculator games to anime fan site to a handmade content management system. Writing about this subject is weird to me. I am a programmer. I get paid to turn caffeine into software… but I feel like I am clueless when it comes to explaining to other people how to get started doing what I do. I learned to program for incredibly silly reasons. Here are a few of my wonderful early projects:

  • cheated at a game on my TI-83 (Falldown without wall detection).
  • announced my interest in Dragon Ball Z and anime in general with an anime fan website which eventually turned into me creating a database-driven review and article management system.
  • created a text-based multiplayer game (also spawned from my anime website).
  • visualized where on the screen I put my cursor while using my computer.
  • saw the “mood” of Reddit by aggregating the upvotes of top posts and comments of the top subreddits.
  • many more… with some I’m too ashamed of to put here.

Every single one of these projects has been enjoyable and none of it was a direct result of reading a book and following examples. I learned by literally stitching snippets of code together along with A LOT of trial and error. Eventually, I learned how important reference documentation is. Eventually, I learned how to make “real” applications where database passwords aren’t sitting in the source code. Eventually, I learned version control. I guess my point here is that there’s a lot to learn. Focus on doing something interesting. Solve problems. Create problems. Get messy. Even if you fail (and you will fail a lot) remember that you are learning and this is all part of the process.

Programming is literature. You can’t write amazing works after learning about grammar and sentence structure. This will take some time.

]]>Economists with (virtual) Gunshttps://kmcd.dev/posts/economists-with-guns/Fri, 11 Aug 2023 00:00:00 +0000https://kmcd.dev/posts/economists-with-guns/ This is a story about how I came to create evepraisal.com, a popular tool for Eve Online that is commonly used to price check cargo scans, contracts, EFT fittings, assets, and more. If none of that means anything to you that&#39;s totally fine! I barely know, too.

Note: This post was originally posted in 2015. I am re-posting it with some small edits. Updates are at the bottom.

This is a story about how I came to create evepraisal.com, a popular tool for Eve Online that is commonly used to price check cargo scans, contracts, EFT fittings, assets, and more. If none of that means anything to you that’s totally fine! I barely know, too.

Eve Online has always been an interesting game to me. For the uninitiated, Eve Online is a space-themed single-sharded MMO game that boasts an incredibly realistic economy and allows players to take high-stakes risks. This is the game that has had several extremely large fights over player-controlled territories involving thousands of people. It’s a game where politics, espionage and propaganda are effective weapons against your enemy.

Burn Jita

2 years ago, during a large player-made event called “Burn Jita”, an in-game alliance named Goonswarm and various other null-sec entities sought to blockade the biggest market hub in the game in hopes that the established trading center of Eve Online would be significantly impacted. I happened to be in an alliance comprising “various other null-sec entities” so I participated in the event. How could I not? Here’s how the scheme worked:

  • Designated people would cargo scan ships passing through nearby systems headed toward Jita. The information would be relayed through in-game and out-of-game channels.
  • A fleet of ships would be waiting to strike juicy targets foolish enough to enter or leave the trade hub with valuable cargo. The ships they were flying were cheap and could deal a lot of instant damage.
  • Once a transport ship with a significant amount of goods entered the system the fleet would strike. Jita is located in high sec which means every attacking ship would pretty quickly be destroyed by the computer-controlled police force, but the damage output of the entire fleet assured that the transport ship would turn into a pile of wreckage and loot.
  • Rinse; repeat.

Now take a second to appreciate the fact that several of these fleets were operating at any given time and the ships used to do these attacks were provided, free of charge, by each participating alliance.

My role in the process above was being one of the people who did step 1. I did a fair amount of ganking with the fleet but what I really enjoyed was pointing to a person and then waiting 10 minutes to see that his ship turned into a mangled wreck and his goods (or what’s left of them) were stolen to aid the further destruction of people like him. It was my (and my counterpart’s) actions that started the chain of events leading to the death of an enterprising player who paid no attention to player events. The act of firing space bullets was a detail.

Necessity is the mother of invention

After a few hours of scanning the cargo of every ship in sight, I realized that many freighters were stuffing their cargo hold with lots of useless junk to hide the true value of the cargo that they were carrying. This was a problem. Deciding who to attack was a time-sensitive task and not having a reliable cost estimate of what the ship contained made things harder than they had to be.

That’s where something clicked in my mind and over the next 3 hours I made a web-based tool that completely solved the ambiguity of this entire process and changed ganking into a legitimate economically-driven career that boasts cost/benefit analysis similar to (yet more exciting than) most of the other more reputable careers in Eve Online. That’s a big claim, but I don’t think I am far off.

Let me explain…

When you cargo-scan a potential target’s cargo hold, you get a list of all its contents. Up until recently, all you could do was look at the list and that was it. However, CCP recently added the ability to copy/paste from many different places in the Eve Online UI including the cargo scan result window. Cool, I thought. I could have users to give me a list of all the items in the potential target’s hanger. Now I just needed a way to calculate the prices. Eve Online has an amazing developer community. There were (and are) existing websites that had daily database dumps of aggregate pricing data for every market item in the game. I downloaded one of the database dumps and used that data as my price reference. After that, it was a matter of making a small website where people could give it cargo scan results and it would give back a reasonably accurate guess of the cargo’s value in less than a second. These estimates could be much better than what a human could do manually at the same time.

I randomly linked the website that I had just created in one of the fleet channels and it was a short time later that someone in Goonswarm broadcasted the link to the Goonswarm jabber server. I instantly got a flood of interested people trying out the tool and, to my surprise, the site held up fine. Later, I added the ability to share results so that the many scanners wouldn’t have to communicate the estimate they’re seeing from my website which further streamlined the process. In hindsight, what seemed like a small feature that was fairly easy to implement turned out to be the biggest selling point. The immediate feedback I got from people was invaluable and shaped my thoughts on the problem. To this day I’ve never had a bad experience asking people (even those who didn’t like the tool) what could be better.

The Burn Jita event continued for a couple more days and instead of cargo scanning ships, I was working on improving the tool that others used to find targets to gank more efficiently. The rest is history.

Update (2015)

The website slowly evolved into what is now known as Evepraisal. I added support for many different formats and switched from using price data from a static database dump to using more frequently updated sources like Eve-Central and Eve-marketdata.com. I’ve heard from several people who are career gankers in Eve who leverage Evepraisal to be profitable.

I am still waiting for the day that CCP makes a suitable replacement to my tool, but until then I’ll still be slowly improving on the tool that’s become a ganker’s best friend and a scammer’s worst nightmare.

Update (2020)

It’s sad to say but I’ve let Evepraisal kind of stagnate for a while. It sits in a weird position for me of being an extremely neat facet of my development as a programmer but also it weird to work on because I do not play Eve Online anymore. Since 2015, I’ve made several large changes to Evepraisal that were almost invisible to users. I rewrote the entire app in a different programming language. I sourced the market data from CCP’s own API (that didn’t exist before), which is very lucky because Eve-central eventually completely dissipated. I sourced type information from CCP’s static data dump. After making those two changes the amount of issues Evepraisal has had is extremely small. The requests for new features obviously come in, but I wanted the tool to be targeted at doing one specific thing well. For a while, I hoped that other tools would meet or surpass Evepraisal’s functionality. It took a long time but I believe there are several really good alternatives.

However, there have been two major issues. First, the Google Sheets addon that I made and published became unpublished due to Google completely changing how you administer Google Sheets add-ons. They also required a lot more from developers to improve their storefront… But it was just too many hoops for me to deal with. I attempted but I hit one roadblock too many and gave up. The second issue wasn’t my fault either. To make things more consistent, CCP started changing the names of several items. In the past when this happens it isn’t a big deal since the static data dump that CCP provides always had updates. But ever since the beginning of 2020, CCP has not kept the static data up-to-date with the name changes. It boggles my mind why they would do it this way. I’ve checked and there’s no reference to the new names anywhere in that package of static data files. So, alas, I resorted to actually changing Evepraisal’s code to support type aliases and hard-coded that list of aliases.

So here I am. A lot of stuff is happening in my life and I still don’t want to spend too much time on a 7-year-old project. I do have many, many ideas of how I might want to present market data. Eve Online is still an amazing MMO because of its open APIs and willingness to support third-party developers. And it’s the only MMO that regularly has actual large-scale player-driven events. With all of that said I plan to keep Evepraisal running for as long as people use it.

Update (2023)

I’ve decided to shut it down. See more about the reasoning here but you can kind of see my attitude 3 years ago.

]]>Visualizing the Internet (2023)https://kmcd.dev/posts/internet-map-2023/Tue, 01 Aug 2023 00:00:00 +0000https://kmcd.dev/posts/internet-map-2023/ <p> <img hspace="5" src="https://kmcd.dev/posts/internet-map-2023/geo-mercator-apac_hu_5f30212a05023f3e.webp" /> </p> Journey into the depths of the Internet with this incredible map showcasing undersea cables and internet exchanges. I recently expanded on my Internet map visualization that showed all of the undersea internet cables that run along the bottom of the oceans and seas. This time, I also added dots that represent the locations of all of the advertised internet exchanges in the world. The brighter/greener/bigger the dot, the more bandwidth the internet exchange supports.

Map of the Internet

And here is the same map but without country borders. I think this one looks beautiful:

Map of the Internet

Click here for full resolution image (warning, it’s big)

Click here for full resolution image (no borders) (warning, this one is also big)

What’s an internet exchange?

Internet Exchange:

An Internet exchange, often referred to as an Internet Exchange Point (IXP), is a physical infrastructure or facility where Internet Service Providers (ISPs) and other network operators connect their networks together to exchange Internet traffic. The primary purpose of an internet exchange is to improve the efficiency of data exchange between different networks by allowing direct peering between them. Instead of sending data through multiple third-party networks, which can lead to increased latency and costs, ISPs can directly exchange traffic at an internet exchange, resulting in faster and more cost-effective data transmission.

Internet exchanges play a crucial role in the functioning of the global internet. By facilitating direct peering, they help reduce the reliance on expensive long-distance links and enhance the overall performance and resilience of the internet. They also promote competition among ISPs and encourage the growth of internet infrastructure in specific regions.

Traceroute Command:

The traceroute command is a network diagnostic tool used to trace the route that data packets take from one computer to another across a network, typically the Internet. It helps identify the network path taken by the data, showing the sequence of intermediate devices (routers) that the packets pass through before reaching the destination.

$ sudo mtr apple.com
                            My traceroute  [v0.95]
kevins-mbp-m1.local (192.168.88.31) -> apple.com (17.22023-08-02T13:19:54+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                      Packets               Pings
 Host                               Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. router.lan                       0.0%     5    5.2   4.8   4.3   5.2   0.4
 2. 10.24.0.1                        0.0%     4    4.7   4.6   4.3   5.0   0.3
 3. soeborg2.net.gigabit.dk          0.0%     4    5.6   6.6   5.5   9.7   2.1
 4. apple-1.equinix-am1.nl-ix.net    0.0%     4   15.6  15.5  14.1  16.6   1.0
 5. apple.com.sg                     0.0%     4   15.2  15.0  14.5  15.6   0.5

In the provided example output:

  • kevins-mbp-m1.local is the name of the source host or the computer from which the traceroute is initiated. That’s my laptop, so I know that it is located in Copenhagen, Denmark.
  • apple.com is the destination host or the target server to which the traceroute is being performed.

The columns in the output represent the following information:

  • Host: The intermediate routers or nodes in the network path.
  • Loss%: The percentage of packet loss experienced while reaching each router.
  • Snt: The number of packets sent to each router.
  • Last: The last round-trip time taken to reach the router.
  • Avg: The average round-trip time to reach the router.
  • Best: The best (minimum) round-trip time to reach the router.
  • Wrst: The worst (maximum) round-trip time to reach the router.
  • StDev: The standard deviation of round-trip times to the router.

Identifying Exchange Location using Reverse IP Lookup:

In the traceroute output, you can sometimes infer the location of an internet exchange based on the names of the intermediate routers. Many internet exchanges are named explicitly to indicate their location. For example, in the provided traceroute, the router with the name apple-1.equinix-am1.nl-ix.net suggests that it is associated with the Equinix Amsterdam (AM1) data center connected through the NL-IX internet exchange. So my request was probably handled inside of Luttenbergweg 4, Amsterdam, Netherlands or… this building:

Equinix Amsterdam (AM1)

However, it’s important to note that not all routers or nodes in a traceroute will have such descriptive names. Some routers might be labeled with IP addresses or generic names that do not reveal their location. In such cases, it becomes more challenging to pinpoint the exact location of an internet exchange solely based on the traceroute output. Additionally, the reverse IP lookup method depends on the accuracy and up-to-date information in the IP address registries, and some routers might not be accurately represented.

Because of all of this, we can see the path that is taken when connecting to a server, which I think is really cool. With my example, I can know that it takes around 15.5 milliseconds for my network traffic to reach the Netherlands and I can tell that Apple is peering directly in this facility. You can verify this fact by looking at PeeringDB. Essentially, if you want apple.com to load faster for your users who are near(ish) to Amsterdam, you may want to run some fiber optic cable to this internet exchange. Put another way, the closer you are to the dots the closer you are to the internet.

Observations from the map

Now here’s the pretty part. Here are close-ups of different parts of the map with some general observations.

North America

North America

The internet exchanges in North American look how I’d imagine them. There are more and higher bandwidth internet exchanges on the eastern side of the US and a good amount along the west coast. However, there are some very important internet exchanges in Colorado. It likely serves as a good halfway point between east and west to transition traffic to different backbone providers.

Europe

Europe

After seeing the map from North America it’s rather surprising to see how dense European internet exchanges are. There’s almost too much to comment on here but I will say that it’s interesting how important Amsterdam, Frankfurt, and London are.

Asia-Pacific

Asia-Pacific

The area around Asia looks rather crazy, but it’s important to note that fiber links can visit islands along the way in order to give them internet access. Those islands will have networking gear that ‘drops’ and ‘adds’ certain channels of the signal. The network device is typically a ROADM (reconfigurable optical add-drop multiplexer). I bet you can guess why they made that an acronym. Also, I think open peering in China is not really a thing as I believe most traffic goes through so-called backbone providers instead of through internet exchanges. This is likely related to the great firewall of China. On the map, you only see a good number of internet exchanges in Hong Kong.

Africa

Africa

Africa has many optical cables that travel along coasts. I suspect the terrestrial optical networks aren’t as well developed in most parts of Africa. There are exceptions though. There is a lot of bandwidth capacity in internet exchanges in Capetown and Pretoria. Also, it’s crazy to see how many optical cables lie at the bottom of the Suez canal.

South America

South America

In South America we have Fortaleza and Sao Paula have a lot of landings and internet exchanges.

Svalbard

Svalbard

Svalbard is a set of islands owned by Norway that is waaay far north. It’s one of those places that gets 6 months of day followed by 6 months of night. And it’s always super cold. It’s interesting to see two undersea cables to a place like this. Svalbard is where the global seed vault lives, with seeds from all over the world. If you want to use these two cables you can check out some webcams that are set up in Svalbard.

Maldives

Maldives

Maldives is another interesting place where fiber optic cables live. You might notice that the fiber optic cables cross each other. It appears that it was done in order to connect islands that the first cable didn’t service before. The cables were commissioned around 5 years apart. It must have been a hard 5 years if you lived on one of the islands that didn’t get high-speed internet on the first round.

How it’s made

First, this is where I got the data from:

  • PeeringDB - PeeringDB is a user-driven database that offers information about network interconnection facilities and peering arrangements, supporting network administrators in optimizing Internet connectivity.
  • Simple Maps - I used this dataset to geolocate all internet exchanges in PeeringDB using the city and country fields.
  • Submarine Cable Map - TeleGeography maintains a database of all major submarine fiber optic cables and their status.

Next, these are the different tools/languages that I used:

  • Javascript (nodejs)
    • D3 - D3.js is a powerful JavaScript library for data visualization that allows developers to create interactive and dynamic charts, graphs, and other visual representations on the web using HTML, SVG, and CSS.
      • d3-node - a library that helps with running d3 inside of a nodejs environment
      • d3-geo - a library that handles translating coordinates for different map projections. I know it’s controversial now, but the Earth isn’t flat. So to make a flat image you have to pick how you are going to translate the globe coordinates onto a map. Despite its flaws, I used the Mercator projection because it is by far the most popular map projection.
  • The Go Programming Language - This is my current working language, so it’s what I used to integrate with the PeeringDB API and do some data processing/validation.
    • PeeringDB - A library for talking to the PeeringDB API.

The pipeline looks like this:

graph LR
    submarinecables[Submarine Cable Database] --> golang[Go Script]
    peeringdb[PeeringDB] --> golang[Go Script]
    geocities[Geolocation Database] --> golang[Go Script]
    golang[Go Script] --> node[Node JS Script]
    node[Node JS Script] --> svg[SVG]
    svg[SVG] --> convert[ImageMagick Convert] --> png[PNG]
    svg[SVG] --> morgify[ImageMagick Morgify] --> jpg[Displate JPG]
    style svg stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style jpg stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;
    style png stroke:#f66,stroke-width:2px,stroke-dasharray: 5, 5;

Mapping of Cities to lat/long GPS coordinates

I had so many data validation issues when doing this project. I should have expected this more since PeeringDB is user-managed and there’s no validation of the City field.

I had a lot of random issues mapping internet exchanges with their GPS coordinates. First, the geolocation database I was using is not complete. It only has around 43 thousand cities but there are many, many more towns and cities. The way I handled that was to manually map to the closest city that I do have geo coordinates. Additionally, the city field would be misspelled, formatted in a way differently than my database or the name of the city may have changed. This happens a lot because there are usually multiple ways to convert many languages into Latin characters. And sometimes cities just change their names. All-in-all I have 95 internet exchanges that I have had to manually map. I can’t guarantee that they are all correct… but it is good enough. You can see all of the places that I manually mapped on github.

Here are my favorites:

  • There were a few exchanges that spelled their city and sometimes the name of the exchange incorrectly. I’ve sent a few emails to make them aware and a few have fixed it!
  • There is an exchange named “Example IX” which is obviously just used as an example: entry on PeeringDB
  • The accepted Western name for the capital city of Ukraine changed from Kiev to Kyiv due to the efforts of a campaign called KyivNotKiev. There are several internet exchanges still using the old spelling in its address.

Thanks for reading

So far I’m pretty happy with the results of this little project. If I were to work more on this I would want there to be an actual heatmap instead of just drawing dots. I have attempted to do this but it was taking too long to figure out how to properly do it specifically with the node-d3 library. Also, there’s probably a lot of data validation that I could do… but I do feel like it’s not my job to fix this database so I’m probably not going to do that for the sake of a small side project.

The part that is missing still is the terrestrial fiber links. There’s not good public data on those for several reasons, but you can be assured that there are significant backbone fiber optic cables buried nearby almost every major highway and rail line in the US. So imagine lines that nearly mimic the US road system and you’ll have an idea of what that map would look like. That may be the next step for this map since it’s a bit hard to understand that we only have undersea cables.

References:

]]>softlayer-python: language bindings/CLI for a cloud companyhttps://kmcd.dev/posts/softlayer-python/Mon, 31 Jul 2023 00:00:00 +0000https://kmcd.dev/posts/softlayer-python/ <p> <img hspace="5" src="https://kmcd.dev/posts/softlayer-python/cover_hu_6e49055ded8250.webp" /> </p> I wrote and maintained language bindings for a large cloud company. Join me as I reflect on that experience. I used to work for a public cloud company called SoftLayer. As a cloud company, there is an API that customers can use to provision new virtual servers, load balancers, firewalls and whatever else you might want. On our team, we used SoftLayer services as a customer might and we ended up proving new products and just… experiencing what it was like as a customer. I loved the concept. Our team heavily used this practice of so-called “eating your own dog food.”

When dogfooding our services it became painfully obvious that our API was extremely hard to use. When making tooling we had to go through extremely complex ordering APIs that were designed for internal use and were exposed publicly for convenience. I preferred to do this work in Python at the time, so I used our public API bindings. This was the current state of the project on github. Let me point out some of the issues I had:

  • Didn’t work with Python 3.
  • Used class variables incorrectly, making it impossible to use multiple instances of the client at the same time.
  • Had an awkward use of dictionary accessors that made things more confusing as a user.
  • Had absolutely zero unit tests.
  • The Python module name used upper-case characters.

So… Immediately, I wanted to improve things. However, this was the first open-source project that I would modify that had a non-trivial number of users. Because of this, I learned to make strategic changes that followed the pattern of:

  • Add several unit tests for the part of the code that
  • Add a new interface that
  • Use the new interface in the implementation of the old interface (to reduce code duplication)
  • Add deprecation warnings for the old interface.
  • After enough time, bump the major version of the library and remove all code that is deprecated.

This pattern can be incredibly tedious. However, if you are maintaining a project used by many then you need to worry about upgrading. You need to write release notes. You need to give enough examples. You need to provide an upgrade path when you want to make breaking changes. This is super boring work, but it’s the difference between “when I upgrade this package everything breaks” and “when I upgrade this package, I get more cool new stuff”.

For perspective, here’s what the code looks like now. Note that there are now almost 2,000 unit tests. Note that the tests are run with several different versions of Python. And you should also note that there’s an entire command line client in the repo as well!

The CLI was also created from the same motivations. For us, it makes automating and testing things much easier. Instead of clicking through a virtual machine creation web form for 20 minutes I could just copy/paste a command that I’ve run before that could specify everything I would have to type or select anyway. In fact, here’s an example of that!

$ slcli vs create --hostname=example --domain=softlayer.com -f B1_1X2X25 -o DEBIAN_LATEST_64  --datacenter=ams01 --billing=hourly
This action will incur charges on your account. Continue? [y/N]: y
    :..........:.................................:......................................:...........................:
    :    ID    :               FQDN              :                 guid                 :         Order Date        :
    :..........:.................................:......................................:...........................:
    : 70112999 : testtesttest.test.com : 1abc7afb-9618-4835-89c9-586f3711d8ea : 2019-01-30T17:16:58-06:00 :
    :..........:.................................:......................................:...........................:
    :.........................................................................:
    :                            OrderId: 12345678                            :
    :.......:.................................................................:
    :  Cost : Description                                                     :
    :.......:.................................................................:
    :   0.0 : Debian GNU/Linux 9.x Stretch/Stable - Minimal Install (64 bit)  :
    :   0.0 : 25 GB (SAN)                                                     :
    :   0.0 : Reboot / Remote Console                                         :
    :   0.0 : 100 Mbps Public & Private Network Uplinks                       :
    :   0.0 : 0 GB Bandwidth Allotment                                        :
    :   0.0 : 1 IP Address                                                    :
    :   0.0 : Host Ping and TCP Service Monitoring                            :
    :   0.0 : Email and Ticket                                                :
    :   0.0 : Automated Reboot from Monitoring                                :
    :   0.0 : Unlimited SSL VPN Users & 1 PPTP VPN User per account           :
    :   0.0 : 2 GB                                                            :
    :   0.0 : 1 x 2.0 GHz or higher Core                                      :
    : 0.000 : Total hourly cost                                               :
    :.......:.................................................................:

Here we’re creating a VM inside of the Amsterdam data center that uses the latest Debian image with 2GB of RAM, a single CPU core and 25GB of disk space. All by copying a single command. This is super powerful. If you’re wondering why everything is free it’s because our account had special billing. 😛

After you create a VM, you can also list the running instances to see it:

$ slcli vs list
:.........:............:....................:.......:........:................:..............:....................:
:    id   : datacenter :       host         : cores : memory :   primary_ip   :  backend_ip  : active_transaction :
:.........:............:....................:.......:........:................:..............:....................:
: 1234567 :   sjc01    :  test.example.com  :   4   :   4G   :    12.34.56    :   65.43.21   :         -          :
:.........:............:....................:.......:........:................:..............:....................:

The story is the same for several other products. Here’s a list of what products are supported today:

  • Account Management
  • Block Storage
  • Bandwidth Pools
  • CDN
  • Dedicated Hosts
  • DNS
  • Email
  • File Storage
  • Firewall
  • Global IP
  • Dedicated Hardware
  • Disk Images
  • IPSec
  • Licenses
  • Load Balancers
  • NAS
  • Object Storage
  • Ordering/Quotes
  • SSH Keys and Certificates
  • Security Groups
  • Subnets
  • Support Tickets
  • Users
  • VLANs
  • Virtual Servers

It’s remarkable how far this project has come. This started as a way to create some VMs with a script but it became an extremely important interface for the entire suite of cloud products. Slowly, the CLI grew to what it is today. This wasn’t a result of just me. Instead, several members of mine and other people’s teams contributed to the project. There were periods when I didn’t write a lot of code but would spend most of my time reviewing and guiding others to make their contributions. I learned a lot about the importance of enforcing a single style and keeping the quality of code high. Essentially, the ways customers interfaced was through the website, API or the CLI.

The CLI also drove more usage of the Python client. It encouraged this growth in several ways:

  • It showcased what was possible with the API in a way that the documentation just can’t do.
  • It acted as a good reference for “how can I do this with the API”. The more we added to the CLI, the fewer extra examples we needed to make. It is very important that the CLI was also open source for this reason.
  • It had a verbose flag that showed the API calls that were being made, greatly increasing visibility into how it works.

Here’s an example of what a command looks like when running a command using the verbose flag.

$ slcli -v vs detail 74397127
Calling: SoftLayer_Virtual_Guest::getObject(id=74397127, mask='id,globalIdentifier,fullyQualifiedDomainName,hostname,domain', filter='None', args=(), limit=None, offset=None))
Calling: SoftLayer_Virtual_Guest::getReverseDomainRecords(id=77460683, mask='', filter='None', args=(), limit=None, offset=None))
:..................:..............................................................:
:       name       :                            value                             :
:..................:..............................................................:
:  execution_time  :                          2.020334s                           :
:    api_calls     :        SoftLayer_Virtual_Guest::getObject (1.515583s)        :
:                  : SoftLayer_Virtual_Guest::getReverseDomainRecords (0.494480s) :
:     version      :                   softlayer-python/v5.7.2                    :
:  python_version  :           3.7.3 (default, Mar 27 2019, 09:23:15)             :
:                  :              [Clang 10.0.1 (clang-1001.0.46.3)]              :
: library_location : /Users/chris/Code/py3/lib/python3.7/site-packages/SoftLayer  :
:..................:..............................................................:

The more v characters you add, the more verbose the output gets. If you use -vvv then you will get the equivalent cURL commands to make the same API calls, which should be clear enough for any developer to make a client against the API.

$ slcli -vvv account summary
curl -u $SL_USER:$SL_APIKEY -X GET -H "Accept: */*" -H "Accept-Encoding: gzip, deflate, compress"  'https://api.softlayer.com/rest/v3.1/SoftLayer_Account/getObject.json?objectMask=mask%5B%0A++++++++++++nextInvoiceTotalAmount%2C%0A++++++++++++pendingInvoice%5BinvoiceTotalAmount%5D%2C%0A++++++++++++blockDeviceTemplateGroupCount%2C%0A++++++++++++dedicatedHostCount%2C%0A++++++++++++domainCount%2C%0A++++++++++++hardwareCount%2C%0A++++++++++++networkStorageCount%2C%0A++++++++++++openTicketCount%2C%0A++++++++++++networkVlanCount%2C%0A++++++++++++subnetCount%2C%0A++++++++++++userCount%2C%0A++++++++++++virtualGuestCount%0A++++++++++++%5D'

In summary, this was an incredibly successful side project. What started as a small script for internal use turned into a Swiss army knife that was a completely new way to access all products that SoftLayer offered. I learned so much about maintaining an open-source project, choosing reliable libraries to build on, code quality/style, and so much more.

References:

]]>SwFTP: SFTP/FTP Server For Openstack Swifthttps://kmcd.dev/posts/swftp/Sun, 30 Jul 2023 00:00:00 +0000https://kmcd.dev/posts/swftp/ <p> <img hspace="5" src="https://kmcd.dev/posts/swftp/cover_hu_f456d62d712f6990.webp" /> </p> Describing an old project of mine from 2014; an SFTP/FTP interface over an object storage API using Python Twisted. I used to work for a public cloud company called SoftLayer. Before S3’s API was the de-facto API standard that object storage services used several object storage APIs seemed like they could claim that crown. The company I worked for, SoftLayer, had recently come out with an Object Storage service based on the OpenStack Swift project. It came with its API, which is great. However, at the time, it was hard to get buy-in from customers to use under-supported APIs. They’d have to use unfamiliar tooling or even develop tooling themselves just to transfer files around… And if they currently had a product that didn’t support OpenStack Swift they may just be stuck. So I was charged with coming up with a solution for these customers.

After testing out a few different ideas I ended up deciding on the SFTP protocol. Object Storage doesn’t translate 1:1 with a “normal” filesystem but it was close enough. This might seem a strange decision in modern days where everyone has S3 integration but this was done as a way to integrate with older products and give customers a much more familiar feel with interacting with their data. I decided to call the project SwFTP.

graph LR
    cyberduck((Cyberduck)) -- SFTP --> swftp((SwFTP))
    filezilla((FileZilla)) -- SFTP --> swftp((SwFTP))
    client((SFTP/FTP Client)) -- SFTP or FTP --> swftp((SwFTP))
    swftp((SwFTP)) -- HTTPS --> swift((Swift))

I was all set. I needed to just… implement an SFTP server. Oof. That’s actually very challenging. You see, the SFTP protocol has lived for over a decade at that point so it has gone through several RFC drafts and it supports various extensions… And it’s built on top of SSH, which is also fairly complex to implement. So I needed to use something more proven. I needed a library that I could plug in OpenStack Swift integration into without needing to implement the wire protocol itself. Again, at the time this kind of ability was fairly rare. I was very experienced with Python and a bit with Go (nowadays I would totally write this project in Go, using this SFTP library and implementing the fs.Walker interface). That would have made my life so much simpler. But no, I was essentially stuck with Python’s Twisted. Twisted has a library called “Conche” which implements the SSH protocol and it allows you to hook into the SFTP subsystem, which is exactly what I needed. Twisted seemed to be the best option at the time, but it was (and still is) very hard to work with. The failure modes can be very complex. Plus, the SFTP protocol is fairly complex and SFTP clients will behave in vastly different ways. For instance, some clients, in order to maximize throughput, will concurrently send several batches of data at once when uploading a file without waiting for the acknowledgment to be received. We can’t behave similarly with an object storage API call so I needed to force the concurrent write requests to queue up properly until it was their turn to be sent down the wire to Swift. This, alone, is complex but I was also using an unfamiliar framework that has completely different sync primitives than I’ve used in the past… So I found this work to be very challenging… but in the end, it was very rewarding.

SwFTP was never the only project I was working on so my attention was split multiple ways. Despite that, after a year of working on this project, it was finally good enough to deploy to production as a supported service. Testing took a long time because, as I said, many SFTP clients behave very differently. I am happy about where I got the functional test framework since I was able to easily write code that would reproduce errors that we saw when testing, including some super complicated cases of race conditions. From this experience, I’ve learned some very important lessons about functional and manual testing.

All-in-all, SwFTP ended up being a success and a lot of data was transferred using this service. Thanks to my manager and support from others in the company I was able to perform all of this iteration and development as an open source project. There were no SoftLayer-specific implementation details included here so others could (and did) deploy the project for their own OpenStack Swift clusters.

By the way, if you’re having issues pronouncing “SwFTP” in your head then you aren’t alone. I used to call it something like “Swefteepee”.

References:

]]>Video: Morning Copenhagen Commutehttps://kmcd.dev/posts/morning-copenhagen-commute/Sun, 30 Jul 2023 00:00:00 +0000https://kmcd.dev/posts/morning-copenhagen-commute/ A video showing my commute to work on a bike from the island of Amager to the heart of Copenhagen, Denmark.
Youtube Link (if embed doesn’t work)

One year ago I made the leap and moved from Dallas, Texas to Copenhagen, Denmark with my family. There were many reasons for this move but one of the biggest motivators was to be able to live in a city where cycling was treated as a first-class mode of transportation. Cycling is efficient, great for your health, great for your mental health and is just a more pleasant way to get to work.

Last year I went from never riding a bike (seriously, I had only been on a bike 3 times, ever) to cycling being an integral part of my life. I am so thankful to live in such a bike-friendly city of Copenhagen and I’m lucky to have my new job in the city center. Linked above is an admittedly boring video of my commute to work but being able to bike to work is such an accomplishment for me that I figured I’d share it here.

]]>Goodbye Evepraisalhttps://kmcd.dev/posts/goodbye-evepraisal/Mon, 24 Jul 2023 00:00:00 +0000https://kmcd.dev/posts/goodbye-evepraisal/ <p> <img hspace="5" src="https://kmcd.dev/posts/goodbye-evepraisal/cover_hu_f33c30b6d1654b6e.webp" /> </p> Saying goodbye to an amazing long-running project; evepraisal price checking utility for the MMORPG game Eve Online

I posed a new article about operating Evepraisal for over a decade. It covers the technical details and lessons that I learned throughout the years.

Evepraisal has been a really fun project for me. I think it has made a real impact on how Eve Online is played by many players. However, the time has come to shut it down.

First off, if you are looking for alternatives, you may want to try one of these websites:

Recently, CCP banned the IP address that I currently use to gather market data from Eve Online’s ESI API. I started to work towards mitigating the issue by contacting support, but I stopped to think for a second about just… not. This may not surprise most people based on how nothing changed with the website for years, but I haven’t played Eve Online for several years now. As such, I’ve been mostly doing extremely rare maintenance on the tool. I have enough monitoring that usually I am alerted when something breaks (that’s how I even know that CCP banned my IP from ESI) and I spend some time fixing it and then I move on with my life. However, I’ve become more and more out of touch with the game. It appears to be changing and it just feels weird running a tool for a game I don’t have installed anywhere… for years…

So I’ve decided to just pull the plug. I’m sorry that advertisements may have been annoying but it essentially only made enough money to keep the site running. Thank you to everyone who has supported this project over the years. Eve Online has been a great game for many people for many years. I am happy to have had my role, however small, in shaping the game.

The source code for Evepraisal is open to all to use and modify. You can find it at github.com/evepraisal/go-evepraisal. Also, see my other post about Evepraisal for more information about how Evepraisal was made and what made it special.

o7, fly safe

If you have an interest in programming with Go, the Internet, gRPC or just want to read words that I write, check out my blog. You can start by reading the other evepraisal-related posts.

]]>Visualizing the spectrum of the sun (Part 2)https://kmcd.dev/posts/sun-spectra-image-v2.svg/Mon, 12 Sep 2022 00:00:00 +0000https://kmcd.dev/posts/sun-spectra-image-v2.svg/ <p> <img hspace="5" src="https://kmcd.dev/posts/sun-spectra-image-v2.svg/thumbnail_hu_e9954082d0ce26c3.webp" /> </p> Revisiting visualizing the light that comes from the sun. Spectrum analysis, wavelengths of light, data visualization Basic Details

You may know about my previous sun spectra project where I created a visualization of the sun’s observed electromagnetic spectrum. Well, this is the same thing but I made some different decisions on how to visualize it this time and I think it turned out beautifully. I thought it looked so great that you can get it on Displate!

Github | All output images | Displate

The sun’s spectra (visible light)

This represents the visible light that the sun puts out. Notice the holes. These holes and dim spots tell us what the sun is made of, how it works and even the temperature of different parts of the sun.

Notice that you can zoom in an incredible amount. This is possible because these images are SVG files that use vectors to represent the image instead of a pixel map. This allows the images to essentially be infinitely zoomable. Spectra of the sun in the visible spectrum Click here to see the full resolution

The sun’s spectra (full spectrum)

Here’s the entire spectrum. As you may know, there is light that we cannot see because it either has too high or too low of a frequency. This visualization shows you the intensity levels for frequencies even outside of the visual spectrum. The pinkish color at the top is Ultraviolet light. The magenta-colored section under read is infrared light. Everything below that can be classified as radio waves. Scroll down a bit more to see an image with these sections annotated.

Spectra of the sun in all spectrums Click here to see the full resolution

The sun’s spectra (annotated)

Here is a version of the image that has text which describes what you’re seeing.

Spectra of the sun, annotated Click here to see the full resolution

Spectroscopy is a very interesting science and the basis for it is rooted in our understanding of quantum physics. We know a lot about how the sun and other stars work using this science. However, we don’t fully understand EVERY observation in these spectral lines. There’s always more to learn and understand. Physics in particular has several massive mysteries just waiting to be solved.

A better explanation can be found in this video](https://www.youtube.com/watch?v=gVZwdYZqCUI) from the youtube channel Be smart.

]]>Visualizing the Internet (2022)https://kmcd.dev/posts/internet-map-2022/Sat, 26 Feb 2022 00:00:00 +0000https://kmcd.dev/posts/internet-map-2022/ I drew a pretty map that shows the underwater cables that carry our data around the world; fiber optic cables, submarine cables

There is an updated version of this map detailed in this updated post.

Basic Details

I used data from the submarinecablemap.com website to create my visualization of Submarine Cables that live under our oceans and carry the majority of trans-continental internet traffic. Mostly, I wanted a ‘dark mode’ version of the map but I also plan on adding some interesting annotations from different sources and computing some metrics… Like there is enough fiber optic cable under the oceans to wrap the earth over 103 times! These SVGs were made with javascript, d3. I also used this experience to look at different map projections, which is neat.

Github | All output images

Here are the resulting images.

geo-mercator.svg

Map of the Internet
click here for full resolution

geo-conic-conformal.svg

Map of the Internet
click here for full resolution

geo-conic-equal-area.svg

Map of the Internet
click here for full resolution

geo-natural-earth-1.svg

Map of the Internet
click here for full resolution

]]>Evepraisal: A price estimation tool for Eve Onlinehttps://kmcd.dev/posts/evepraisal.com/Sun, 20 Feb 2022 00:00:00 +0000https://kmcd.dev/posts/evepraisal.com/ <p> <img hspace="5" src="https://kmcd.dev/posts/evepraisal.com/thumbnail_hu_7b3a5865607964e2.webp" /> </p> Evepraisal - Introducing the price estimation tool for the MMORPG game Eve Online that I created. Sometimes the best way to play is not to play Basic Details

Evepraisal is a web-based bulk price estimation tool for the game Eve Online. It has gained wide use and has a fairly high amount of traffic for such a simple utility. Almost anyone who plays the game has used it.

Go version on GitHub | Python version on Github

How it was created

This is a story about how I came to create evepraisal.com, a popular tool for Eve Online that is commonly used to price check cargo scans, contracts, EFT fittings, assets, and more. If none of that means anything to you that’s totally fine! I barely know, too.

Eve Online has always been an interesting game for me. For the uninitiated, Eve Online is a space-themed single-sharded MMO game that boasts an incredibly realistic economy and allows players to take high-stakes risks. This is the game that has had several extremely large fights over player-controlled territories involving thousands of people. It’s a game where politics, espionage and propaganda are effective weapons against your enemy.

Burn Jita

2 years ago, during a large player-made event called “Burn Jita”, an in-game alliance named Goonswarm and various other null-sec entities sought to blockade the biggest market hub in the game in hopes that the established trading center of Eve Online would be significantly impacted. I happened to be in an alliance comprising “various other null-sec entities” so I participated in the event. How could I not? Here’s how the scheme worked:

  1. Designated people would cargo scan ships passing through nearby systems headed toward Jita. The information would be relayed through in-game and out-of-game channels.
  2. A fleet of ships would be waiting to strike juicy targets foolish enough to enter or leave the trade hub with valuable cargo. The ships they were flying were cheap and could deal a lot of instant damage.
  3. Once a transport ship with a significant amount of goods entered the system the fleet would strike. Jita is located in high sec which means every attacking ship would pretty quickly be destroyed by the computer-controlled police force, but the damage output of the entire fleet assured that the transport ship would turn into a pile of wreckage and loot.
  4. Rinse; repeat.

Now take a second to appreciate the fact that several of these fleets were operating at any given time and the ships used to do these attacks were provided, free of charge, by each participating alliance.

My role in the process above was being one of the people who did step 1. I did a fair amount of ganking with the fleet but what I really enjoyed was pointing to a person and then waiting 10 minutes to see that his ship turned into a mangled wreck and his goods (or what’s left of them) were stolen to aid the further destruction of people like him. It was my (and my counterpart’s) actions that started the chain of events leading to the death of an enterprising player who paid no attention to player events. The act of firing space bullets was a detail.

Necessity is the mother of invention

After a few hours of scanning the cargo of every ship in sight, I realized that many freighters were stuffing their cargo hold with lots of useless junk to hide the true value of the cargo that they were carrying. This was a problem. Deciding who to attack was a time-sensitive task and not having a reliable cost estimate of what the ship contained made things harder than they had to be.

That’s where something clicked in my mind and over the next 3 hours I made a web-based tool that completely solved the ambiguity of this entire process and changed ganking into a legitimate economically-driven career that boasts cost/benefit analysis similar to (yet more exciting than) most of the other more reputable careers in Eve Online. That’s a big claim, but I don’t think I am far off.

How it works

When you cargo-scan a potential target’s cargo hold, you get a list of all its contents. Up until recently, all you could do was look at the list and that was it. However, CCP recently added the ability to copy/paste from many different places in the Eve Online UI including the cargo scan result window. Cool, I thought. I could have users give me a list of all the items in the potential target’s hanger. Now I just needed a way to calculate the prices. Eve Online has an amazing developer community. There were (and are) existing websites that had daily database dumps of aggregate pricing data for every market item in the game. I downloaded one of the database dumps and used that data as my price reference. After that, it was a matter of making a small website where people could give it cargo scan results and it would give back a reasonably accurate guess of the cargo’s value in less than a second. These estimates could be much better than what a human could do manually at the same time.

I randomly linked the website that I had just created in one of the fleet channels and it was a short time later that someone in Goonswarm broadcasted the link to the Goonswarm jabber server. I instantly got a flood of interested people trying out the tool and, to my surprise, the site held up fine. Later, I added the ability to share results so that the many scanners wouldn’t have to communicate the estimate they’re seeing from my website which further streamlined the process. In hindsight, what seemed like a small feature that was fairly easy to implement turned out to be the biggest selling point. The immediate feedback I got from people was invaluable and shaped my thoughts on the problem. To this day I’ve never had a bad experience asking people (even those who didn’t like the tool) what could be better.

The Burn Jita event continued for a couple more days and instead of cargo scanning ships, I was working on improving the tool that others used to find targets to gank more efficiently. The rest is history.

Screenshot of appraisal

The website slowly evolved into what is now known as Evepraisal. I added support for many different formats and switched from using price data from a static database dump to using more frequently updated sources like Eve-Central, Eve-marketdata.com and eventually the official ESI API had support for everything IU needed. I’ve heard from several people who are career gankers in Eve who leverage Evepraisal to be profitable.

Screenshot of evepraisal.com homepage

I’ve since shutdown the website. See more about the reasoning here.

]]>Visualizing the spectrum of the sunhttps://kmcd.dev/posts/sun-spectra-image.svg/Sat, 19 Feb 2022 00:00:00 +0000https://kmcd.dev/posts/sun-spectra-image.svg/ <p> <img hspace="5" src="https://kmcd.dev/posts/sun-spectra-image.svg/thumbnail_hu_7379da2aa18637b4.webp" /> </p> Visualizing the visual spectrum of the sun. Spectrum analysis, wavelengths of light, data visualization Basic Details

We all know the light that we get from the sun is white, meaning it contains about the same amount of every single color. If it had more green than other colors then the sun would give off green light. That’s awesome. Except… it’s not really true. There are certain colors that the sun does NOT give us. There are essentially holes in every rainbow. And the images below show exactly where those holes are.

Github | All output images

The sun’s spectra (visible light)

Spectra of the sun in the visible spectrum Click here to see the full resolution

The sun’s spectra (full spectrum)

I also made a version that shows the NON-visible spectrum. I can’t use pretty colors for this because we literally don’t have colors to map this part of the spectrum to. So I used greyscale (a gradient from black to white) to denote the intensity of this kind of light emitted from the sun. The image looks blurry but that’s an artifact of how precise the data is.

Spectra of the sun in all spectrums Click here to see the full resolution

The sun’s spectra (annotated)

I also made a version that has text that describes what you’re seeing.

Spectra of the sun, annotated Click here to see the full resolution

]]>