Protocol Buffers (protobuf) Reference

Kip Landergren

(Updated: )

My cheat sheet for Protocol Buffers (protobuf) covering installation and helpful resources.

Contents

Installation

Output Languages

go (Go Programming Language golang)

protoc-gen-go

Go Import Paths

A Go import path can be specified within the .proto file using:

option go_package = "example.com/foo";

or on the command line with the go_opt compiler flag:

--go_opt=M$PROTO_FILE=$GO_IMPORT_PATH

Compiler Invocation
go_out path/to/output/directory the directory protoc should write the generated Go output
go_opt paths=import (default) the output file is placed in a directory named after the Go package’s import path
module=$PREFIX the output file is placed in a directory named after the Go package’s import path, but with the specified directory prefix removed from the output filename; use this if outputting files directly into an existing Go module
paths=source_relative the output file is placed in the same relative directory as the input file; any specified Go import path is ignored
M$PROTO_FILE=$GO_IMPORT_PATH maps a single PROTO_FILE to the specified GO_IMPORT_PATH; use multiple --go_opt flags to map more than one file

A foo.proto with a Go import path of example.com/project will result in an output directory path of example.com/project/foo.pb.go

Resources

Frequently Asked Questions (FAQs)

Why did Protocol Buffers seem to jump to versions?

The versioning scheme was changed to allow more nimble per-language generator updates. The minor and patch versions of per-language generators remain coupled to protocol buffers. More details in the news item: Changes announced May 6, 2022 - Versioning, Python Updates, and JavaScript Support

Protocol Buffers Terminology

protoc
the protocol buffers compiler
native bindings
the automatically generated software libraries
wire format
the protocol buffers binary output format

protobuf-study

README.md

# protobuf-study

Study area for [Protocol Buffers (protobuf)](https://protobuf.dev/)

examples/golang-01-basics/Makefile

# for demonstration only so always run clean
bin/addressbook: clean
	# create the golang package from person.proto
	mkdir ./addressbook/generated
	protoc --fatal_warnings --go_opt=module=example.com/addressbook --go_out=./addressbook/ ./protos/person.proto
	# the generated golang library will contain imports that must be resolved. running `go mod tidy` will resolve them
	cd addressbook && go mod tidy
	# build the binary
	mkdir bin
	go build -C ./addressbook -o ../bin

.PHONY: clean
clean:
	rm -rf ./addressbook/generated/
	rm -rf ./bin

examples/golang-01-basics/README.md

# golang 01: basics

Generates a client golang package directly into a golang module for use.

## Usage
```
$ make
```

examples/golang-01-basics/addressbook/addressbook.go

package main

import (
	"fmt"

	pb "example.com/addressbook/generated"
)

func main() {
	p := pb.Person{FirstName: "Kip"}

	fmt.Printf("hello %s\n", p.FirstName)
}

examples/golang-01-basics/addressbook/go.mod

module example.com/addressbook

go 1.21

require google.golang.org/protobuf v1.32.0

examples/golang-01-basics/addressbook/go.sum

github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I=
google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=

examples/golang-01-basics/protos/person.proto

syntax = "proto3";

package protos;

// note that the prefix of this matches the module name in `go.mod`. If you
// were planning to output directly into multiple modules with different prefix
// paths you would likely specify during `protoc` invocation using `go_opt` to
// pair proto files with go import paths
option go_package = "example.com/addressbook/generated";

message Person {
  string first_name = 1;
}

examples/golang-02-well-known-types/Makefile

# for demonstration only so always run clean
bin/addressbook: clean
	# create the golang package from person.proto
	mkdir ./addressbook/generated
	protoc --fatal_warnings --go_opt=module=example.com/addressbook --go_out=./addressbook/ ./protos/person.proto
	# the generated golang library will contain imports that must be resolved. running `go mod tidy` will resolve them
	cd addressbook && go mod tidy
	# build the binary
	mkdir bin
	go build -C ./addressbook -o ../bin

.PHONY: clean
clean:
	rm -rf ./addressbook/generated/
	rm -rf ./bin

examples/golang-02-well-known-types/README.md

# golang 02: well known types

Expands on the basic example by using a [Well Known Type](https://protobuf.dev/reference/protobuf/google.protobuf/)

## Usage
```
$ make
```

examples/golang-02-well-known-types/addressbook/addressbook.go

package main

import (
	"fmt"

	pb "example.com/addressbook/generated"
	// this import declaration was found by inspecting the generated `person.pb.go`
	timestamppb "google.golang.org/protobuf/types/known/timestamppb"
)

func main() {
	p := pb.Person{FirstName: "Kip", LastUpdate: timestamppb.Now()}

	fmt.Printf("hello %s, the last update is: %v\n", p.FirstName, p.LastUpdate)
}

examples/golang-02-well-known-types/addressbook/go.mod

module example.com/addressbook

go 1.21

require google.golang.org/protobuf v1.32.0

examples/golang-02-well-known-types/addressbook/go.sum

github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I=
google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=

examples/golang-02-well-known-types/protos/person.proto

syntax = "proto3";

package protos;

// From the official documentation:

// The protocol compiler searches for imported files in a set of directories
// specified on the protocol compiler command line using the -I/--proto_path
// flag. If no flag was given, it looks in the directory in which the compiler
// was invoked. In general you should set the --proto_path flag to the root of
// your project and use fully qualified names for all imports.

// Certain types are able to be imported because of their inclusion with
// protocol buffers itself. These "well-known types" are encouraged to be used.
import "google/protobuf/timestamp.proto";

// note that the prefix of this matches the module name in `go.mod`. If you
// were planning to output directly into multiple modules with different prefix
// paths you would likely specify during `protoc` invocation using `go_opt` to
// pair proto files with go import paths
option go_package = "example.com/addressbook/generated";

message Person {
  string first_name = 1;

  // note that the field's type must be fully qualified
  google.protobuf.Timestamp last_update = 2;
}

examples/golang-03-generated-module/Makefile

# for demonstration only so always run clean
bin/addressbook: clean go-genproto
	# create the golang module from person.proto
	protoc --fatal_warnings --go_opt=module=example.com/company/go-genproto --go_out=./go-genproto/ ./protos/person.proto
	# the generated golang library will contain imports that must be resolved. running `go mod tidy` will resolve them
	cd go-genproto && go mod tidy
	# fix up the addressbook search
	cd addressbook && go mod edit -replace example.com/company/go-genproto=../go-genproto
	# fix up the requirement
	cd addressbook && go mod tidy
	# build the binary
	mkdir bin
	go build -C ./addressbook -o ../bin

.PHONY: go-genproto
go-genproto:
	mkdir go-genproto
	cd go-genproto && go mod init example.com/company/go-genproto

.PHONY: clean
clean:
	rm -rf ./go-genproto
	rm -rf ./bin

examples/golang-03-generated-module/README.md

# golang 03: generated module

Expands on the previous examples by creating a stand alone module that can be
used by multiple libraries

## Usage
```
$ make
```

examples/golang-03-generated-module/addressbook/addressbook.go

package main

import (
	"fmt"

	personpb "example.com/company/go-genproto/personpb"
	// this import declaration was found by inspecting the generated `person.pb.go`
	timestamppb "google.golang.org/protobuf/types/known/timestamppb"
)

func main() {
	p := personpb.Person{FirstName: "Kip", LastUpdate: timestamppb.Now()}

	fmt.Printf("hello %s, the last update is: %v\n", p.FirstName, p.LastUpdate)
}

examples/golang-03-generated-module/addressbook/go.mod

module example.com/addressbook

go 1.21.6

require (
	example.com/company/go-genproto v0.0.0-00010101000000-000000000000
	google.golang.org/protobuf v1.32.0
)

replace example.com/company/go-genproto => ../go-genproto

examples/golang-03-generated-module/addressbook/go.sum

github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I=
google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=

examples/golang-03-generated-module/go.mod

module example.com/company/go-genproto

go 1.21.6

replace example.com/greetings => ../greetings

examples/golang-03-generated-module/protos/person.proto

syntax = "proto3";

package protos;

// From the official documentation:

// The protocol compiler searches for imported files in a set of directories
// specified on the protocol compiler command line using the -I/--proto_path
// flag. If no flag was given, it looks in the directory in which the compiler
// was invoked. In general you should set the --proto_path flag to the root of
// your project and use fully qualified names for all imports.

// Certain types are able to be imported because of their inclusion with
// protocol buffers itself. These "well-known types" are encouraged to be used.
import "google/protobuf/timestamp.proto";

option go_package = "example.com/company/go-genproto/personpb";

message Person {
  string first_name = 1;

  // note that the field's type must be fully qualified
  google.protobuf.Timestamp last_update = 2;
}