Protobuf and gRPC rules for Bazel

Bazel rules for building Protocol Buffers ± gRPC code and libraries from proto_library targets

Announcements

• 2019/10/10: Bazel 1.0.0 has been released but is not yet supported fully by rules_proto_grpc. Please follow the Bazel 1.0.0 Support issue if you are waiting for support, which will be released in version 1.0.0 of rules_proto_grpc. If you are seeing issues with Bazel 1.0.0 that are not mentioned or linked in that issue, please add a comment or open a new issue.

Contents:

• Overview
• Installation
• Rules
  • Android
  • Closure
  • C++
  • C#
  • D
  • Go
  • Go (gogoprotobuf)
  • grpc-gateway
  • gRPC-Web
  • grpc.js
  • Java
  • Node.js
  • Objective-C
  • PHP
  • Python
  • Ruby
  • Rust
  • Scala
  • Swift
• Example Usage
• Migration
• Developers
  • Code Layout
  • Rule Generation
  • How-it-works
  • Developing Custom Plugins
• License
• Contributing

Overview

These rules provide Protocol Buffers (Protobuf) and gRPC rules for a range of languages and services.

Each supported language ({lang} below) is generally split into four rule flavours:

• {lang}_proto_compile: Provides generated files from the Protobuf protoc plugin for the language. e.g for C++ this provides the generated *.pb.cc and *.pb.h files.

• {lang}_proto_library: Provides a language-specific library from the generated Protobuf protoc plugin outputs, along with necessary dependencies. e.g for C++ this provides a Bazel native cpp_library created from the generated *.pb.cc and *pb.h files, with the Protobuf library linked. For languages that do not have a 'library' concept, this rule may not exist.

• {lang}_grpc_compile: Provides generated files from both the Protobuf and gRPC protoc plugins for the language. e.g for C++ this provides the generated *.pb.cc, *.grpc.pb.cc, *.pb.h and *.grpc.pb.h files.

• {lang}_proto_library: Provides a language-specific library from the generated Protobuf and gRPC protoc plugins outputs, along with necessary dependencies. e.g for C++ this provides a Bazel native cpp_library created from the generated *.pb.cc, *.grpc.pb.cc, *.pb.h and *.grpc.pb.h files, with the Protobuf and gRPC libraries linked. For languages that do not have a 'library' concept, this rule may not exist.

Therefore, if you are solely interested in the generated source code artifacts, use the {lang}_{proto|grpc}_compile rules. Otherwise, if you want a ready-to-go library, use the {lang}_{proto|grpc}_library rules.

These rules are derived from the excellent stackb/rules_proto and add aspect-based compilation to all languages, allowing for all proto_library options to be expressed in user code.

Installation

Add rules_proto_grpc your WORKSPACE file:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_proto_grpc",
    urls = ["https://github.com/rules-proto-grpc/rules_proto_grpc/archive/0.2.0.tar.gz"],
    sha256 = "1e08cd6c61f893417b14930ca342950f5f22f71f929a38a8c4bbfeae2a80d03e",
    strip_prefix = "rules_proto_grpc-0.2.0",
)

load("@rules_proto_grpc//:repositories.bzl", "rules_proto_grpc_toolchains")
rules_proto_grpc_toolchains()

It is recommended that you use the tagged releases for stable rules. Master is intended to be 'ready-to-use', but may be unstable at certain periods. To be notified of new releases, you can use GitHub's 'Watch Releases Only' on the repository.

Important: You will also need to follow instructions in the language-specific README.md for additional workspace dependencies that may be required.

Rules

Language	Rule	Description
Android	android_proto_compile	Generates an Android protobuf .jar artifact (example)
Android	android_grpc_compile	Generates Android protobuf+gRPC .jar artifacts (example)
Android	android_proto_library	Generates an Android protobuf library using android_library from rules_android (example)
Android	android_grpc_library	Generates Android protobuf+gRPC library using android_library from rules_android (example)
Closure	closure_proto_compile	Generates Closure protobuf .js files (example)
Closure	closure_proto_library	Generates a Closure library with compiled protobuf .js files using closure_js_library from rules_closure (example)
C++	cpp_proto_compile	Generates C++ protobuf .h & .cc artifacts (example)
C++	cpp_grpc_compile	Generates C++ protobuf+gRPC .h & .cc artifacts (example)
C++	cpp_proto_library	Generates a C++ protobuf library using cc_library, with dependencies linked (example)
C++	cpp_grpc_library	Generates a C++ protobuf+gRPC library using cc_library, with dependencies linked (example)
C#	csharp_proto_compile	Generates C# protobuf .cs artifacts (example)
C#	csharp_grpc_compile	Generates C# protobuf+gRPC .cs artifacts (example)
C#	csharp_proto_library	Generates a C# protobuf library using core_library from rules_dotnet (example)
C#	csharp_grpc_library	Generates a C# protobuf+gRPC library using core_library from rules_dotnet (example)
D	d_proto_compile	Generates D protobuf .d artifacts (example)
D	d_proto_library	Generates a D protobuf library using d_library from rules_d (example)
Go	go_proto_compile	Generates Go protobuf .go artifacts (example)
Go	go_grpc_compile	Generates Go protobuf+gRPC .go artifacts (example)
Go	go_proto_library	Generates a Go protobuf library using go_library from rules_go (example)
Go	go_grpc_library	Generates a Go protobuf+gRPC library using go_library from rules_go (example)
Java	java_proto_compile	Generates a Java protobuf srcjar artifact (example)
Java	java_grpc_compile	Generates a Java protobuf+gRPC srcjar artifact (example)
Java	java_proto_library	Generates a Java protobuf library using java_library (example)
Java	java_grpc_library	Generates a Java protobuf+gRPC library using java_library (example)
Node.js	nodejs_proto_compile	Generates Node.js protobuf .js artifacts (example)
Node.js	nodejs_grpc_compile	Generates Node.js protobuf+gRPC .js artifacts (example)
Objective-C	objc_proto_compile	Generates Objective-C protobuf .m & .h artifacts (example)
Objective-C	objc_grpc_compile	Generates Objective-C protobuf+gRPC .m & .h artifacts (example)
Objective-C	objc_proto_library	Generates an Objective-C protobuf library using objc_library (example)
PHP	php_proto_compile	Generates PHP protobuf .php artifacts (example)
PHP	php_grpc_compile	Generates PHP protobuf+gRPC .php artifacts (example)
Python	python_proto_compile	Generates Python protobuf .py artifacts (example)
Python	python_grpc_compile	Generates Python protobuf+gRPC .py artifacts (example)
Python	python_grpclib_compile	Generates Python protobuf+grpclib .py artifacts (supports Python 3 only) (example)
Python	python_proto_library	Generates a Python protobuf library using py_library (example)
Python	python_grpc_library	Generates a Python protobuf+gRPC library using py_library (example)
Python	python_grpclib_library	Generates a Python protobuf+grpclib library using py_library (supports Python 3 only) (example)
Ruby	ruby_proto_compile	Generates Ruby protobuf .rb artifacts (example)
Ruby	ruby_grpc_compile	Generates Ruby protobuf+gRPC .rb artifacts (example)
Ruby	ruby_proto_library	Generates a Ruby protobuf library using ruby_library from rules_ruby (example)
Ruby	ruby_grpc_library	Generates a Ruby protobuf+gRPC library using ruby_library from rules_ruby (example)
Rust	rust_proto_compile	Generates Rust protobuf .rs artifacts (example)
Rust	rust_grpc_compile	Generates Rust protobuf+gRPC .rs artifacts (example)
Rust	rust_proto_library	Generates a Rust protobuf library using rust_library from rules_rust (example)
Rust	rust_grpc_library	Generates a Rust protobuf+gRPC library using rust_library from rules_rust (example)
Scala	scala_proto_compile	Generates a Scala protobuf .jar artifact (example)
Scala	scala_grpc_compile	Generates Scala protobuf+gRPC .jar artifacts (example)
Scala	scala_proto_library	Generates a Scala protobuf library using scala_library from rules_scala (example)
Scala	scala_grpc_library	Generates a Scala protobuf+gRPC library using scala_library from rules_scala (example)
Swift	swift_proto_compile	Generates Swift protobuf .swift artifacts (example)
Swift	swift_grpc_compile	Generates Swift protobuf+gRPC .swift artifacts (example)
Swift	swift_proto_library	Generates a Swift protobuf library (example)
Swift	swift_grpc_library	Generates a Swift protobuf+gRPC library (example)
Go (gogoprotobuf)	gogo_proto_compile	Generates gogo protobuf .go artifacts (example)
Go (gogoprotobuf)	gogo_grpc_compile	Generates gogo protobuf+gRPC .go artifacts (example)
Go (gogoprotobuf)	gogo_proto_library	Generates a Go gogo protobuf library using go_library from rules_go (example)
Go (gogoprotobuf)	gogo_grpc_library	Generates a Go gogo protobuf+gRPC library using go_library from rules_go (example)
Go (gogoprotobuf)	gogofast_proto_compile	Generates gogofast protobuf .go artifacts (example)
Go (gogoprotobuf)	gogofast_grpc_compile	Generates gogofast protobuf+gRPC .go artifacts (example)
Go (gogoprotobuf)	gogofast_proto_library	Generates a Go gogofast protobuf library using go_library from rules_go (example)
Go (gogoprotobuf)	gogofast_grpc_library	Generates a Go gogofast protobuf+gRPC library using go_library from rules_go (example)
Go (gogoprotobuf)	gogofaster_proto_compile	Generates gogofaster protobuf .go artifacts (example)
Go (gogoprotobuf)	gogofaster_grpc_compile	Generates gogofaster protobuf+gRPC .go artifacts (example)
Go (gogoprotobuf)	gogofaster_proto_library	Generates a Go gogofaster protobuf library using go_library from rules_go (example)
Go (gogoprotobuf)	gogofaster_grpc_library	Generates a Go gogofaster protobuf+gRPC library using go_library from rules_go (example)
grpc-gateway	gateway_grpc_compile	Generates grpc-gateway .go files (example)
grpc-gateway	gateway_swagger_compile	Generates grpc-gateway swagger .json files (example)
grpc-gateway	gateway_grpc_library	Generates grpc-gateway library files (example)
grpc.js	grpcjs_grpc_compile	Generates protobuf closure grpc .js files (example)
grpc.js	grpcjs_grpc_library	Generates protobuf closure library .js files (example)
gRPC-Web	closure_grpc_compile	Generates Closure *.js protobuf+gRPC files (example)
gRPC-Web	commonjs_grpc_compile	Generates CommonJS *.js protobuf+gRPC files (example)
gRPC-Web	commonjs_dts_grpc_compile	Generates commonjs_dts *.js protobuf+gRPC files (example)
gRPC-Web	ts_grpc_compile	Generates CommonJS *.ts protobuf+gRPC files (example)
gRPC-Web	closure_grpc_library	Generates protobuf closure library *.js files (example)

Example Usage

These steps walk through the steps to go from a raw .proto file to a C++ library:

Step 1: Write a Protocol Buffer file (example: thing.proto):

syntax = "proto3";

package example;

import "google/protobuf/any.proto";

message Thing {
    string name = 1;
    google.protobuf.Any payload = 2;
}

Step 2: Write a BAZEL.build file with a native proto_library rule:

proto_library(
    name = "thing_proto",
    srcs = ["thing.proto"],
    deps = ["@com_google_protobuf//:any_proto"],
)

In this example we have a dependency on a well-known type any.proto, hence the proto_library to proto_library dependency ("@com_google_protobuf//:any_proto")

Step 3: Add a cpp_proto_compile rule (substitute cpp_ for the language of your choice).

NOTE: In this example thing.proto does not include service definitions (gRPC). For protos with services, use the cpp_grpc_compile rule instead.

# BUILD.bazel
load("@rules_proto_grpc//cpp:defs.bzl", "cpp_proto_compile")

cpp_proto_compile(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)

But wait, before we can build this, we need to load the dependencies necessary for this rule (from cpp/README.md):

Step 4: Load the workspace macro corresponding to the build rule.

# WORKSPACE
load("@rules_proto_grpc//cpp:repositories.bzl", "cpp_repos")

cpp_repos()

We're now ready to build the rule:

Step 5: Build it!

$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

If we were only interested in the generated file artifacts, the cpp_grpc_compile rule would be fine. However, for convenience we'd rather have the outputs compiled into an *.so file. To do that, let's change the rule from cpp_proto_compile to cpp_proto_library:

# BUILD.bazel
load("@rules_proto_grpc//cpp:defs.bzl", "cpp_proto_library")

cpp_proto_library(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)

$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-bin/example/proto/libcpp_thing_proto.a
  bazel-bin/example/proto/libcpp_thing_proto.so  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

This way, we can use //example/proto:cpp_thing_proto as a dependency of any other cc_library or cc_binary rule as per normal.

NOTE: the cpp_proto_library implicitly calls cpp_proto_compile, and we can access that rule by adding _pb at the end of the rule name, like bazel build //example/proto:cpp_thing_proto_pb.

Migration

For users migrating from the stackb/rules_proto rules, please see the help at MIGRATION.md

Developers

Code Layout

Each language {lang} has a top-level subdirectory that contains:

1. {lang}/README.md: Generated documentation for the language rules.

2. {lang}/repositories.bzl: Macro functions that declare repository rule dependencies for that language.

3. {lang}/{rule}.bzl: Rule implementations of the form {lang}_{kind}_{type}, where kind is one of proto|grpc and type is one of compile|library.

4. {lang}/BUILD.bazel: proto_plugin() declarations for the available plugins for the language.

5. example/{lang}/{rule}/: Generated WORKSPACE and BUILD.bazel demonstrating standalone usage of the rules.

6. {lang}/example/routeguide/: Example routeguide example implementation, if possible.

The repository root directory contains the base rule defintions:

• plugin.bzl: A build rule that defines the name, tool binary, and options for a particular proto plugin.

• aspect.bzl: Contains the implementation of the compilation aspect. This is shared by all rules and is the heart of rules_proto_grpc; it calls protoc with a given list of plugins and generates output files.

Additional protoc plugins and their rules are scoped to the github repository name where the plugin resides.

Rule Generation

To help maintain consistency of the rule implementations and documentation, all of the rule implementations are generated by the tool //tools/rulegen. Changes in the main README.md should be placed in tools/rulegen/README.header.md or tools/rulegen/README.footer.md. Changes to generated rules should be put in the source files (example: tools/rulegen/java.go).

How-it-works

Briefly, here's how the rules work:

1. Using the proto_library graph, an aspect walks through the ProtoInfo providers on the deps attribute to {lang}_{proto|grpc}_compile. This finds all the directly and transitively required proto files, along with their options.

2. At each node visited by the aspect, protoc is invoked with the relevant plugins and options to generate the desired outputs. The aspect uses only the generated proto descriptors from the ProtoInfo providers.

3. Once the aspect stage is complete, all generated outputs are optionally gathered into a final output tree.

4. For {lang}_{proto|grpc}_library rules, the generated outputs are then aggregated into a language-specific library. e.g a .so file for C++.

Developing Custom Plugins

Generally, follow the pattern seen in the multiple language examples in this repository. The basic idea is:

1. Load the plugin rule: load("@rules_proto_grpc//:plugin.bzl", "proto_plugin").
2. Define the rule, giving it a name, options (not mandatory), tool, and outputs.
3. tool is a label that refers to the binary executable for the plugin itself.
4. Choose your output type (pick one!):
   • outputs: a list of strings patterns that predicts the pattern of files generated by the plugin. For plugins that produce one output file per input proto file
   • out: the name of a single output file generated by the plugin.
   • output_directory: Set to true if your plugin generates files in a non-predictable way. e.g. if the output paths depend on the service names.
5. Create a compilation rule and aspect using the following template:

load("@rules_proto_grpc//:plugin.bzl", "ProtoPluginInfo")
load(
    "@rules_proto_grpc//:aspect.bzl",
    "ProtoLibraryAspectNodeInfo",
    "proto_compile_aspect_attrs",
    "proto_compile_aspect_impl",
    "proto_compile_attrs",
    "proto_compile_impl",
)

# Create aspect
example_aspect = aspect(
    implementation = proto_compile_aspect_impl,
    provides = [ProtoLibraryAspectNodeInfo],
    attr_aspects = ["deps"],
    attrs = dict(
        proto_compile_aspect_attrs,
        _plugins = attr.label_list(
            doc = "List of protoc plugins to apply",
            providers = [ProtoPluginInfo],
            default = [
                Label("//<LABEL OF YOUR PLUGIN>"),
            ],
        ),
        _prefix = attr.string(
            doc = "String used to disambiguate aspects when generating outputs",
            default = "example_aspect",
        )
    ),
    toolchains = ["@rules_proto_grpc//protobuf:toolchain_type"],
)

# Create compile rule to apply aspect
_rule = rule(
    implementation = proto_compile_impl,
    attrs = dict(
        proto_compile_attrs,
        deps = attr.label_list(
            mandatory = True,
            providers = [ProtoInfo, ProtoLibraryAspectNodeInfo],
            aspects = [example_aspect],
        ),
    ),
)

# Create macro for converting attrs and passing to compile
def example_compile(**kwargs):
    _rule(
        verbose_string = "{}".format(kwargs.get("verbose", 0)),
        merge_directories = True,
        **{k: v for k, v in kwargs.items() if k != "merge_directories"}
    )

License

This project is derived from stackb/rules_proto under the Apache 2.0 license and this project therefore maintains the terms of that license. An overview of the changes can be found at MIGRATION.md.

Contributing

Contributions are very welcome. Please see CONTRIBUTING.md for further details.