Feasibility of supporting pb imports in grpc-transcode plugin

[AlexAi27]

Feasibility of supporting pb imports in grpc-transcode plugin

https://github.com/starwing/lua-protobuf/blob/master/README.md

With documents of lua-protobuf, the unknown_import call back of protoc object can be used to dealing with unloaded import protos. Following is an example that dealing with nested proto imports, and its output.

local pb = require "pb"
local protoc = require "protoc"

pb.state(nil)
protoc.reload()

local p  = protoc.new()
p.include_imports = true
p.unknown_import = function (self, module_name)
    print("loading "..module_name.."...")
    local f, err = io.open("../../../third_party/"..module_name, "r")
    local str = f:read("a")
    f:close()
    p:load(str, module_name)
    return p.loaded
end

local file = io.open("../xxx.proto", "r")
local content = file:read("a")
file:close()
local ok = p:load(content, "xxx.proto")
print(ok)


-- outputs:
-- loading google/api/annotations.proto...
-- loading google/api/http.proto...
-- loading google/protobuf/descriptor.proto...
-- loading protoc-gen-swagger/options/annotations.proto...
-- loading protoc-gen-swagger/options/openapiv2.proto...
-- loading google/protobuf/any.proto...
-- loading google/protobuf/struct.proto...
-- loading validate/validate.proto...
-- loading google/protobuf/duration.proto...
-- loading google/protobuf/timestamp.proto...
-- true

The difficulty of dealing with proto imports is that the proto files in apisix are stored in ETCD in /{prefix}/proto/ without directory structure.

Cases of using protobuf

There may be three cases that using protobuf, which are:

• [Case 1] Single protobuf file

my-pb-project
└── myproto.proto

• [Case 2] Single protobuf file with third-party dependencies

.
├── my-pb-project
│   └── myproto.proto
└── third_party
    └── google
        ├── api
        │   ├── annotations.proto
        │   └── http.proto
        └── protobuf
            ├── any.proto
            ├── api.proto
            ├── compiler
            │   └── plugin.proto
            ├── descriptor.proto
            ├── duration.proto
            ├── empty.proto
            ├── field_mask.proto
            ├── source_context.proto
            ├── struct.proto
            ├── timestamp.proto
            ├── type.proto
            └── wrappers.proto

• [Case 3] Multiple user's protobuf files depend each other

.
├── my-pb-project
│   ├── myproto1.proto // depend on pkg and third_party
│   ├── myproto.proto // depend on pkg
│   └── pkg
│       ├── depend1.proto
│       └── depend2.proto
└── third_party
    └── google
        ├── api
        │   ├── annotations.proto
        │   └── http.proto
        └── protobuf
            ├── any.proto
            ├── api.proto
            ├── compiler
            │   └── plugin.proto
            ├── descriptor.proto
            ├── duration.proto
            ├── empty.proto
            ├── field_mask.proto
            ├── source_context.proto
            ├── struct.proto
            ├── timestamp.proto
            ├── type.proto
            └── wrappers.proto

ways to deal with directory structure of proto files

use apisix config path

We can assume that /usr/local/apisix/conf/protos/ is the protobuf dependencies searching directory. We can put all third_party dependencies and dependencies that written by users in this searching directory. In this way, apisix can check whether the unknown import proto files exists in the searching directory, and load proto files automatically.

use ETCD

Same to the previous, storing proto's in etcd with directory structured keys. That is, make prefix /apisix/proto/ in etcd as the proto searching directory. This solution may break the code logic and admin api of storing and fetching proto content. Another difficulty is how to load directories into etcd.

[spacewander]

We just supported it in the master branch.

If your .proto file contains imports, or you want to combine multiple .proto files into a proto, you can use .pb file to create the proto.

https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/grpc-transcode.md