bug: GRPC transcode can't work with protobuf.Struct

[gaoxingliang]

Current Behavior

Problem

The inner struct object has not been parsed correctly. and is empty when using grpc-transcode plugin.

details

I have a grpc serivce with has below definition:

--- the request part
message RawRequest {
  google.protobuf.Struct reqdata = 1;
  string type = 2;
  string serviceurl = 3;
  string source = 4;
  int32 page = 5;
  int32 size = 6;
}

and I try to send a post request in postman with below request args:

{
    "type":"7",
    "serviceurl" :"xxxx",
    "reqdata" :{
        "key":"yyyy",
        "key_type":"zzzz",
        "model":"tttt"
    },
    "source":"3"
}

and When I checking the logs, I found the reqdata object is empty (not null).

Expected Behavior

The struct object should has data.

Error Logs

No response

Steps to Reproduce

1. Create a routing from rest to grpc with grpc transcode.
2. and checking whether the grpc service side can receive the correct data.

Environment

apisix:3.0.0

[kingluo]

@gaoxingliang
It seems that there is bug for lua-protobuf (used by APISIX) to import file in proto file.
When I copy all definitions except package google.protobuf; from protobuf/src/google/protobuf/struct.proto, everything is ok.

test_struct.proto

syntax = "proto3";
package test;
//import "protobuf/src/google/protobuf/struct.proto";
//copied from struct.proto
option cc_enable_arenas = true;
option go_package = "google.golang.org/protobuf/types/known/structpb";
option java_package = "com.google.protobuf";
option java_outer_classname = "StructProto";
option java_multiple_files = true;
...
message RawRequest {
    //google.protobuf.Struct reqdata = 1;
    Struct reqdata = 1;
    string type = 2;
    string serviceurl = 3;
    string source = 4;
    int32 page = 5;
    int32 size = 6;
}

test_struct.lua

local pb = require "pb"
local protoc = require "protoc"
local cjson = require"cjson"

assert(protoc:loadfile("test_struct.proto"))
local data = [[
{
    "type":"7",
    "serviceurl" :"xxxx",
    "reqdata" :{
        "fields": {
            "name": {"number_value":3},
            "key": {"string_value":"yyyy"}
        }
    },
    "source":"3"
}
]]
local tbl = cjson.decode(data)
print(cjson.encode(tbl))
local data = pb.encode("test.RawRequest", tbl)
local decoded = pb.decode("test.RawRequest", data)
print(cjson.encode(decoded))

$ luajit test_struct.lua
{"source":"3","reqdata":{"fields":{"name":{"number_value":3},"key":{"string_value":"yyyy"}}},"serviceurl":"xxxx","type":"7"}
{"reqdata":{"fields":{"name":{"number_value":3,"kind":"number_value"},"key":{"string_value":"yyyy","kind":"string_value"}}},"serviceurl":"xxxx","source":"3","page":0,"size":0,"type":"7"}

[gaoxingliang]

@kingluo I checked the page:
https://github.com/starwing/lua-protobuf#type-mapping
It seems not support struct. I created an issue to check how to workaround. starwing/lua-protobuf#236

[kingluo]

@gaoxingliang

Struct is just a map with oneof value. They are ok in lua-protobuf:

"map" | "enum" | "message": whether the type is a map_entry type, enum type or message type.

https://github.com/starwing/lua-protobuf/blob/master/test.lua#L657

[oneof_name, oneof_index]: if this is a oneof field, this is the oneof name and index

https://github.com/starwing/lua-protobuf/blob/master/test.lua#L719

If you paste the definitions into one file without importing, the Struct works as expected. You could have a try.

{
   "reqdata":{
      "fields":{
         "name":{
            "number_value":3,
            "kind":"number_value"
         },
         "key":{
            "string_value":"yyyy",
            "kind":"string_value"
         }
      }
   },
   "serviceurl":"xxxx",
   "source":"3",
   "page":0,
   "size":0,
   "type":"7"
}

[kingluo]

@gaoxingliang I confirm it's a bug of grpc-transcode plugin.

Here is the bugfix (I will submit PR later):
master...kingluo:apisix:grpc_transcode_fix_map_type

Here is the test:
https://gist.github.com/kingluo/2a6af0b600cc9804870985458c472350