Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auth: support "authorization" token for grpc-gateway #7999

Merged
merged 3 commits into from
Jun 16, 2017

Conversation

hexfusion
Copy link
Contributor

The allows native use of grpc-gateway authorization header, providing direct access to metadata context while using intuitive naming.

Fixes #6643

auth/store.go Outdated
if !uok {
plog.Warningf("invalid auth token: %s", token)
return nil, ErrInvalidAuthToken
for k := range md {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid looping over all the metadata?

token := ""
if token, ok = md["token"]; !ok {
    token = md["authorization"]
}
if token == "" {
    return nil, nil
}
authInfo, uok := as.authInfoFromToken(ctx, token)
...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -8,6 +8,16 @@
"http",
"https"
],
"securityDefinitions" : {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this be clobbered after running scripts/genproto.sh?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@heyitsanthony yes, please excuse my ignorance I am just learning how all of this tooling works together. I will research proper solution further.

auth/store.go Outdated
plog.Warningf("invalid auth token: %s", token)
return nil, ErrInvalidAuthToken
}
plog.Debugf("checking value in loop %s", k)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we do want the debugging logging here, it needs to contain more context. checking value in a loop is too vague.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiang90 yes sir. I will clean this up on my next run, I appreciate the notes.

auth/store.go Outdated
plog.Warningf("invalid auth token: %s", token)
return nil, ErrInvalidAuthToken
for k := range md {
if k == "token" || k == "authorization" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The already existing key "token" is used because I defined it in here: https://github.com/coreos/etcd/blob/master/clientv3/client.go#L168
If "authorization" cannot be changed because of swagger, I think changing the existing "token" would be good in the future. It breaks compatibility so it cannot be done immediately. Possibly putting a comment of todo would be fine?

Copy link
Contributor Author

@hexfusion hexfusion Jun 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mitake grpc-gateway for now only allows direct access to authorization header. This may change if you follow issue. Otherwise I think that token could be used if you passed header Grpc-Metadata-Token but this is not very intuitive. grpc-ecosystem/grpc-gateway#311

So perhaps time will resolve this question. In the meantime I will add your comment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't think unifying the key names immediately. Just putting a comment or opening an issue for a remainder is ok.

@mitake
Copy link
Contributor

mitake commented Jun 1, 2017

@hexfusion thanks a lot for your PR. I think your change can support jwt tokens so the commit title shouldn't say about simple token explicitly.

@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 1, 2017

@mitake well I can see that adding these security definitions to the root of swagger is a complicated matter to automate, Currently proto files do not support the definition directly. So using the methods offered by grpc-gateway will not facilitate.

I have found ways of merging swagger json or defining a json/yml stub using go-swagger but this is not ideal either. I would like this to be seamless and not a hack of genproto.sh but that maybe the only option until you decide to move to OpenApi 3 and perhaps protobuf adds support for security definitions compatibe with swagger. My final solution involved using https://goswagger.io/generate/spec/meta.html. This seems interesting concept for automation but again not optimal.

But this is only my discovery if you have other ways please let me know. I will keep digging for now :)

@hexfusion hexfusion changed the title grpc-gateway: Support simple token via authorization header. grpc-gateway: Support token via authorization header. Jun 2, 2017
@heyitsanthony
Copy link
Contributor

heyitsanthony commented Jun 2, 2017

@hexfusion a hack in genproto.sh that inserts the swagger snippet into the generated json should be OK, the script is already doing some gross stuff to get around inadequate/impossible tooling so the seamless ship has sailed on that one.

Could this also have a curl e2e test like the ones in https://github.com/coreos/etcd/blob/master/e2e/v3_curl_test.go? It'd help as an example for how to use auth with grpc/json in addition to checking it all works via CI.

@gyuho gyuho force-pushed the master branch 2 times, most recently from 44ca396 to 4301f49 Compare June 2, 2017 15:53
@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 6, 2017

@mitake I created a Go script to append swagger security stubs using https://github.com/go-openapi tools in a programmatic way. Although I could not find a clean way to do this directly using proto files, I feel this is a fair compromise. The downside is the spec tools use a different order then method used by grpc-gateway so the diff is quite large. But this order should stay consistent on future iterations. If this seems reasonable I will finish the curl tests. Finally to deal with missing descriptions as a result of the process I resulted to appending (empty) vs "".

@heyitsanthony
Copy link
Contributor

@hexfusion please rebase this patchset? It's difficult to see how the code differs from master and review.

The approach looks OK but the schwag package should probably checkout to a fixed commit SHA like the other go get packages in the script.

@mitake
Copy link
Contributor

mitake commented Jun 8, 2017

@hexfusion I see, now the script/genproto.sh is updated so I think your way would be acceptable. Do you have any opinions about the change? @xiang90 @heyitsanthony

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch 2 times, most recently from eff9df1 to 67de94b Compare June 9, 2017 02:18
@heyitsanthony
Copy link
Contributor

@hexfusion OK, this is shaping up. Two last nits:

  1. This would be good as just two separate patches:
  • auth: support "authorization" token for grpc-gateway (with the auth/store.go changes)
  • scripts: generate swagger with authorization support (with the scripts/* and swagger json files)
  1. Need a test case in e2e/v3_curl_test.go that tries to access etcd using auth through curl.

Thanks!

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch 2 times, most recently from 14d2326 to 6c4dfff Compare June 10, 2017 16:35
@hexfusion
Copy link
Contributor Author

@heyitsanthony PTAL, thanks!

@hexfusion hexfusion changed the title grpc-gateway: Support token via authorization header. auth: support "authorization" token for grpc-gateway Jun 10, 2017
check(t, err)

cURLRes, err := proc.ExpectFunc(lineFunc)
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)

check(t, err)

jerr := json.Unmarshal([]byte(cURLRes), &authRes)
check(t, jerr)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, json.Unmarshall([]byte(cURLRes), &authRes))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@heyitsanthony thanks for notes, will resolve.


// put "bar" into "foo"
putreq, err := json.Marshal(&pb.PutRequest{Key: []byte("foo"), Value: []byte("bar")})
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)


// create root user
userreq, err := json.Marshal(&pb.AuthUserAddRequest{Name: string("root"), Password: string("toor")})
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)


// create root role
rolereq, err := json.Marshal(&pb.AuthRoleAddRequest{Name: string("root")})
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)

)
cmdArgs = cURLPrefixArgs(epc, "POST", cURLReq{endpoint: "/v3alpha/auth/authenticate", value: string(authreq)})
proc, err := spawnCmd(cmdArgs)
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)


// auth request
authreq, err := json.Marshal(&pb.AuthenticateRequest{Name: string("root"), Password: string("toor")})
check(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testutil.AssertNil(t, err)

@@ -162,3 +168,90 @@ func TestV3CurlTxn(t *testing.T) {
t.Fatalf("failed put with curl (%v)", err)
}
}

type AuthResponse struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use etcdserverpb.AuthenticationResponse instead of redeclaring?

@@ -4,7 +4,7 @@
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
// http://www.apache.org/licenses/LICENSE-2.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert?

@@ -24,6 +24,12 @@ import (
"github.com/grpc-ecosystem/grpc-gateway/runtime"
)

func check(t *testing.T, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can replace with testutil.AssertNil

@hexfusion
Copy link
Contributor Author

@heyitsanthony could you please help me better understand the root of this problem?

https://play.golang.org/p/o-s0_LI9Tr

@gyuho
Copy link
Contributor

gyuho commented Jun 13, 2017

@hexfusion Try without double-quotes on numbers inside JSON?

e.g.

{"header":{"cluster_id":8146361247912014845,"member_id":14895263327052482431,...

@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 13, 2017

@gyuho right but this is the JSON format returned by curl in the test. I don't believe I can control this format it returns this same quoting if I were to perform the curl command manually.

Appending json:",string" allows this usage, but this I believe would need to be a manual grooming as I don't believe it is supported by proto.

https://play.golang.org/p/W0KBfT_bM5

I guess the main issue is https://developers.google.com/protocol-buffers/docs/proto3#json

The uint64 and int64 types are json strings not numbers.

This is very frustrating :)

@heyitsanthony
Copy link
Contributor

there was some discussion about json output w/r/t 64-bit values on #7396; the decision at the time was to stick to the defaults

@hexfusion
Copy link
Contributor Author

@heyitsanthony I understand, so could I groom the JSON manually for the test or how to proceeded?

@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 13, 2017

perhaps something like this?

    cURLRes, err := proc.ExpectFunc(lineFunc)
    testutil.AssertNil(t, err)

    authRes := make(map[string]interface{})
    testutil.AssertNil(t, json.Unmarshal([]byte(cURLRes), &authRes))

    token, ok := authRes["token"].(string)
    if !ok {
        t.Fatalf("failed invalid token in authenticate response with curl")
    }

@heyitsanthony
Copy link
Contributor

@hexfusion yeah that looks OK

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch 2 times, most recently from a8a2e20 to 5b1a74f Compare June 13, 2017 20:38
@hexfusion
Copy link
Contributor Author

@heyitsanthony PTAL, let me know if you need anything else. Thanks!

Copy link
Contributor

@heyitsanthony heyitsanthony left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks!

@heyitsanthony
Copy link
Contributor

heyitsanthony commented Jun 14, 2017

@hexfusion can that Merge branch 'master' into grpc-gateway-auth commit be removed? Then this should be good to merge.

Defer to @mitake

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch from b4ea954 to 5b1a74f Compare June 14, 2017 23:12
@hexfusion
Copy link
Contributor Author

@heyitsanthony @mitake ok I rebased but it shows a basic conflict that I resolved causing the merge message last time. So maybe that can be resolved during merge? The change is very obvious. Anyways thanks to all for the assistance and input, very exciting!

@heyitsanthony
Copy link
Contributor

@hexfusion can you rebase on the current master branch? You should be able to resolve the conflict from there (the git SHAs were updated for a newer protobuf version)

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch from 15c6baa to 5b1a74f Compare June 15, 2017 00:09
@codecov-io
Copy link

codecov-io commented Jun 15, 2017

Codecov Report

❗ No coverage uploaded for pull request base (master@ee0c805). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff            @@
##             master   #7999   +/-   ##
========================================
  Coverage          ?   76.8%           
========================================
  Files             ?     342           
  Lines             ?   26701           
  Branches          ?       0           
========================================
  Hits              ?   20508           
  Misses            ?    4751           
  Partials          ?    1442

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee0c805...b19e2fa. Read the comment docs.

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch from 13bca87 to b19e2fa Compare June 15, 2017 00:20
@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 15, 2017

@heyitsanthony you sir are correct, resolved.

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch from b19e2fa to 5d869e0 Compare June 15, 2017 10:54
GRPC_GATEWAY_SHA="18d159699f2e83fc5bb9ef2f79465ca3f3122676"
# exact version of packages to build
GOGO_PROTO_SHA="8d70fb3182befc465c4a1eac8ad4d38ff49778e2"
GRPC_GATEWAY_SHA="84398b94e188ee336f307779b57b3aa91af7063c"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GOGO_PROTO_SHA="100ba4e885062801d56799d78530b73b178a78f3"
GRPC_GATEWAY_SHA="18d159699f2e83fc5bb9ef2f79465ca3f3122676"

otherwise clobbers update from 4ebeba0

@hexfusion hexfusion force-pushed the grpc-gateway-auth branch from 5d869e0 to c27634c Compare June 15, 2017 17:42
@hexfusion
Copy link
Contributor Author

hexfusion commented Jun 15, 2017

@heyitsanthony yes I updated, PTAL, thank you!

@xiang90
Copy link
Contributor

xiang90 commented Jun 16, 2017

@hexfusion thanks. merging.

@davissp14
Copy link
Contributor

Hey guys, are you guys planning on rolling this into 3.2.7?

@heyitsanthony
Copy link
Contributor

heyitsanthony commented Aug 22, 2017

@davissp14 it's planned for 3.3; backports are tagged with kind/need-backport

fuweid added a commit to fuweid/etcd that referenced this pull request Apr 26, 2023
The schwag was introduced to generate swagger with authorization support
[1](1) in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2](2). After several years, I
think it's good to use upstream protoc supports.

NOTE:

The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:

```bash
latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
curl -s "${latest_commit}"  | jq -S . > /tmp/old.json
cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
diff /tmp/old.json /tmp/new.json
```

```diff
1525a1526
>       "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n    Foo foo = ...;\n    Any any;\n    any.PackFrom(foo);\n    ...\n    if (any.UnpackTo(&foo)) {\n      ...\n    }\n\nExample 2: Pack and unpack a message in Java.\n\n    Foo foo = ...;\n    Any any = Any.pack(foo);\n    ...\n    if (any.is(Foo.class)) {\n      foo = any.unpack(Foo.class);\n    }\n\n Example 3: Pack and unpack a message in Python.\n\n    foo = Foo(...)\n    any = Any()\n    any.Pack(foo)\n    ...\n    if any.Is(Foo.DESCRIPTOR):\n      any.Unpack(foo)\n      ...\n\n Example 4: Pack and unpack a message in Go\n\n     foo := &pb.Foo{...}\n     any, err := ptypes.MarshalAny(foo)\n     ...\n     foo := &pb.Foo{}\n     if err := ptypes.UnmarshalAny(any, foo); err != nil {\n       ...\n     }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n    package google.profile;\n    message Person {\n      string first_name = 1;\n      string last_name = 2;\n    }\n\n    {\n      \"@type\": \"type.googleapis.com/google.profile.Person\",\n      \"firstName\": <string>,\n      \"lastName\": <string>\n    }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n    {\n      \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n      \"value\": \"1.212s\"\n    }",
1527a1529
>           "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n  value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n  URL, or have them precompiled into a binary to avoid any\n  lookup. Therefore, binary compatibility needs to be preserved\n  on changes to types. (Use versioned type names to manage\n  breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
1530a1533
>           "description": "Must be a valid serialized protocol buffer of the above specified type.",
```

REF:

1: <etcd-io#7999 (comment)>
2: <grpc-ecosystem/grpc-gateway#547>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid added a commit to fuweid/etcd that referenced this pull request Apr 26, 2023
The schwag was introduced to generate swagger with authorization support
[1](1) in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2](2). After several years, I
think it's good to use upstream protoc supports.

NOTE:

The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:

```bash
latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
curl -s "${latest_commit}"  | jq -S . > /tmp/old.json
cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
diff /tmp/old.json /tmp/new.json
```

```diff
--- /tmp/old.json       2023-04-26 10:58:07.142311861 +0800
+++ /tmp/new.json       2023-04-26 10:58:12.170299194 +0800
@@ -1523,11 +1523,14 @@
       "type": "object"
     },
     "protobufAny": {
+      "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n    Foo foo = ...;\n    Any any;\n    any.PackFrom(foo);\n    ...\n    if (any.UnpackTo(&foo)) {\n      ...\n    }\n\nExample 2: Pack and unpack a message in Java.\n\n    Foo foo = ...;\n    Any any = Any.pack(foo);\n    ...\n    if (any.is(Foo.class)) {\n      foo = any.unpack(Foo.class);\n    }\n\n Example 3: Pack and unpack a message in Python.\n\n    foo = Foo(...)\n    any = Any()\n    any.Pack(foo)\n    ...\n    if any.Is(Foo.DESCRIPTOR):\n      any.Unpack(foo)\n      ...\n\n Example 4: Pack and unpack a message in Go\n\n     foo := &pb.Foo{...}\n     any, err := ptypes.MarshalAny(foo)\n     ...\n     foo := &pb.Foo{}\n     if err := ptypes.UnmarshalAny(any, foo); err != nil {\n       ...\n     }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n    package google.profile;\n    message Person {\n      string first_name = 1;\n      string last_name = 2;\n    }\n\n    {\n      \"@type\": \"type.googleapis.com/google.profile.Person\",\n      \"firstName\": <string>,\n      \"lastName\": <string>\n    }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n    {\n      \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n      \"value\": \"1.212s\"\n    }",
       "properties": {
         "type_url": {
+          "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n  value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n  URL, or have them precompiled into a binary to avoid any\n  lookup. Therefore, binary compatibility needs to be preserved\n  on changes to types. (Use versioned type names to manage\n  breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
           "type": "string"
         },
         "value": {
+          "description": "Must be a valid serialized protocol buffer of the above specified type.",
           "format": "byte",
           "type": "string"
         }
```

REF:

1: <etcd-io#7999 (comment)>
2: <grpc-ecosystem/grpc-gateway#547>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid added a commit to fuweid/etcd that referenced this pull request Apr 26, 2023
The schwag was introduced to generate swagger with authorization support
[1](1) in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2](2). After several years, I
think it's good to use upstream protoc supports.

NOTE:

The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:

```bash
$ # use jq -S to sort the key
$ latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
$ curl -s "${latest_commit}"  | jq -S . > /tmp/old.json
$ cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
$ diff /tmp/old.json /tmp/new.json
```

```diff
--- /tmp/old.json       2023-04-26 10:58:07.142311861 +0800
+++ /tmp/new.json       2023-04-26 10:58:12.170299194 +0800
@@ -1523,11 +1523,14 @@
       "type": "object"
     },
     "protobufAny": {
+      "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n    Foo foo = ...;\n    Any any;\n    any.PackFrom(foo);\n    ...\n    if (any.UnpackTo(&foo)) {\n      ...\n    }\n\nExample 2: Pack and unpack a message in Java.\n\n    Foo foo = ...;\n    Any any = Any.pack(foo);\n    ...\n    if (any.is(Foo.class)) {\n      foo = any.unpack(Foo.class);\n    }\n\n Example 3: Pack and unpack a message in Python.\n\n    foo = Foo(...)\n    any = Any()\n    any.Pack(foo)\n    ...\n    if any.Is(Foo.DESCRIPTOR):\n      any.Unpack(foo)\n      ...\n\n Example 4: Pack and unpack a message in Go\n\n     foo := &pb.Foo{...}\n     any, err := ptypes.MarshalAny(foo)\n     ...\n     foo := &pb.Foo{}\n     if err := ptypes.UnmarshalAny(any, foo); err != nil {\n       ...\n     }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n    package google.profile;\n    message Person {\n      string first_name = 1;\n      string last_name = 2;\n    }\n\n    {\n      \"@type\": \"type.googleapis.com/google.profile.Person\",\n      \"firstName\": <string>,\n      \"lastName\": <string>\n    }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n    {\n      \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n      \"value\": \"1.212s\"\n    }",
       "properties": {
         "type_url": {
+          "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n  value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n  URL, or have them precompiled into a binary to avoid any\n  lookup. Therefore, binary compatibility needs to be preserved\n  on changes to types. (Use versioned type names to manage\n  breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
           "type": "string"
         },
         "value": {
+          "description": "Must be a valid serialized protocol buffer of the above specified type.",
           "format": "byte",
           "type": "string"
         }
```

REF:

1: <etcd-io#7999 (comment)>
2: <grpc-ecosystem/grpc-gateway#547>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid added a commit to fuweid/etcd that referenced this pull request Apr 26, 2023
The schwag was introduced to generate swagger with authorization support
[1][1] in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2][2]. After several years, I
think it's good to use upstream protoc supports.

NOTE:

The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:

```bash
$ # use jq -S to sort the key
$ latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
$ curl -s "${latest_commit}"  | jq -S . > /tmp/old.json
$ cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
$ diff --color -u /tmp/old.json /tmp/new.json
```

```diff
--- /tmp/old.json       2023-04-26 10:58:07.142311861 +0800
+++ /tmp/new.json       2023-04-26 10:58:12.170299194 +0800
@@ -1523,11 +1523,14 @@
       "type": "object"
     },
     "protobufAny": {
+      "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n    Foo foo = ...;\n    Any any;\n    any.PackFrom(foo);\n    ...\n    if (any.UnpackTo(&foo)) {\n      ...\n    }\n\nExample 2: Pack and unpack a message in Java.\n\n    Foo foo = ...;\n    Any any = Any.pack(foo);\n    ...\n    if (any.is(Foo.class)) {\n      foo = any.unpack(Foo.class);\n    }\n\n Example 3: Pack and unpack a message in Python.\n\n    foo = Foo(...)\n    any = Any()\n    any.Pack(foo)\n    ...\n    if any.Is(Foo.DESCRIPTOR):\n      any.Unpack(foo)\n      ...\n\n Example 4: Pack and unpack a message in Go\n\n     foo := &pb.Foo{...}\n     any, err := ptypes.MarshalAny(foo)\n     ...\n     foo := &pb.Foo{}\n     if err := ptypes.UnmarshalAny(any, foo); err != nil {\n       ...\n     }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n    package google.profile;\n    message Person {\n      string first_name = 1;\n      string last_name = 2;\n    }\n\n    {\n      \"@type\": \"type.googleapis.com/google.profile.Person\",\n      \"firstName\": <string>,\n      \"lastName\": <string>\n    }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n    {\n      \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n      \"value\": \"1.212s\"\n    }",
       "properties": {
         "type_url": {
+          "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n  value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n  URL, or have them precompiled into a binary to avoid any\n  lookup. Therefore, binary compatibility needs to be preserved\n  on changes to types. (Use versioned type names to manage\n  breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
           "type": "string"
         },
         "value": {
+          "description": "Must be a valid serialized protocol buffer of the above specified type.",
           "format": "byte",
           "type": "string"
         }
```

REF:

1: <etcd-io#7999 (comment)>
2: <grpc-ecosystem/grpc-gateway#547>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

8 participants