[grpc] java-client -> go-server interop test fail

[HailongWen]

The response by go-server does not contain tag "method".

SEVERE: Client received wrong TagContext. Got [Tag{key=TagKey{name=operation}, value=TagValue{asString=interop-test}}, Tag{key=TagKey{name=project}, value=TagValue{asString=open-census}}], want [Tag{key=TagKey{name=method}, value=TagValue{asString=interop.EchoService/echo}}, Tag{key=TagKey{name=operation}, value=TagValue{asString=interop-test}}, Tag{key=TagKey{name=project}, value=TagValue{asString=open-census}}].

With @songy23 's help, I found the behavior change is due to census-instrumentation/opencensus-go#566. Which behavior is correct? Should the "method" be propagated or not? If not, we should probably change the java code.

[songy23]

FYI the method tag will be automatically propagated in gRPC Java: https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/CensusStatsModule.java#L332. While in Go, it's no longer propagated after census-instrumentation/opencensus-go#566.

/cc @dinooliva @sebright @rakyll @Ramonza @bogdandrutu

[songy23]

IIUC C++ also propagates method tag: https://github.com/census-instrumentation/opencensus-cpp/blob/master/opencensus/plugins/grpc/internal/server_filter.cc#L114.

[sebright]

I think gRPC should record stats with the method tag but not propagate it, since the server already knows what method was called and can add the tag back.

[semistrict]

We deliberately stopped propagating it because it doesn't really make sense outside the scope of the original request.

[songy23]

Make sense. I think we need to specify this in gRPC propagation spec (census-instrumentation/opencensus-specs#66) if not propagating method tag is the desired bebavior.

[dinooliva]

The grpc client doesn't need to propagate the method tag but the grpc server does need to add it to the context where the rpc is being processed so there should be a method tag in scope on the server side.

Also, if the grpc server subsequently makes a call as a grpc client, then the method tag will be propagated since it is in scope (I don't believe gRPC has any logic for explicitly removing the method tag).