GD2 Tracing: Add infrastructure to trace GD2 operations across gluster nodes.

[sseshasa]

Use opencensus plugins to trace GD2 calls across gluster nodes. Also, as part of this, add changes to trace the volume create calls across gluster nodes.

[kshlm]

Sorry I didn't bring this is up in our standup earlier today. We're discussing changes to the transaction framework #1003, which when implemented would remove the grpc based communication for transactions. So I'd prefer if you could wait a little while longer before proceeding with this. If you've already worked on it, I'll be happy to take it in, and make changes as necessary when the transaction framework is updated.

[sseshasa]

The code changes are complete. With the changes, traces are being captured across the grpc nodes and shown on the Jaeger UI. But like I mentioned in our standup, I had to bump up the version of etcd to 3.3.0 and add a couple of override clauses in the gopkg.toml file. Since override clauses are generally not acceptable, I was trying to see how best I can avoid them. If the new design is to remove the grpc based communication, then this code change would be irrelevant. I can wait until the new design is implemented and then come back to this.

[kshlm]

Grpc would still be used to send out the peer probe and detach requests. So the changes you do now are still relevant. Implementation of the new design for the transaction framework will take some time. So since you're ready with the changes for the current framework, please do submit them. This will give us a look at how tracing a transaction across the cluster would look, and would set a baseline for tracing output what we'd need to provide when the new design is implemented.

[sseshasa]

Sample screen-shot of how a trace looks like for a 3 node gluster cluster after creating a replica 3 volume.

[prashanthpai]

@sseshasa That's going to be very useful. Thanks 👍