Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Official" Wireshark dissector #3303

Closed
FlorianWolters opened this issue Jul 2, 2017 · 18 comments
Closed

"Official" Wireshark dissector #3303

FlorianWolters opened this issue Jul 2, 2017 · 18 comments

Comments

@FlorianWolters
Copy link

Note: I know this is not the best place to post this kind of request, but I do not know where to post this elsewhere. Hopefully other people are also interested in this.


I do wonder what tools the folks at Google use internally to debug and analyze the Protobuf messages transferred over the wire (if using network communication).

The defacto standard to do network analyzes is the network tool Wireshark. The functionality of Wireshark can be extended with a Protocol Dissector (i.e. a plug-in) to support a (custom) protocol.

I've researched a bit and found the following dissectors for Protobuf:

There are plenty of requests for a working and user-friendly dissector on the WWW, e.g.:

I would be very interested in some kind of official Wireshark protocol dissector with the following features:

  1. Supports both proto3 and proto2.
  2. Supports both UDP and TCP (see point 3). The port ranges for both protocols should be configurable with the GUI of Wireshark (i.e. when to activate the dissector).
  3. Supports the configuration of a custom prefix with length-prefix framing. The custom prefix is configured by:
    1. The total length (in bytes) of the custom prefix.
    2. The position of the length-prefix.
    3. The length (in bytes) of the length-prefix.
      That would allow to support TCP as well as custom protocols that define additional fields.
  4. Supports both generic decoding as well as specific decoding (with the help of provided .proto files).
    1. generic: Displays a aessage as [numeric field id:value] pairs. This does not require the .proto file(s).
    2. specific: Displays a message as [field name:value] pairs. This requires the .proto file(s) for the specific protocol. The addition of .proto file(s) should be possible via the GUI of Wireshark and the directory path to the official Protobuf .proto files should be configurable as well (e.g. $PROTOBUF_ROOT/include/google/protobuf).

At the end of the day it would be very nice to have some kind of official protocol dissector contributed to Wireshark.

Are there any efforts towards an official Protobuf dissector for Wireshark? What does the community think? Is this considered useful? Or did I overlook an already existing solution that satisfies the requirements stated above?

@xfxyjwf
Copy link
Contributor

xfxyjwf commented Jul 5, 2017

In Google, to send protobuf over the wire, you will be using the rpc library (the internal version of http://grpc.io/) instead of writing your own socket code, and a set of tools/libraries are provided to inspect and diagnose the rpc traffic. For example, with a command line tool you can easily write up a proto message in text format and send it to an arbitrary rpc server and get the response in text format. Hardly anyone needs to inspect the network traffic to understand the proto data being sent and it's also very hard to do because the rpc library will compress and encrypt the proto data. People who work on the rpc library probably need to use wireshark or tcpdump to inspect the network traffic, but again they don't need to inspect the proto data (I know they send payload of 1 byte in their tests). So pretty much nobody in Google will need this protobuf dessector for Wireshark and I doubt it will ever be officially supported...

@ruudsieb
Copy link

ruudsieb commented Jul 27, 2017

Actually after reading the OP I tried using the Length Prefixed Protocol Buffer Dissector together with the protobuf_dissector (forked). And it was pretty easy to setup and get it to work in my context which is; Wireshark on Windows, TCP connection, length prefix (little endian, for this I made a small change on the lppb), proto2 message format.

@FlorianWolters
Copy link
Author

@ruudsieb Did you get it to work with the proto3 language?

@ruudsieb
Copy link

Sorry, forgot to mention that, no, I'm using proto2 message format.

@jfjffilk
Copy link

@FlorianWolters I am also trying to write a wireshark plugin in c to parse google protocol buffers over TCP, but not finding proper source to know how. Have you found any solution for it?

@FlorianWolters
Copy link
Author

@jfjffilk No, as mentioned in the OP there is currently no out-of-the-box solution working for the proto3 language. For general information how-to implement a Wireshark dissector refer to Chapter 9. Packet dissection of the official Wireshark documentation. But I strongly recommend to contact the author(s) of the Wireshark Protobuf Dissector and to analyse/fork/extend that project before beginning from scratch and implementing your own dissector. Please read the OP again for problems related to the existing Protobuf dissectors.

@ndandanov
Copy link

I would also be interested in an official Wireshark dissector for protobuf and perhaps gRPC.

I agree that it may not be of much use in Google's environment, however I think that many developers and testers would benefit from it.

@ndandanov
Copy link

ndandanov commented Nov 30, 2017

Sorry for double-posting, but I tried a few things after posting above.

I found out, that in the official Wireshark repo on GitHub, there is a gRPC dissector - namely the wireshark/epan/dissectors/packet-grpc.c located here: https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-grpc.c

It seems that it was officially added in the end of September this year:
https://github.com/wireshark/wireshark/commits/master/epan/dissectors/packet-grpc.c

Note that the original Wireshark repo is located on their website. Please check https://www.wireshark.org/develop.html.

I opted to see how this dissector is working and cloned the Wireshark repo:

git clone https://code.wireshark.org/review/wireshark

My system is a 64-bit Arch Linux, so I built Wireshark with the following commands:

./autogen.sh 
./configure 
make

(using this as a reference: https://www.wireshark.org/docs/wsdg_html_chunked/ChSrcBuildFirstTime.html)

It took around 20-30 minutes to build. The Wireshark version is:
screenshot_2017-11-30_17-09-00

I then ran it as root (in order to be able to capture any traffic) and started capturing the loopback interface.

I cloned, built and started the official gRPC hello world server and client from the grpc-java repo:
https://github.com/grpc/grpc-java/tree/master/examples

The console outputs are, as expected:
screenshot_2017-11-30_16-38-16

The captured packets in Wireshark are:
screenshot_2017-11-30_16-39-16

You can download the .pcapng file from my repo:
https://github.com/ndandanov/wireshark-grpc/blob/master/grpc-hello-world.pcapng

Here you can see that they are decoded as TCP packets.

I clicked with the right mouse button on a packet and selected Decode As...:
screenshot_2017-11-30_16-39-39-1

And configured the packets as HTTP2:
screenshot_2017-11-30_16-40-24

And the packets are now correctly recognized as either HTTP2 or gRPC:
screenshot_2017-11-30_16-40-37

Let's take a look at the packet which carries the actual gRPC hello world request:
screenshot_2017-11-30_16-45-49

I had also opened the .proto file (located here https://github.com/grpc/grpc-java/blob/master/examples/src/main/proto/helloworld.proto) for reference.

The actual value of field 1 from the message HelloRequest, which is string name, is "world" (or 77 6f 72 6c 64 in hex):

screenshot_2017-11-30_16-49-54

And we can verify this:
screenshot_2017-11-30_16-49-39

The gRPC hello world response was of course "Hello world":
screenshot_2017-11-30_16-50-18

The Wireshark interpretation of fields and their values could definitely be extended. I did not come across any option to specify and input .proto file to extract the field names and correct value types. This leads to some doubles represented as just uint64 in Wireshark.

Nevertheless, I think this is a significant improvement in the right direction.

Please share your opinion on this dissector. Has anyone else tried it and published some results/improvements?
Is there any public information regarding the release of this dissector?
Kind regards!

@xfxyjwf
Copy link
Contributor

xfxyjwf commented Nov 30, 2017

@ndandanov Great findings! Could you also post this to grpc github repo? I think lots of grpc users will find this useful.

@ndandanov
Copy link

Hi @xfxyjwf, thank you for the kind words!
I added the above to the following issue: grpc/grpc#13586

I also recorded two videos which show the process in depth. They are somewhat lengthy and unfortunately I could not replicate the same successful packet dissection - probably due to the slightly newer development version of Wireshark.

Nevertheless, they could be of help to somebody:
gRPC Wireshark dissector - Part 1: https://youtu.be/LoGArC58Row
gRPC Wireshark dissector - Part 2: https://youtu.be/Wj9VoJQf-qc

@abhinava
Copy link

@ndandanov This is really great! Do you happen to have any packet captures (PCAP files)?

@abhinava
Copy link

@ghost
Copy link

ghost commented Jun 11, 2019

Thank you for all the posts here. Posting them at an issue doesn't seem like the best place though. I'd recommend creating a page somewhere, and add a link to that page at https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md. You would probably want to reach out to gRPC and figure out where to best post it there too. Thanks!

@ghost ghost closed this as completed Jun 11, 2019
@huangqiangxiong
Copy link

Hope following information might be useful:

New features about Protobuf and gRPC dissectors have been added into Wireshark since version 3.2.0:

  1. Protobuf files (*.proto) can now be configured to enable more precise parsing of serialized Protobuf data (such as gRPC).
  2. The message of stream gRPC method can now be parsed with supporting of HTTP2 streaming mode reassembly feature.
  3. User can specify protobuf search paths (where has *.proto files), and the UDP ports to protobuf message type maps at the Protobuf protocol preferences.
  4. If your own dissectors need invoke protobuf dissector, you can pass the message type to Protobuf dissector by data parameter (in C) or pinfo->private_table["pb_msg_type"] (pinfo.private["pb_msg_type"] in lua).

Another two new features will be released in 3.3.0 or 3.4.0:

  1. Protobuf fields can be dissected as wireshark (header) fields that allows user input the full names of Protobuf fields or messages in Filter toolbar for searching.
  2. Dissector based on Protobuf can register itself to a new 'protobuf_field' dissector table, which is keyed with the full names of fields, for further parsing fields of BYETS or STRING type.

Preferences userguide:
https://www.wireshark.org/docs/wsug_html_chunked/ChProtobufSearchPaths.html
https://www.wireshark.org/docs/wsug_html_chunked/ChProtobufUDPMessageTypes.html

@jtattermusch
Copy link
Contributor

@huangqiangxiong would you be interested in writing a guest blogpost for https://grpc.io/blog/ about how to use wireshark with grpc and what are some of the new features?

@huangqiangxiong
Copy link

huangqiangxiong commented Mar 11, 2020 via email

@jtattermusch
Copy link
Contributor

Btw a blogpost is created by opening a PR (e.g. grpc/grpc.io#121).

huangqiangxiong added a commit to huangqiangxiong/grpc.io that referenced this issue Oct 20, 2020
huangqiangxiong added a commit to huangqiangxiong/grpc.io that referenced this issue Nov 28, 2020
chalin pushed a commit to huangqiangxiong/grpc.io that referenced this issue Dec 4, 2020
chalin pushed a commit to huangqiangxiong/grpc.io that referenced this issue Dec 5, 2020
chalin added a commit to grpc/grpc.io that referenced this issue Dec 5, 2020
…#467)

* Blogpost about how to use wireshark with protobuf and grpc dissectors

Add this blogpost as recommended on
protocolbuffers/protobuf#3303

* Update content according to last comments

1. change title to "How to analyze gRPC with Wireshark"
2. add link to github id (after the name attribute).
3. add "Protocol Buffers" for "Protobuf" in first sentence.
4. add summary info about the intent and scope of this blog.
5. move the history to a nearly end section.
6. add a short section about how to capture gRPC traffic.
7. change "official *.proto library of Protobuf" to "Protocol Buffers Well-Known Types"
8. remove the section about JSON-serialized capture.
9. remove the sections about build-in UDP and custom dissectors, instead leaving a link to wireshark protobuf wiki page.
10. remove sample link of Protobuf only capture from referrence section.

* Update how-to-analyze-grpc-with-wireshark.md

Updated according to comments.
1. Update date.
2. Fixed some link errors.

* PR #467 copyedits, spelling and text wrap 2020-11-27

* PR #467 correct clerical and grammar errors 2020-11-28

also update date

* Mark as draft

Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants