Packet viewer

[akorb]

@TabeaSpahn
@polybassa

We would be interested in the opinions of the maintainers.
It took some time but we think it's finally stable enough to be shown in a PR.

Python 2 and Python 3 supported.

Things that probably have to be changed:

• Merge tests
• Add urwid as an optional dependency to scapy and maybe output a nice error message if the packet viewer is imported even though urwid is not installed
• Add urwid to Travis server
• Fix flake8 errors
• Remove debugging scripts
• Improve documentation (for example add a screenshot of the viewer)
• Show time relative to first packet

Requirements:
pip3 install urwid

Cool feature I want to highlight:
It's a TUI working fully over ssh which means also mouse interaction is possible over ssh.

[codecov]

Codecov Report

Merging #2603 into master will decrease coverage by 0.50%.
The diff coverage is 44.14%.

@@            Coverage Diff             @@
##           master    #2603      +/-   ##
==========================================
- Coverage   88.20%   87.69%   -0.51%     
==========================================
  Files         254      266      +12     
  Lines       54349    55006     +657     
==========================================
+ Hits        47936    48235     +299     
- Misses       6413     6771     +358     

Impacted Files	Coverage Δ
scapy/modules/packet_viewer/pop_ups.py	17.39% <17.39%> (ø)
scapy/modules/packet_viewer/extended_edit.py	26.92% <26.92%> (ø)
scapy/modules/packet_viewer/edit_view.py	28.00% <28.00%> (ø)
scapy/modules/packet_viewer/viewer.py	30.95% <30.95%> (ø)
scapy/modules/packet_viewer/extended_listbox.py	33.33% <33.33%> (ø)
scapy/modules/packet_viewer/main_window.py	38.59% <38.59%> (ø)
scapy/modules/packet_viewer/packet_list_view.py	38.70% <38.70%> (ø)
scapy/modules/packet_viewer/__init__.py	60.00% <60.00%> (ø)
scapy/modules/packet_viewer/row_formatter.py	71.23% <71.23%> (ø)
scapy/modules/packet_viewer/details_view.py	81.81% <81.81%> (ø)
... and 19 more

[polybassa]

Some infos for DetailsView
hex_text should be aligned left self.hex_text = Text("", align="left")
The BlackAndWhite theme should be used for hexdump:

        cf = conf.color_theme
        conf.color_theme = BlackAndWhite()
        self.detail_text.set_text(packet.show(dump=True))
        self.hex_text.set_text(hexdump(packet, dump=True))
        conf.color_theme = cf

The DetailsView should be scroll-able and detail_text and hex_text should have fixed areas

[polybassa]

Actually, the BlackAndWhite theme should be applied in viewer.py, already:

    cf = conf.color_theme
    conf.color_theme = BlackAndWhite()
    loop = MainLoop(main_window, palette)
    main_window.base_widget.main_loop = loop
    main_window.base_widget.packet_view.main_loop = loop
    loop.run()
    conf.color_theme = cf

This solves some presentation issues for me.

[akorb]

For a first impression of how the viewer looks

[gpotter2]

Too bad it relies on platform specific dependencies (urwid isn't available on Windows, I couldn't test it), but it looks really dope. 👍

However I'm wondering if this shouldn't be a different project from scapy alltogether (I'm open to discussion, don't know what the other maintainers think). Isn't it a bit.. out of space?
IMHO Scapy should focus on being the main framework/library more than anything else. I wonder if it doesn't get confusing if we start shipping more "tools" (as in, external scripts not accessible through an API, hidden in a subfolder). This PR is very cool, but I'm not sure if people are downloading scapy to use it as a GUI. We can already pipe sniffing to Wireshark, and I personally think that's enough for most people.

The "tools" folder was rather more for scapy development than a place to host people's script. I'm stretching it a bit here: this PR is really cool, but it's a general thought.

What do you think?

[polybassa]

@gpotter2 Thanks for opening this discussion. I will share some of my opinions, because I'm somehow involved in this PR.

1. The tools folder
   I can truly understand your concerns, that more and more stuff (especially the automotive tools) blow up the framework and the tools folder. A general strategy for Scapy would be necessary at this point. Should Scapy only be a framework for packet crafting? What's ok to be in, whats not?
   Is it ok, to have some useful tools present in the main repository?
   In my case, since most of the tools are still growing, I'm very happy, to have them mainline, since there is not a fixed release version, yet. So it would be hard to create a independent repository, only for some tools. Also, I don't want to fork and maintain a new fork, since often changes I encounter would also help the main tool. Syncing that would create unnecessary overhead, in my opinion.

2. The PacketViewer
   In the beginning, we made this viewer only for some automotive layer communication, like CAN. I personally thought, a lightweight and flexible way to view packets from a socket or a PacketList would be a good feature to Scapy. (This questions again point 1. Framework only, or open for tools?) We got inspired by MITM-Proxy and thought that such a addition would be valuable. Therefore we decided to focus on a very minimal implementation, which is also compatible with any other Packet types in Scapy. Our strategy here is to make layer specific configurations for the representation of a packet (A example is in this PR in isotp.py). A simple call viewer(somePacketList, basecls=TCP) opens the PacketViewer and allows inspection, editing, sorting or filtering of Packets. In combination with a socket, also interactive reply of packets might be possible (again inspired from MITM Proxy). Since Scapy is so generic, our goal is, to provide this kind of features to all layers and contributions in Scapy. The lack of windows support for urwid is a bummer, which needs a solution.

What do you think?

[akorb]

I just did some testing for urwid+scapy on Windows.

cygwin

urwid works, but scapy does not:

WSL 1

urwid also works and so does scapy:

(WSL 1 + Ubuntu)

From the weird question marks in each line mitmproxy suffered as well. They fixed it with this:
https://github.com/mitmproxy/mitmproxy/blob/8dfb8a9a7471e00b0f723de66d3b63bd089e9802/mitmproxy/tools/console/window.py#L316-L323

But while scapy itself works fine, there are problems with the sockets.

It works fine with virtual PythonCAN since everything happens in userspace and is virtual.
But Ethernet-related sockets use actual linux RAW sockets.
And WSL 1 does not support RAW sockets.
microsoft/WSL#2538

Traceback (most recent call last):
  File "scapy/start_packet_viewer.py", line 15, in <module>
    socket = L2Socket("eth0")
  File "/home/andreas/scapy/scapy/arch/linux.py", line 448, in __init__
    self.ins = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(type))  # noqa: E501
  File "/usr/lib/python3.8/socket.py", line 231, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 97] Address family not supported by protocol

Working with TCP works because this does not require RAW Sockets. Tested with following code:

from scapy.supersocket import StreamSocket
from scapy.packet import Raw
from scapy.tools.packet_viewer.viewer import viewer

import socket
s = socket.socket()
ss = StreamSocket(s, Raw)

def start():
    s.connect(("example.org", 80))
    ss.send(Raw("GET /\r\n"))

viewer(ss, started_callback=start)
ss.close()

Since WSL is intended to implement Linux userspace, not Linux kernelspace, the modules for CAN and ISOTP are not available:

andreas@DESKTOP-0JLP9RQ:~$ sudo modprobe can
modprobe: FATAL: Module can not found in directory /lib/modules/4.4.0-18362-Microsoft
andreas@DESKTOP-0JLP9RQ:~$ sudo modprobe vcan
modprobe: FATAL: Module vcan not found in directory /lib/modules/4.4.0-18362-Microsoft
andreas@DESKTOP-0JLP9RQ:~$ sudo modprobe peak_pci
modprobe: FATAL: Module peak_pci not found in directory /lib/modules/4.4.0-18362-Microsoft

This situation will improve with WSL 2 which will probably support RAW sockets and a more complete linux kernel.
But on the WSL 2 FAQ Microsoft also states:

In initial releases of WSL 2 hardware access support will be limited, e.g: you will be unable to access the GPU, serial or USB devices. However, adding better device support is high on our backlog, [...]

Thus we can't be sure what will be possible with CAN stuff on WSL 2 but I'm pretty sure at least the vcan will work.

Conclusion

I see two possibilities what we can do.

1. No native Windows support and only restricted on WSL 1 (might get better with WSL 2)
2. Change from urwid to another user interface library

This was a lot of information, but I wanted it to be documented.

[gpotter2]

I now have WSL2 and was able to test it. Some thoughts:

• it's really cool. I think this could move to the modules folder, and you can add this to the list of commands by using the command decorator (see config.py).
• There should be a help command or something similar. I found about d by randomly pressing keys 😄
• Does it work with normal sockets ? Your example works but using

a = conf.L2socket()
viewer(a)

it gives no result. Am i doing something wrong?

• maybe it would be cool to have a way to export as pcap from within the GUI. maybe it's easier to just use the results and pass it to wpcap
• You should use a scapy.plist object to return the results.
• lots of funky characters on Windows ([?]). This is likely an issue with my shell (Windows Terminal)

[polybassa]

@gpotter2 At the moment, there is a basecls filtering implemented. Try:

a = conf.L2socket()
viewer(a, basecls=Ether)

[akorb]

@gpotter2
@polybassa is right. In the moment this is required because sometimes there are packets which are not of type Ether but Dot3 and this would lead to a crash since this type has different columns.

>>> a = conf.L2socket()
>>> sniff(opened_socket=a, prn=type)                                                                                                                                                                                                         
<class 'scapy.layers.l2.Ether'>
<class 'scapy.layers.l2.Dot3'>
<class 'scapy.layers.l2.Dot3'>
<class 'scapy.layers.l2.Ether'>
<class 'scapy.layers.l2.Ether'>

Thanks for your great feedback!
I just pushed two commits.
A PacketList is returned now and I also applied a fix for the [?] problem. Could you please verify if it now works properly on WSL2? If not, could you please send me the output of:

import platform
platform.platform()

I'm glad you like it 😄

[gpotter2]

Is there a way to "full screen" the packet infos? It's pretty cropped

[polybassa]

Wieso hier nicht OrderedDict lassen? Jetzt gibt man hier nur Kontrolle ab und der Code wird dadurch komplizierter. Auch macht das so hauptsächlich für Fxx keys Sinn. Sobald man einen anderen Key hat, wird es ziemlich zufällig, ob es gut wird oder nicht.

I didn't removed the OrderedDict.

[polybassa]

Dear maintainers, could you please have a look at the packet viewer in the current version? We will do unit tests and minor refactorings, but the we want to fix the features.

load_module("packet_viewer.viewer")
s = L3PacketSocket("wlan0")
a = viewer(s)

[akorb]

The viewer is more robust and complete now.
I also added some stuff to the documentation. As long as I don't find any new critical issues it's ready from my side.

[akorb]

Is there a way to "full screen" the packet infos? It's pretty cropped

There is now. Edit (F7) can be toggled.
hidden -> small -> big -> hidden -> ...

[polybassa]

Since 2.4.4 is out now, can we merge this PR to either contrib/ or modules/? The major code affects only one directory. So we could test this packet_viewer until the next major release. If we encounter to much problems and issues, we could just remove it.

[akorb]

Codacy/PR Quality Review — Not up to standards. This pull request quality could be better.

We're using eval for the creation of packets and the filter.

For the creation of packets we validate the input and shrink its execution context to the only necessary requirements.
https://github.com/akorb/scapy/blob/packet_viewer/scapy/modules/packet_viewer/main_window.py#L289

Since the filter is easier, there we only shrinked the execution context.
https://github.com/akorb/scapy/blob/packet_viewer/scapy/modules/packet_viewer/main_window.py#L343

Question to the mantainers, is this sufficient?

[akorb]

I added some development documentation to Scapy -> Development -> Packet Viewer.

Is this the correct place to put it? @guedou

[polybassa]

Dear maintainers, is there any possibility to move forward with this PR?

[polybassa]

Hi, we moved to a stand-alone project for now. If this becomes stable enough, maybe we can merge it one day into mainline. https://pypi.org/project/scapy-packet-viewer/
There is still the possibility to have this under github.com/secdev/ to have it more related to Scapy

[guedou]

You should consider adding a reference to this code into https://github.com/secdev/awesome-scapy. You can add a visualization section before the misc one.