Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop sniff() asynchronously #989

Closed
HenningCash opened this issue Dec 15, 2017 · 20 comments
Closed

Stop sniff() asynchronously #989

HenningCash opened this issue Dec 15, 2017 · 20 comments

Comments

@HenningCash
Copy link
Contributor

Hi there,
we are using scapy to built something similar to tcpdump.
The capturing process runs in its own thread:

def run(self):
    try:
        sniff(iface=self.interface, filter=self.filter, prn=self._loop_callback, store=0)
    except BaseException as e:
        log.exception(e)
    finally:
        self.output.close()

def _loop_callback(self, pkg: Packet):
    if self._stop_flag is True:
        raise KeyboardInterrupt
    else:
        self.output.write(pkg)

Basically we have a flag that is set externally from the main thread if the capturing process should stop. This works as long as a package is received after the stop flag is set. But if there is no package the thread (recv()) hangs until the next package is received and the callback is fired.

We could built something like this by passing the timeout to sniff():

    while not self._stop_flag:
        sniff(iface=self.interface, filter=self.filter, prn=self._loop_callback, store=0, timeout=1)

But I think with this approach there might be packages that get lost between the sniffing calls, am I right?

Are there any other approaches to stop sniff() async without losing packages?

@gpotter2
Copy link
Member

gpotter2 commented Dec 16, 2017

This is actually quite easy to do, using the stop_filter function:

image

Obviously, this will have to wait for a new packet to be sniffed (otherwise recieving on the socket is stuck)

https://github.com/secdev/scapy/blob/master/scapy/sendrecv.py#L622-L625

stop_filter: Python function applied to each packet to determine if
we have to stop the capture after this packet.
Ex: stop_filter = lambda x: x.haslayer(TCP)

@HenningCash
Copy link
Contributor Author

Your example works, if there is a packet. As the documentation says stop_filter is a "Python function applied to each packet". If there is no packet (e.g. if you use a very restrictive filter and/or there is very little traffic) the process/thread will wait and exit only after the next captured packet when stop_filter is applied.

#!/usr/bin/env python3

import time, threading
from scapy.all import sniff
e = threading.Event()
def _sniff(e):
    a = sniff(filter="tcp port 80", stop_filter=lambda p: e.is_set())
    print("Stopped after %i packets" % len(a))

print("Start capturing thread")
t = threading.Thread(target=_sniff, args=(e,))
t.start()

time.sleep(3)
print("Try to shutdown capturing...")
e.set()

# This will run until you send a HTTP request somewhere
# There is no way to exit clean if no package is received
while True:
	t.join(2)
	if t.is_alive():
		print("Thread is still running...")
	else:
		break

print("Shutdown complete!")

bildschirmfoto 2017-12-18 um 14 06 49

@gpotter2
Copy link
Member

gpotter2 commented Dec 18, 2017

Well we could add the stop_event check when every packet is recieved, but this would actually slow down scapy :/
Would this be a proper option to you ?

I don't think they would be a proper way we could internally abort sniffing if no packets are recieved, as we rely on the select (unix) or winpcap (windows) calls, which we cannot abort...

@guedou
Copy link
Member

guedou commented Dec 18, 2017

I will suggest that you build your own packet capture logic using Scapy SuperSocket and select, such as:

from scapy.all import *
import select

s = conf.L2listen()
rlist = select.select([s], [], [])
if rlist:
    p = s.recv()
    print p.summary()
s.close()

@HenningCash
Copy link
Contributor Author

@guedou that looks promising. I will give it a try within the next week.

Do you think its worth the effort implementing this as new feature for scapy? Passing in a threading.Event and checking for isSet() should be simple

@guedou
Copy link
Member

guedou commented Dec 19, 2017 via email

@guedou
Copy link
Member

guedou commented Dec 19, 2017

Feel free to reopen if needed.

@guedou guedou closed this as completed Dec 19, 2017
@HenningCash
Copy link
Contributor Author

The suggested custom implementation with SuperSocket worked perfectly fine, thank you!

@SkypLabs
Copy link

SkypLabs commented Mar 2, 2018

I just published a blog post about a way to address this issue: http://blog.skyplabs.net/2018/03/01/python-sniffing-inside-a-thread-with-scapy/ . I hope it will help you.

@guedou
Copy link
Member

guedou commented Mar 2, 2018 via email

@louisabraham
Copy link

@SkypLabs Your post is about stop_filter.

A little observation: one could also use stop_callback which is called more often.

Apart from that, I wrote a custom version of sniff to address this issue optimally (using select). Feel free to use it.
It should also be possible to integrate it in the code, but I don't know if it is a common issue.

def sniff(store=False, prn=None, lfilter=None,
          stop_event=None, refresh=.1, *args, **kwargs):
    """Sniff packets
sniff([count=0,] [prn=None,] [store=1,] [offline=None,] [lfilter=None,] + L2ListenSocket args)

  store: wether to store sniffed packets or discard them
    prn: function to apply to each packet. If something is returned,
         it is displayed. Ex:
         ex: prn = lambda x: x.summary()
lfilter: python function applied to each packet to determine
         if further action may be done
         ex: lfilter = lambda x: x.haslayer(Padding)
stop_event: Event that stops the function when set
refresh: check stop_event.set() every refresh seconds
    """
    s = conf.L2listen(type=ETH_P_ALL, *args, **kwargs)
    remain = None
    lst = []
    try:
        while True:
            if stop_event and stop_event.is_set():
                break
            sel = select([s], [], [], refresh)
            if s in sel[0]:
                p = s.recv(MTU)
                if p is None:
                    break
                if lfilter and not lfilter(p):
                    continue
                if store:
                    lst.append(p)
                c += 1
                if prn:
                    r = prn(p)
                    if r is not None:
                        print(r)
    except KeyboardInterrupt:
        pass
    finally:
        s.close()

    return plist.PacketList(lst, "Sniffed")

@SkypLabs
Copy link

@SkypLabs Your post is about stop_filter.

No, using stop_filter was a design choice for "letting the time to the sniff function to terminate its job by itself, after which the sniffing thread will be force-stopped and its socket properly closed from the main thread.". My post is not about stop_filter but about a way to stop the sniffing loop programmatically at any time.

A little observation: one could also use stop_callback which is called more often.

stop_callback is not part of Scapy but has been added to the fork scapy3k.

Apart from that, I wrote a custom version of sniff to address this issue optimally (using select). Feel free to use it.

As I have already said in this issue, "using a SuperSocket with a timeout works fine but this method is resource-consuming since you have to check every X seconds if the sniffing loop should be stopped or not."

@louisabraham
Copy link

louisabraham commented Apr 17, 2018

@SkypLabs maybe I missed something, but I think your method doesn't actually quit the sniffer. It works only if you have a lot of packets. Try it with an impossible filter (like 'tcp port 12321'), and you will see it doesn't stop.

My bad for stop_callback :)

I don't think one check every second is a major performance issue. The most efficient way would maybe to add a dumb pipe between threads to the select, but I don't think it is worth it (after all we code in Python so performance isn't our main concern)

@SkypLabs
Copy link

@SkypLabs maybe I missed something, but I think your method doesn't actually quit the sniffer. It works only if you have a lot of packets. Try it with an impossible filter (like 'tcp port 12321'), and you will see it doesn't stop.

Yes it does, even when there is no traffic (even with tcp port 12321). I wouldn't have posted an article about a solution to this issue if I wasn't sure about it. I'm not claiming it's the best solution ever though, just that it works well.

My bad for stop_callback :)

No worries :)

I don't think one check every second is a major performance issue. The most efficient way would maybe to add a dumb socket between threads to the select, but I don't think it is worth it (after all we code in Python so performance isn't our main concern)

Well, I suppose we can debate about it for a long. I just prefer not to trigger a check every second :)

@louisabraham
Copy link

Oh, I think I understand now, but I still don't agree.

I think that in your solution, the event doesn't do anything if the select doesn't receive any packet (and I think you agree). So it's just nice in most cases but does not solve anything in general.

I tested, and what your code does is closing a socket while select listens in a separate thread.
On macOS 10.12.6, it causes OSError: [Errno 9] Bad file descriptor.

This is in fact undefined behavior, see:

The best solution would really be to use a pipe.

@SkypLabs
Copy link

I think that in your solution, the event doesn't do anything if the select doesn't receive any packet (and I think you agree).

Yes, the event is just here to tell the sniffing loop to stop but won't have any effect if the select function is blocked, waiting for new packets, hence the timeout before force-stopping the sniffing loop.

I tested, and what your code does is closing a socket while select listens in a separate thread.

Actually, my code force-stops the sniffing loop and then closes the socket. There is no race between the two threads.

@robgom
Copy link

robgom commented Jun 22, 2018

Can it be reopened?
We observe the problem on Windows.
With scapy 2.3.1 we distributed local version of scapy, which used the following pattern:

p = None
while stop_filter and stop_filter(p):
... the loop as normal

On Windows it works, because there's PcapTimeout exception raised in case of no packets, which allows the condition to be checked.
It breaks backward compatibility, because it requires stop_filter to be aware of possible "None" as packet.

What about the mentioned stop_sniff additional parameter to sniff as alternative? On Windows it would work, as indicated before, even if stop wouldn't occur immediately. I don't know about other platforms, though.

What's your opinion?

Regards,
Robert

@gpotter2
Copy link
Member

gpotter2 commented Jun 8, 2019

Implemented AsyncSniffer in #1999. Feel free to have a look.

To answer the long unanswered questions:

  • The code provided SkypLabs would indeed freeze on select() on non-windows platforms. It was however a pretty good workaround before Sendrecv overhaul: async sniffing & major cleanup #1999, except when no packets would be received.
  • stop_callback adds unnecessary slowness, doesn't fix the issue without a timeout enabled.
  • On Windows it works, because there's PcapTimeout exception raised in case of no packets.

Correct. PcapTimeout behavior is going to be removed anyway.

  • Actually, my code force-stops the sniffing loop and then closes the socket. There is no race between the two threads.

This makes no sense. You can't cancel a select() call. In the best case, you could send a packet to yourself and make the socket receive it, but that won't work on some special sockets which don't support loopback.

I'll be locking this thread in a few days, as it's still the footer of a closed issue.
To anyone reading this, feel free to test #1999 out. Should be merged soon

@SkypLabs
Copy link

This makes no sense. You can't cancel a select() call. In the best case, you could send a packet to yourself and make the socket receive it, but that won't work on some special sockets which don't support loopback.

Yes you're right. I just realised my mistake. I've spoken too fast.

Thank you @gpotter2 for your great work on #1999 by the way. I'm glad to see a permanent solution being integrated directly into Scapy.

@porceCodes
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants