Raw Socket on Linux - Outbound data missing

[los93sol]

I am using a raw socket on linux to capture all traffic on a specific interface and port. The issue I'm having is that on eth0 I see only the inbound traffic, but on the lo interface I can see both inbound and outbound. Is there a trick to getting the outbound data on the eth0 interface?

            _socket = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.Tcp);
            _socket.Bind(new IPEndPoint(unicastAddress.Address, port));
            _socket.BeginReceive(_byteData, 0, _byteData.Length, SocketFlags.None, new AsyncCallback(OnReceive), null);

[karelz]

cc @wfurt

[wfurt]

https://www.linuxquestions.org/questions/programming-9/problem-sniffing-with-raw-sockets-222971/

You would need to use AF_PACKET(https://www.binarytides.com/packet-sniffer-code-in-c-using-linux-sockets-bsd-part-2/)

However that addressFamily is currently not supported. Your best option may be invoking libpcap.

[los93sol]

@wfurt Thanks for confirming this is not currently supported. Is this something that’s on the roadmap for future versions? In the meantime I ended up doing what you suggested and leveraged SharpPcap to wrap libpcap but this introduces additional dependencies that I’d prefer to avoid. I am going to take a peek at System.IO.Sockets and see if it’s possible to accomplish this with extension methods as I assume it’s probably just configuring the driver.

[wfurt]

The difficulty is that that interface is Linux specific and it would be very difficult to support it cross platforms. If anything, I could possibly see generic packet capture API in NetworInfo class but I don't know if that would pass API review board. It would certainly needs some more thinking and preparation.

Another path you can try to explore is using tcpdump. You can do something like tcpdump -w /dev/stdout and read packets from stdout (or use pipes from 2.1)

[karelz]

Generic packet capture API seems like huge overkill. Given it is rare scenario, I would prioritize mainline scenarios first.
Having external dependency for rare scenario is perfectly fine. We do not promise to implement and wrap everything in the world :)
Adding Linux-only addressFamilies may be reasonable - @wfurt please send email to our team to find out if others are open to that. Thanks!

[karelz]

We (Networking team) are in general fine with the API addition, assuming it solves the original problem - @wfurt will investigate.

[sgf]

any plan or process for this ?
i want to use raw socket in .net core on linux to create a raw ip header packets process server. 😂

[wfurt]

This should now work with 3.0 Preview or daily builds.

Note, that what we get is basic ability to create Socket with AddressFamily.Packet. If somebody wants to do anything fancy like setting BPF filter, you will need to get socket.Handle and pinvoke functions from libc or libpcap. By default, you will get all packets from all interfaces and that may or may not be what you want.
I added simple example allowing to take interface index (as int for simplicity) and bind to it.

using System;
using System.Net;
using System.Net.Sockets;
using System.Runtime.InteropServices;

namespace Capture
{
    class Program
    {
        /*
          // from linux/if_packet.h
          struct sockaddr_ll {
            Int16 family;
            Int16 protocol;
            Int32 ifindex;
            Int32 pad1;
            Int32 pad2;
            Int32 pad3;
        }
        */
        class LLEndPoint : EndPoint
        {
            private Int32   _ifIndex;

            public LLEndPoint(int interfaceIndex)
            {
                _ifIndex = interfaceIndex;
            }

            public override SocketAddress Serialize()
            {
                var a = new SocketAddress(AddressFamily.Packet, 20);
                byte[] asBytes = BitConverter.GetBytes(_ifIndex);
                a[4] = asBytes[0];
                a[5] = asBytes[1];
                a[6] = asBytes[2];
                a[7] = asBytes[3];

                return a;
            }
        }
        static void doBind(Socket s, int ifIndex)
        {
            var address = new LLEndPoint(ifIndex);

            s.Bind(address);
        }

        static void Main(string[] args)
        {
            Int16 protocol = 0x800; // IP.
            var _socket = new Socket(AddressFamily.Packet, SocketType.Raw, (ProtocolType)System.Net.IPAddress.HostToNetworkOrder(protocol));

            if (args.Length > 0)
            {
                doBind(_socket, int.Parse(args[0]));
            }

            byte[] packet = new byte[1508];

            int count = 10;
            while (count > 0)
            {
                    int packetLen = _socket.Receive(packet);
                    Console.WriteLine("got packet {0} {1} - {2}", packetLen, new IPAddress(packet.AsSpan().Slice(26, 4)), new IPAddress(packet.AsSpan().Slice(30,4)));
                    count--;
             }
        }
    }
}

[los93sol]

@wfurt Thanks for working on this. I actually just grabbed preview6 and gave it a try using the example you posted, but it seems to have the same issue I originally had where the lo interface sees both in and outbound, but on an external interface it only sees the inbound data and doesn't see any of the outbound at all.

[wfurt]

do you have sample code? And of course you need to run it as root or you would get permission denied error.

[los93sol]

I’ll put a repro together and link it here as soon as I can. It will take a little bit before I can do it though because to test it I swapped out libpcap in my existing project with the sample you posted here and ran my normal traffic tests for that project against it

[los93sol]

I put together a quick repro of this at https://github.com/los93sol/RawSocketSample
I am running this from my Windows box so I just added docker support and a docker compose project so you can easily stage remote traffic hitting eth0 and it also has a service that hits the socket server running in the same container so you can see that on lo both inbound and outbound are visible to the raw socket, but on eth0 it's just the inbound

[los93sol]

@wfurt Let me know if you want to see anything else, I just slammed that sample together pretty quickly to demo how eth0 is missing, just comment the block in PacketCapture that does the bind to the LLEndPoint and you can see it does see both inbound and outbound when the traffic is local.

[wfurt]

When socket is created, there should be no difference to libpcap/tcpdump or any direct use of the socket.

Thanks for the repro but as I run directly on Linux I did quick check with my sample code. It seems like the problem is in the bind example. If I add

   a[3] = 3;   // ETH_P_ALL
   a[10] = 4;  // PACKET_OUTGOING

I can see packets in both directions.

got packet 66 10.37.129.2 - 10.37.129.3
got packet 1458 10.37.129.2 - 10.37.129.3
got packet 66 10.37.129.3 - 10.37.129.2
got packet 114 10.37.129.2 - 10.37.129.3
got packet 66 10.37.129.3 - 10.37.129.2
got packet 430 10.37.129.3 - 10.37.129.2

it was mostly meant as guide how to use Bind() without introducing AF specific c# structure. You can always skip it and p/invoke bind from libc.

If that does not work you can use 'strace -f -e trace=network xxx' to see what the difference is between your app and tcpdump. I may not be able to get back to this for a while as I need to focus on remaining 3.0 issues.

[los93sol]

@wfurt Thank you so much! Now that I'm taking another look at the man pages for the native structs I understand much better what's happening and how this is working.

[los93sol]

In case anyone comes across this thread and is looking for examples check my repo that was posted here, I have PACKET_FANOUT and BPF filters working now

[los93sol]

Sorry to continue on a closed thread, but the discussion here seems relevant. Looking at the source it looks like .NET Core is using a system call to recvmsg under the hood when reading from the socket. For a raw socket it seems it is probably more desirable to leverage SO_RX_RING to minimize the number of system calls and allocations. Is MemoryMappedFiles intended to be useful for these types of things or am I off down the wrong path completely?

[smuellener]

Hello @los93sol your example repo looks very interesting! Question: would it also be possible to send and receive Layer 2 packets (only MAC Source, MAC Destination, EtherType and Payload) without them having to be IPv4 for example?

[wfurt]

yes, that should work @smuellener. If you look at my sample code it gets first IP address fropm offset 26 -> the buffer contains whole payload including L2 header.