Design challenge: ethernet mirror

[lukego]

Challenge: How do you design an "ethernet mirror" application that receives packets on a port and retransmits exactly the same packets back? The design should be optimized for high packet rates.

Intel 10G

This should be fairly easy on the Intel 82599. Just create one intel_app and wire the transmit and receive links together. Done.

ConnectX-4 100G

This seems to become tricky on the ConnectX-4 when we consider the performance data on #1007. Suppose that we need to use 4 pairs of TX+RX queues in order to optimize the packet rate. Then the challenge is how to ensure that we send packets back in the same order that they were received.

Sub-problems:

• Received packets need to be spread across the receive queues. RSS/hashing is not optimal because it will only work for specific traffic mixes. May require something more like round-robin packet dispatch. (I am not sure if this is available on ConnectX-4.)
• Received packets need to be put into order. May need to examine precise hardware timestamps in order to work out the order in which packets receive on different queues should be transmitted. The 100G ConnectX-4 seems to be able to timestamp recieved packets using a 156 MHz clock (would seem sufficient for minimum one tick per packet).
• Transmitted packets need to be put into the order that was determined for receive. The packets are being sent on separate send queues and so we would need to somehow control the order in which the NIC processes packets from these queues.

Fun challenge! I am not sure if this would be easy or impossible on ConnectX-4. The first place to look for answers is the PRM. I would be very interested if anybody has a proposed (partial) solution.