This doc is for brainstorming. Watch out for likely duplicates, clean them up progressively.
Turn these notes into real issue or bug descriptions and move them into other docs like xdp_issues.org
The ip utility (from iproute2) does not support specifying, that an XDP prog need to sue generic-XDP.
I need this when testing veth, which now got native-XDP.
Ref slides: http://people.netfilter.org/hawk/presentations/NetConf2017_Seoul/XDP_devel_update_NetConf2017_Seoul.pdf
Idea slide 31: On XDP-redirect packets can be dropped, after the XDP prog have run, and today we have a ‘drop-tracepoint’ to handle/debug this. It would be more interesting to call a new XDP-type program that get the about-to-be-dropped packet, and allow it to take a new decision. Wondering what this can be used for, click the youtube link in the slide, and blow-your-mind! (https://youtu.be/BO0QhaxBRr0)
Want to track when BPF progs gets loaded/unloaded
- Both for system audit and troubleshooting purposes
BPF code do have LSM/SElinux hooks
- LSM maintainers request, use Audit subsystem instead
- Jiri Olsa currently working on this
E.g. AF_XDP via xdpsock uses redirect, and there is not feedback when running prog on driver without redirect-support. Need to look in dmesg to see big splast.
E.g. Loading XDP/ebpf prog with HW offload on Netronome NIC, but wrong firmware, just sats EINVAL. No help or hints that firmware needs to be upgraded.
Same EINVAL when loading on a NIC driver other than Netronome.
ixgbe and i40e choose to violate one-page-per-packet
Two reasons: #1 - extend frame tail, for DNS/other reply use-cases
- NDIV needs this feature to convert to XDP
#2 - simplify XDP convert to SKB (else truesize is a lie)
Keeping DMA mapping in XDP return API.
Ref slides: http://people.netfilter.org/hawk/presentations/NetConf2017_Seoul/XDP_devel_update_NetConf2017_Seoul.pdf
Idea on slide 29 + 30: Implement in BPF-code an Out-of-Order safe way to use cpumap-redirect to dynamically load-balance IP-flows.
c4c202175424 (“Merge branch ‘bpf-sample-cpumap-lb’”) 1bca4e6b1863 (“samples/bpf: xdp_redirect_cpu load balance like Suricata”) 11395686586b (“samples/bpf: add Paul Hsieh’s (LGPL 2.1) hash function SuperFastHash”)
QA is giving my output from xdp_redirect_cpu, but forget to say what interface this is loaded on. Solve by simply outputting the interface in the output.
Missing an ifindex to match on in tracepoint
(Jakub question this)
(Plus other info)
Old list of stuff I need to work-on/fix for XDP/bpf project:
- XDP return frame API (needed by AF_XDP ZC)
- Bulking API for return frame API
- Bulking API for ndo_xdp_xmit
- Address massive XDP regression due to CONFIG_RETPOLINE
- Introduce bulking for generic-XDP (PoC test show +30% perf!!!)
- Fix CPUMAP to work with generic-XDP
- Implement ndo_xdp_xmit for macvlan (fast guest delivery)
- Improve BPF doc
- Improve XDP doc
- Work on XDP article with Toke+Alexei+Daniel
- Better integration of XDP in Suricata (multiple small thing)
- Find XDP feature/capability API (use in Suricata)
- Help integrate AF_XDP in Suricata
- Ship bpftool in distros (start with static linked libbpf)
- Make libbpf a shared lib in distros (fix lib versioning)
- Help (Ahern) get XDP route/FIB lookup helper integrated
- Work on bridge FIB table lookup helper
- XDP get more info transferred to CPUMAP skb creation time
- cpumap: GRO support
- cpumap: RX hash support
- cpumap: HW csum offload/info
- Generalize CPUMAP skb creation, move SKB alloc out of driver code
- Improve XDP cpumap redirect example: flow hashing (fix NIC HW hash)
- Help get AF_XDP API and performance aligned
- Help get AF_XDP zero-copy integrated via XDP return API
- xdp_monitor: improve to show error codes (to easier troubleshoot)
- convert tracepoint to use ifindex instead of names (strcpy overhead)
- XDP_REDIRECT: Detect buggy-drivers forgetting to clear per-CPU map
- Streamline eBPF map-create return codes on errors
- Upstream xdp_bench01 sample to be standard way to measure XDP perf
- Fix that TCP traffic with XDP generic on virtual net_devices are broken
- xdp: avoid leaking info stored in xdp_frame data on page reuse
- XDP_REDIRECT implemement in every driver
- XDP data-meta implemement in every driver
- Improve samples/bpf: XDP progs should take ifconfig/net_device names
- Improve samples/bpf: Avoid including ./arch/x86/include/asm/cpufeature.h
Proprietary commercial companies like NapaTech, which also maps packets into userspace, deliver dynamic descriptor info in-front of the packet, like our data_meta area.
As far as I can see, via their public docs[4], they have 4 different dynamic descriptor formats. With BTF and metadata we should have more flexibility than them :-)
It is a bit interesting to look at what they expose. I recommend looking/clicking at the header-file[5][6][7][8] as it shows they use a lot of C-bit-fields to compress the size. Do BTF support C-bit fields?
[4] https://docs.napatech.com/search/all?query=Dynamic+Packet+Descriptor [5] pktdescr_dyn1.h https://docs.napatech.com/reader/Gtwjm73bddn7nrHz1NxHZw/leAUnFb_t2il~h4y1tNPpw [6] pktdescr_dyn2.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/7cYsE5yb3DLpomSTEeL_bQ [7] pktdescr_dyn3.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/hcQobdatqtY2j2nmrZ577A [8] pktdescr_dyn4.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/2LaoD2p2mvxpkNOVvBPBqg
I find it interesting to see that in (dyn1+2), default decode offsets into the packet of L3 and L4 (and L4 payload), but allows them to be programmable. Also notice the “color” members, which are programmable, and sometimes are use as a 64-bit unique correlation key[8] (e.g. identifying flows).
Maybe I should have looked at their standard format before the dynamic(?) Their pktdescr_std0.h [9] is placed in front of all packets being received by the adapter when the adapter is operating in STANDARD or EXTENDED mode. The Extended descriptors are placed after pktdescr_std0.h [9], which contains a lot of info on the types in different layers, see[10][11][12].
[9] pktdescr_std0.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/mRPP74KNQXIJtSyWBBHMKA [10] pktdescr_ext7.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/GXhjyPfAPJ6Rr7k8KR4KsQ [11] pktdescr_ext8.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/NWaAIAROOdyXvy~OV4pl6A [12] pktdescr_ext9.h https://docs.napatech.com/reader/GHSQQPQbWLPdJUmxIkO91Q/LVIq8m_b0_44QIIAqcdLew