Question about connection tracking

[ivyxyz]

Since connection tracking is implemented by per cpu lru maps, is there affinity of incoming packets to individual cpus? In other words, if a source ip and port pair is assigned a real on a cpu, will another cpu process a packet from the same source ip and port later and assign a different real?

lru_maps_mapping is BPF_MAP_TYPE_ARRAY_OF_MAPS containing a BPF_MAP_TYPE_LRU_HASH map per cpu. Will BPF_MAP_TYPE_LRU_HASH or BPF_MAP_TYPE_LRU_PERCPU_HASH with the BPF_F_NO_COMMON_LRU flag achieve the same goal?

[tehnerd]

prerequest for katran to run is to have irq affinity configured for your NIC. When it is configured to have 1 to 1 mapping between irq and cpu - flow ( packets with same src dst ports and protocol) will always be processed by single cpu. You can find more info on how to configure it (irq affinity) in EXAMPLE.md

[tehnerd]

For the second part of you question - lru map w/ no common lru will allocate per cpu regions as well and would more or less achieve the same goal. But it less efficient in terms of mem usage if number of NICs queues is less than cpu on the system. Or if system has more than one NUMA domains

[ivyxyz]

Thanks for your answers. For the first part, does that mean the NIC or driver is required to place packets with the same 5-tuple on the same rx queue?

If a NIC has 64 queues and the cpu cores are fewer than 64, will no common lru behave the same? Will binding all queues to a single NUMA node help?

[tehnerd]

NICs are doing this by default (this is how rss works. All packets from the same flow will go to the same rx queue)

If NICs has more queues than cpu most likely they will by default use same number of queues as cpu in the system. If it’s not the case - you can configure number of them with “ethtool -L” command. If the number of queues is the same as number of CPUs that no_common_lru will not waste memory (as all of its regions are going to be used)

If you start to bind all IRQs to the same NUMA node - no_common_lru is going to have wasted memory (as it will allocate regions for CPUs on remote numa node, which is never going to be used)

One thing about per_cpu regions is they are allocated for possible_cpus and not for current online CPUs. What we saw from other deployments - that there were servers with 6 online CPUs but with 128 possible (/sys/devices/system/cpu/possible). And this was huge waste of memory

[ivyxyz]

Thanks for your detailed explanation. This explained why I see more values than the number of cpus when dumping bpf per cpu maps. And the number of XDP queues is also equal to possible cpus, which I assume waste some NIC queues. Does XDP use the same rx/tx queues set up in the system, since I saw the "combined + xdp queues < max_tx_queues" limitation?

So no_common_lru will either waste memory by only using one NUMA node, or reduce performance by using all cpus and accessing some memory on the other NUMA node. The workaround is using maps of maps and allocating lru maps on the selected cpus.

The pci slot of the NIC card is typically in one NUMA node. Does processing packets on other NUMA nodes reduce performance?

[tehnerd]

“Does XDP use the same rx/tx queues set up in the system, since I saw the "combined + xdp queues < max_tx_queues" limitation?”

Could you elaborate more on this question? If you are asking “is xdp allocate queue per cpu” - then depends on the driver. most of em do it

As for remote NUMA for nic handling - that depends - do you have enough CPUs on local? If you have enough - you can process everything on local cpu. If you need more cores - then you will pay some coast for processing on remote - but this is still better that not have anything at all.

[tehnerd]

Feel free to reopen if you have more questions. Closing for now