Use mlx5 vf as PF1 proxy

[PlagueCZ]

Given TAP device's limitations, we implemented the PF1-proxy using a VF bound to MLX5 driver (as opposed to VFIO). This VF is created on PF1 to make it easier to set up.

This also enables offloading of the proxy path. For normal operation this is easy, for OSC vith virtual services, there is a little bit more work, but nothing major.

I tried to reorder and squash commits into blocks of work, so this PR should be nicely readable by going through it commit-by-commit.

I had to change the prepare script, but it should be backwards-compatible. There is one reordering (setting switchdev before setting the number of VFs) that is needed for multiport mode. OSC is also now provisioning the HW differently, so I only tested the prepare script's HW setup by hand locally.

Connected to #606

[byteocean]

@PlagueCZ Thanks for providing this enhancement of using VF as the proxy node, which is very interesting & nicely done. I would also suggest to add the tx node of the proxy to the node graph in the documentation, but as a separate one. The prep script works on my testing machine.

[PlagueCZ]

As for the graph schema, I thought about it, but given that virtual services are also not there because they are in an ifdef, I dod not do it.

But if it's wanted I could create two copies of the shema and create a virtual-services-specific one and pf1-proxy-specific one.

[byteocean]

LGTM.

[guvenc]

@PlagueCZ
I am still going through the changes but one thing which popped into my mind is that clang build which is run anyways and produces throw away binary during the CI/CD can include the compiler flag of pf1 proxy so that this newly introduced code path gets compiled at least once during CI/CD.

[PlagueCZ]

We should really run a tester in a job in OSC, but unfortunately that is still not being done.
We should then add another compilation run though, because I already had clang catch a warning about unused variable, which was unused due to pf1-proxy not being there

[guvenc]

@PlagueCZ
I think this PR is in good shape. Thanks also for providing the updated dataplane graph schema.

• Let's add a (temporary) compilation step in Dockerfile where this newly introduced feature is enabled, as discussed.

• And also the clarification whether the RoCE needs to be disabled on firmware level to get a good performance when mpesw is enabled. If so, documenting it in the docs with a sentence or so.

• And also please rebase to main.

Apart from these things, in pretty good shape. Thanks !