Enable Vfio multifunction devices

[christoph-zededa]

see also #3369

[rene]

@christoph-zededa , FYI, Lenovo SE70 (based on Jetson Xavier NX) is a good candidate to test these changes on arm64... it has some peripherals on PCIe bus and it implements SMMU....

[rucoder]

why pcir-pci and no pcie-pcie? Shown PCIe device ans PCI has consequences and usually not a good idea

[christoph-zededa]

pcie-pcie does not exist according to https://github.com/qemu/qemu/blob/master/docs/pcie.txt#L39

[rucoder]

sure, because there is no need to translate. You either create a dedicated port to control BDF assignments or plug device into a PCIe switch

[christoph-zededa]

I tried this:

[device "pci.7"]
  driver = "pcie-root-port"
  port = "17"
  chassis = "7"
  bus = "pcie.0"
  multifunction = "on"
  addr = "0x7"
  slot = "7"



[device "upstream_port.1"]
  driver = "x3130-upstream"
  bus = "pci.7"

[device "downstream_port1"]
  driver = "xio3130-downstream"
  bus = "upstream_port.1"
  slot = "71"

[device "downstream_port2"]
  driver = "xio3130-downstream"
  bus = "upstream_port.1"
  slot = "72"



#[device "pcie-bridge.7"]
#  driver = "pcie-pci-bridge"
#  bus = "pci.7"
#  addr = "0x0"

[device]
  driver = "vfio-pci"
  host = "00:0d.0"
#  bus = "pcie-bridge.7"
  bus = "downstream_port1"
  addr = "0x1"
[device]
  driver = "vfio-pci"
  host = "00:0d.2"
#  bus = "pcie-bridge.7"
  bus = "downstream_port2"
  addr = "0x2"

but I get the same error message:

qemu-system-x86_64:xen1.cfg:264: vfio 0000:00:0d.2: group 7 used in multiple address spaces

[OhmSpectator]

My only real concern I put to the comment on line 1099 (#4394 (comment)); the rest are mostly recommendations and questions.

[OhmSpectator]

We depend on the slice remaining unmodified and unsorted after it is created. If we remove an element from it, the indices can change. Currently, we don't have this behaviour, but it would be beneficial to explicitly state the requirement that the slice remains unmodified. A comment added to the slice would be a good solution.

[OhmSpectator]

In general, it looks fine to me.
I hope @christoph-zededa retest it with the latest changes, and we can run the automated tests as well.

[OhmSpectator]

@christoph-zededa could you please rebase onto the latest master? It should fix the Eden tests run. =)

[OhmSpectator]

It would also be nice to document the trick with the bridge and explain why it's necessary... But I don't see any document where we have our passthrough approach explained. So I don't know where to add it =(

[OhmSpectator]

Release the Kraken!
(run the Eden tests!)

[OhmSpectator]

"You have reached your pull rate limit." >=(

[OhmSpectator]

Another test fails with

The self-hosted runner: buildjet.com_e72335e4-67c2-4a64-8a95-ae86c04c78c0 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

UPD. Three of them failed with this.

[shjala]

at least Eden onboard issue is gone :

        # eden onboard (0.000s)
        > [!exec:bash] stop
        > [!exec:jq] stop
        # enable templates check if we run with TPM enabled (0.039s)
        > exec -t 1m bash set_template_check_enforce.sh true
        [stdout]
        time="2024-11-14T11:18:35Z" level=info msg="Options loaded"
        # Onboarding. (161.052s)
        > eden eve onboard
        [stdout]
        time="2024-11-14T11:18:35Z" level=info msg="Adam waiting for EVE registration (0) of (20)"
        time="2024-11-14T11:18:55Z" level=info msg="Adam waiting for EVE registration (1) of (20)"
        time="2024-11-14T11:19:15Z" level=info msg="Adam waiting for EVE registration (2) of (20)"
        time="2024-11-14T11:19:35Z" level=info msg="Adam waiting for EVE registration (3) of (20)"
        time="2024-11-14T11:19:55Z" level=info msg="Adam waiting for EVE registration (4) of (20)"
        time="2024-11-14T11:20:15Z" level=info msg="Adam waiting for EVE registration (5) of (20)"
        time="2024-11-14T11:20:35Z" level=info msg="Adam waiting for EVE registration (6) of (20)"
        time="2024-11-14T11:20:55Z" level=info msg="Adam waiting for EVE registration (7) of (20)"
        time="2024-11-14T11:21:15Z" level=info msg="Device uuid: 65eb4fe1-5290-41e5-9c0e-3fc99484f1a5"
        config changed, to see config run 'eden controller edge-node get-config'
        time="2024-11-14T11:21:16Z" level=info msg=onboarded
        time="2024-11-14T11:21:16Z" level=info msg="device UUID: 65eb4fe1-5290-41e5-9c0e-3fc99484f1a5"
        > stdout 'onboarded'
        PASS

[OhmSpectator]

@christoph-zededa, all green. Merging. But if you have any thoughts regarding the failure, would be nice to figure out the reason.

[OhmSpectator]

@christoph-zededa, don't we want to backport it to 13.4-stable? As far as it's mentioned in the discussion regarding the 13.4 LTS testing.

[christoph-zededa]

@christoph-zededa, don't we want to backport it to 13.4-stable? As far as it's mentioned in the discussion regarding the 13.4 LTS testing.

done: #4457