Request: Bootloader to not assign initial block of memory mapped I/O addresses

[KeyserSoze1]

This is a proposal for the bootloader to work around some PCIe devices that do not work with 0x00000000 memory mapped I/O addresses.

A few devices have been found that do not work with these low mmio addresses like the ASMedia 2806 PCIe switch (see raspberrypi/linux#6278) and Intel i225 and i226 NICS (see raspberrypi/linux#6134). A dtoverlay was made that works around this issue (see raspberrypi/linux#6186) however this does not affect the bootloader process which prevents booting from NVME behind the ASMedia 2806 for example.

My proposal is for the bootloader to behave similarly to the dtoverlay and skip the initial addresses so these devices can function properly. There is 4GB of available mmio addresses (32 bit anyway) from my understanding so skipping a portion for the bootloader should not cause any issues and allow these devices to work properly. While this is a workaround it seems to have little to no consequence to the bootloader as I can't for see any issue with the bootloader running out of addresses and could only help with compatibility of devices. This is with somewhat limited understanding of the bootloader process so I could be way off and this be a much harder problem than it looks on the outside.

I've posted a UART log below of the issue.

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? Pi5
• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2

• Which firmware version (vcgencmd version)?

2024/06/05 16:41:49 
Copyright (c) 2012 Broadcom
version 6fe0b091 (release) (embedded)

• Which kernel version (uname -a)?

Linux raspberrypi 6.6.31+rpt-rpi-2712 raspberrypi/firmware#1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

Logs
Here's a UART log with BOOT_ORDER=0xf416 and PCIE_PROBE=1 using the ASMedia 2806 PCIe switch.

UART Log (Click to expand)

  4.90 RPi: BOOTSYS release VERSION:6fe0b091 DATE: 2024/06/05 TIME: 16:41:49
  4.94 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1717602109 serial 3da8d558 boardrev d04170 stc 4594148
  4.04 AON_RESET: 00000003 PM_RSTS 00001000
  4.11 RP1_BOOT chip ID: 0x20001927
  4.14 PM_RSTS: 0x00001000
  4.14 part 00000000 reset_info 00000000
  4.18 PMIC reset-event 00000002 rtc 00000004 alarm 00000000 enabled 0
  4.24 uSD voltage 3.3V
  4.43 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267
  4.46 DDR 4267 1 0 64 152
  6.13 OTP boardrev d04170 bootrom a a
  6.14 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000
  6.21 VC-JTAG unlocked
  6.44 RP1_BOOT chip ID: 0x20001927

  7.54 RP1_BOOT chip ID: 0x20001927
  7.54 RP1_BOOT: fw size 25992
  7.09 PCI2 init
  7.09 PCI2 reset
  7.54 PCIe scan 00001de4:00000001
  7.55 RP1_CHIP_INFO 20001927

  7.57 RPi: BOOTLOADER release VERSION:6fe0b091 DATE: 2024/06/05 TIME: 16:41:49
  7.64 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1717602109 serial 3da8d558 boardrev d04170 stc 7664736
  7.74 AON_RESET: 00000003 PM_RSTS 00001000
  7.78 PCIEx1: PWR 0 DET_WAKE 0
  7.07 usb_pd_init status 0
  7.11 XHCI-STOP
  7.11 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  7.16 USBSTS 1
  7.18 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  7.23 xHC0 ports 3 slots 64 intrs 4
  7.35 XHCI-STOP
  7.35 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  7.40 USBSTS 1
  7.42 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  7.47 xHC1 ports 3 slots 64 intrs 4
  7.55 Boot mode: NVME (06) order f41
  8.56 PCI1 init
  8.56 PCI1 reset
  8.14 PCIe scan 00001b21:00002806
  8.15 PCIe scan 00001b21:00002806
  8.18 PCIe scan 00001b21:00002806
  8.21 PCIe scan 00001b21:00002806
  8.24 PCIe scan 00001b21:00002806
  8.27 PCIe scan 00001b21:00002806
  8.30 PCIe scan 00001c5c:0000174a
  8.34 Found device on bridge 3 with vid 1c5c enable 1
  8.38 PCIe scan 00001b21:00002806
  8.42 PCIe scan 00001b21:00002806
  8.45 PCIe scan 000010ec:00008125
  8.48 PCIe scan 00001b21:00002806
  8.51 PCIe scan 000010ec:00008125
  8.54 PCIe scan 00001b21:00002806
  8.57 PCIe scan 00001b21:00002806
  8.60 PCIe scan 00001c5c:0000174a
  8.64 PCIe scan 000010ec:00008125
  8.67 PCIe scan 000010ec:00008125
  8.17 ERROR:  cb 0xfbe09f40 123 src=1b_0000001c dest=0_3be0b000 size 4 debug 26000404 rc -1
  8.23      CS 13000409
  8.25      CB 01df04fa
  8.27      TI 00000000
  8.29     SRC 00000020
  8.31    SRCI 0000101b
  8.33    DEST 3be0b000
  8.36   DESTI 00001000
  8.38     LEN 00000000
  8.40 NEXT_CB 00000000
  8.42   DEBUG 26000404
  8.44  DEBUG2 00010000
  8.47 DMA4 0xfbe09f40 00000000_3be09f40 00000000_01df04fa

[qunanan]

May be this issue or feature would be part of rpi-eeprom.

[KeyserSoze1]

I would gladly re-submit under rpi-eeprom if that makes more sense. If someone can't move this automatically and I haven't heard otherwise, I'll open this new there in a few days.

[popcornmix]

Transferred issue to rpi-eeprom.

[timg236]

No particular need to move issues. I don't think anyone will be able to look at this in the short/medium term