M.2 BCM4350 ASPM

[mishurov]

I've got problems after warm reboot from Linux to MacOS. I mainly work in Linux and occasionally use MacOS to compile and debug something for iOS. The problem is with ASPM, I gather.

Initially I had been using pin masking

The masked pin is 53. According to the PCI-E M.2 specification it is being used for CLKREQ#

Support for the CLKREQ# dynamic clock protocol should be reported using bit 18 in the PCI
Express link capabilities register (offset 0C4h). To enable dynamic clock management, bit 8 of the
Link Control register (offset 010h) is provided. By default, the card shall enable CLKREQ#
dynamic clock protocol upon initial power up and in response to any warm reset by the host system.

After removing the tape and using pci-aspm-default, it works fine if the system initially was powered off. But after rebooting from Linux card is inactive, without setting pci-aspm-default system hangs with something like "couldn't bring the core back after reset" or something. lspci -vvv -s 0.0X.00 in Linux shows that ASPM L0s and L1 are enabled regardless of the kernel parameters etc. MacOS pciutils from homebrew always show that ASPM is disabled. lspci -xxx -s 0.0X.00 also shows that corresponding register is set either to 0x43 or 0x40 in Linux and MacOS accordingly. setpci doesn't help. It looks like Linux puts the card into L1 state and MacOS can't bring the card back because it thinks that ASPM is not supported, although I've enabled it everywhere in configs.

I tried to override in this driver the method _pcie_clkreq, it has no effect. The other symbols such as __ZN15AirPort_BrcmNIC18overrideASPMDeviceEbb make no sense for me, I can't guess what these two booleans in the arguments stand for. And I don't know how to call IOPCIDevice::setASPMState in this Lilu environment.

From the practical point, I rolled back to using the pin masking. But just for the sake of perfectionism, can it be solved, can this driver utilise the ASPM features of the card and do not conflict with the Linux driver?

[Andrey1970AppleLife]

You write about AirportBrcmFixup?

[vit9696]

Regarding setASPMState, you can just call it if you link IOPCIFamily? It is a simple member function, not even virtual. As for the rest, well, needs exploring add debugging.

CC @lvs1974

[lvs1974]

@mishurov, did you try to inject property pci-aspm-default=0?

[mishurov]

@Andrey1970AppleLife, yes, this is about AirportBrcmFixup. Excuse me, please, it slipped my mind that it was a dedicated repo for bugs while I was writing the test.

@vit9696 I couldn't figure out how to link the class. I've read the docstrings in LegacyIOService.h and I've analysed the actual implementation in AirportBrcmFixup. If something crosses you mind, can you provide an example code in those projects that use Lilu?

[mishurov]

@lvs1974 that is basically what's the issue about. Without pin masking and with pci-aspm-default=0 the card is inactive after reboot from Linux because its driver always activates ASPM. I wonder if it is possible to safely activate ASPM in this patcher too.

[vit9696]

@vit9696 I couldn't figure out how to link the class. I've read the docstrings in LegacyIOService.h and I've analysed the actual implementation in AirportBrcmFixup. If something crosses you mind, can you provide an example code in those projects that use Lilu?

Link? Well, you just update Info.plist to contain the kext exporting it in OSBundleLibraries, and then cast a pointer to the class instance pointer?..

[mishurov]

@vit9696 it is interesting, I've just rebooted to MacOS with the masked pin, and lspci shows LnkCtl: ASPM L0s L1 Enabled; for both the root port and the card. I've powered off and booted into MacOS again, the ASPM is enabled. So it looks like the problem is related to CLKREQ.

[07151129]

What's the problem being reported? If you need to disable CLKREQ and keep L0s or L1 ASPM, you can do so by passing the correct option bits via pci-aspm-default:

enum
{
	kIOPCIExpressASPML0s   = 0x00000001,
	kIOPCIExpressASPML1    = 0x00000002,
	kIOPCIExpressCommonClk = 0x00000040,
	kIOPCIExpressClkReq    = 0x00000100
};

[mishurov]

@07151129 Thanks, you're right. Passing kIOPCIExpressASPML0s | kIOPCIExpressASPML1 | kIOPCIExpressCommonClk i.e. 0x43 i.e <integer>67</integer> enables ASPM L0s L1 and CommClk+ according to lspci. If I also disjunct kIOPCIExpressClkReq, the system hangs on boot.

It doesn't solve my problem with Linux though. The Linux driver enables ClockPM+ and after I reboot to MacOS, the card can't discover any SSID. It looks like the card has been put into L1 substate and it can't be resumed without CLKREQ# signals which doesn't work on MacOS and I don't know how to disable it on Linux easily.

[07151129]

You can force disable ASPM on Linux globally via pcie_aspm kernel parameter.

[mishurov]

@07151129 I've already tried, it has no effect. Probably Debian's kernel is compiled without CONFIG_PCIEASPM.
When 5.5 comes to backports, I'll try to configure it via sysfs /sys/bus/pci/devices/.../link_pm/clkpm

https://patchwork.kernel.org/patch/11175639/

Anyway, it is weird that every ASPM feature works except the clock power management in MacOS. All one has to do is to don't disjunct 0x00000100 into the bitmask for pci-aspm-default

I don't know if this is related cos I don't know how it works but there's also an interesting patch regarding LPO and bluetooth.
https://lore.kernel.org/patchwork/patch/1007820/

May be related to this #715

[mishurov]

All in all. Setting 0x42 or 0x43 for pci-aspm-default, i.e. integer 66 or 67 in Clover, enables ASPM levels without CLKREQ#. And simply unloading the module in Linux before reboot, i.e. modprobe -r brcmfmac puts the card into its initial state, initrd rc6 script / systemd service do what I need. I think that the issue can be closed unless someone knows how to make CLKREQ# work for this card.

[vit9696]

Well, sounds good to me as a workaround. Anybody who knows a better route can reply to the closed issue.

[07151129]

CLKREQ control is a part of the open-source platform-independent layer of brcm80211, which is also what AirPortBrcm uses. You can read the source of pcie_clkreq_upd to see what happens there if you want a proper fix, but essentially you are on your own here

[mishurov]

@07151129 yep, brcmsmac to be more precise as I was able to guess, I tried to patch some methods but without any success

[lvs1974]

Hi everybody.
Currently AirportBrcmFixup supports an optional boot-arg an ioreg property brcmfx-aspm (Number), where any set of aspm flags can be set. Respective call of setASPMState does this job.

[buyddy]

All in all. Setting 0x42 or 0x43 for pci-aspm-default, i.e. integer 66 or 67 in Clover, enables ASPM levels without CLKREQ#. And simply unloading the module in Linux before reboot, i.e. modprobe -r brcmfmac puts the card into its initial state, initrd rc6 script / systemd service do what I need. I think that the issue can be closed unless someone knows how to make CLKREQ# work for this card.

I tried to set pci-aspm-default=67 in boot arg, with AirportBrcmFixup 2.0.9. However I am not getting ASPM. lspci shows that "LnkCtl: ASPM Disabled;". Do I have to mask the pin in order to get ASPM working? Or is there something else I am missing?

[lvs1974]

@buyddy: AirportBrcmFixup disables ASPM for dw1820a (0x14e4:0x43a3) by default.
You should use brcmfx-aspm=67 in boot-args.
If you have unmasked pins, most probably your laptop will hang on boot, or will be unstable after sleep/wake.
Even with disabled ASPM you can get some troubles, like: Turn Off WiFi, sleep/wake - Turn On does not work any more.
This is solved only with masked pins.

[buyddy]

@buyddy: AirportBrcmFixup disables ASPM for dw1820a (0x14e4:0x43a3) by default.
You should use brcmfx-aspm=67 in boot-args.
If you have unmasked pins, most probably your laptop will hang on boot, or will be unstable after sleep/wake.
Even with disabled ASPM you can get some troubles, like: Turn Off WiFi, sleep/wake - Turn On does not work any more.
This is solved only with masked pins.

Thanks for the information! Does masking the pin do anything under Windows that I should be aware of?

[mishurov]

@buyddy when I had the masked pin, Windows hanged on boot during warm reboots from Linux, I had to turn off the laptop and turn it on again. Also Windows hanged during driver installation for this card.

[buyddy]

@buyddy when I had the masked pin, Windows hanged on boot during warm reboots from Linux, I had to turn off the laptop and turn it on again. Also Windows hanged during driver installation for this card.

So what is your current setup now? Do you have ASPM?

[mishurov]

@buyddy I rarely boot to MacOS, only when I need to debug a mobile app in the iOS simulator, I don't need power saving features there, I just disabled it and use normal ASPM with CLQREQ under Linux.