Test SATA adapter (I/O Crest 4 port Marvell 9215) #1
Comments
|
I'm going to try out the IO Crest 4-port SATA adapter. |
|
It has arrived! |
|
And... I just realized I have no SATA power supply cable, just the data cable. So I'll have to wait for one of those to come in before I can actually test one of my SATA drives. |
|
First light is good: |
|
Though |
|
I just increased the BAR allocation following the directions in this Gist, but when I rebooted (without the card in), I got: Powering off completely, then booting again, it works. So note to self: if you get a |
|
Ah... looking closer, those 'failed to assign' errors are for IO BARs, which are unsupported on the Pi. So... I posted in the BAR space thread on Pi Forums asking 6by9 if that user has had the same logs and if they can be safely ignored. Still waiting on a way to power my drive so I can do an end-to-end test :) |
|
something else that may be interesting is if you can get a sas adapter/raid card working. I know I was looking into SBCs w/ pcie awhile back for the purpose of building a low power/low heat host for some sas drives I have. (ended up just throwing it in a computer and not running 24/7) |
|
That would be an interesting thing to test, though it'll have to wait a bit as I'm trying to get through some other cards and might also test 2.5 Gbps or 5 Gbps networking if I am able to! |
geerlingguy
changed the title
|
Without the kernel modules enabled, Going to try adding those modules and see what happens! |
# Install dependencies
sudo apt install -y git bc bison flex libssl-dev make libncurses5-dev
# Clone source
git clone --depth=1 https://github.com/raspberrypi/linux
# Apply default configuration
cd linux
export KERNEL=kernel7l # use kernel8 for 64-bit, or kernel7l for 32-bit
make bcm2711_defconfig
# Customize the .config further with menuconfig
make menuconfig
# Enable the following:
# Device Drivers:
# -> Serial ATA and Parallel ATA drivers (libata)
# -> AHCI SATA support
# -> Marvell SATA support
#
# Alternatively add the following in .config manually:
# CONFIG_ATA=m
# CONFIG_ATA_VERBOSE_ERROR=y
# CONFIG_SATA_PMP=y
# CONFIG_SATA_AHCI=m
# CONFIG_SATA_MOBILE_LPM_POLICY=0
# CONFIG_ATA_SFF=y
# CONFIG_ATA_BMDMA=y
# CONFIG_SATA_MV=m
nano .config
# (edit CONFIG_LOCALVERSION and add a suffix that helps you identify your build)
# Build the kernel and copy everything into place
make -j4 zImage modules dtbs # 'Image' on 64-bit
sudo make modules_install
sudo cp arch/arm/boot/dts/*.dtb /boot/
sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm/boot/zImage /boot/$KERNEL.img |
|
Yahoo, it worked! |
|
Repartitioning the drive: Got the following: Rebooted the Pi, then: To format the device, use Then mount it somewhere: |
|
Performance testing of the Kingston SA400S37/240G drive:
Compare that to the same drive over USB 3.0 using a USB to SATA adapter:
So not a night-and-day difference like with the NVMe drives, but definitely and noticeably faster. I'm now waiting on another SSD and a power splitter to arrive so I can test multiple SATA SSDs on this card. And someone just mentioned they have some RAID cards they'd be willing to send me. Might have to pony up for a bunch of hard drives and have my desk turn into some sort of frankemonster NAS-of-many-drives soon! |
|
I'm curious about other OS's. Obviously, Raspbian is a good basis - but as I recall, Fedora Pi 64-bit uses their own custom kernel. I'd be interested in seeing what they've "left in" from the standard kernel config. I'm looking forward to picking one of these up in a month or so when they become available to the public, then I'll give it a try! Side note for your list page - could you include PCI ID's as well as just the brand names of the cards? It'll help avoid confusion where cards have multiple revisions, as well as help non-US users identify comparable cards in their own markets. Great work in the meantime! 👍 |
It would be great to test a RAID card based on Marvell 88SE9128 chipset, because it is used by many suppliers |
|
Trying again today (but cross-compiling this time since it's oh-so-much faster) now that I have two drives and the appropriate power adapters. I'm planning on just testing a file copy between the drives for now, I'll get into other tests later. |
|
Hmm... putting this on pause. My cross compilation is not dropping in the AHCI module for some reason, probably a bad |
|
Also, the adapter gets hot after prolonged use. |
|
(For anyone interested in testing on an LSI/IBM SAS card, check out #18) |
|
My desk is becoming a war zone:
Plan is to set up a RAID (probably either 0 if I feel more YOLO-y or 1/10 if I'm more stable-minded) with either 2 or 4 drives, using mdadm. I was having trouble with the SAS card, not sure if the cards are bad or they just don't work at all with the Pi :( |
|
Testing also with an NVMe using the IO Crest PCIe switch: I'll post some benchmarks copying files between one of the SSDs and the NVMe; will be interesting to see how many MB/sec they can pump through the switch. |
|
For a direct file copy from one drive to another: I got an average of 190 MiB/sec, or about 1.52 Gbps. So two-way, that's 3.04 Gbps (under the 3.2 Gbps I was hoping for, but that's maybe down to PCIe switching? It looks like CPU goes to 99% as SDA takes more than 50% of the CPU—see atop results during a copy:
|
|
Also comparing raw disk speeds through the PCIe switch: Kingston SSD
Samsung EVO 970 NVMe
These were on 64-bit Pi OS... so the numbers are a little higher than the 32-bit Pi OS results from earlier in the thread. But the good news is the PCIe switching seems to not cause any major performance penalty. |
|
Software RAID0 testing using Benchmarking the device:
And during the 4K tests in iozone, I can see the sda/sdb devices are basically getting the same bottlenecks, except with a tiny bit of extra overhead from software-based RAID control:
Then to stop and remove the RAID0 array: |
Thanks. I saw that the required kernel configs were enabled in the kernel config file, but didn't find them in the modules directory. I thought an issue with the distribution and was going to compile the kernel. I didn't know that the extra modules are delivered as a separate package. fortunately I found your comment. |
|
Thanks @BeauSlim for the inputs, today i tested the card with Ubuntu Impish Indri and got all the 4 drives detected; however i see repeated errors mentioned below and troubleshooting them now; [ 788.484701] sd 0:0:0:0: [sda] Synchronizing SCSI cache If you know anything about the errors and suggest what to look at would be of great help... |
Yeah, this Pi SATA stuff is definitely a bit tricky. Googling errors will get you a lot of people saying "your drive is dead", but I bet if you plug that card into a PC, everything will work perfectly even under heavy load. I don't have a 9215. I have a Marvell 9230 and a JMicron 585. The 9230 card runs well aside from a lack of any way to change the RAID config. PMP seems to be referring to a port multiplier? Are your 4 drives in an external enclosure? If so, I'd try connecting drives directly. If not, definitely try different cables. If I push my JMB585 with 4 or 5 drives in a stripe or software RAID 10, it gives me a bunch of errors, but they are mostly "failed command: READ FPDMA QUEUED" which is different from yours. Adding "extraargs=libata.force=noncq" to my cmdline.txt solves that but hurts SATA performance. For other libata.force options to try (like using SATA I speeds), see https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html You might also try adding "extraargs=pcie_aspm=off" to your cmdline.txt to turn off PCIe power management. There is probably a firmware update for your card that you could try. |
|
Thanks @BeauSlim for your inputs; yes i have 4 drives in an IOCRest external enclosure. Definitely drives are not dead as i have tested them with RADXA QUADSATA HAT and they work fine in it will try turning off PCIe power management option & also try different set of cables and update with results. |
|
"You might also try adding "extraargs=pcie_aspm=off" to your cmdline.txt to turn off PCIe power management." I may also try this, my 9215 works absolutely fine with 3x SATA 8TB Drives, the only issue I get occasionally is the drive power down/up sound and a couple of the disks change from say SDA/SDC to SDD/SDE, it does however break OMV when that happens. |
Does this mean, there is no CLI to enable/change hardware RAID modes? |
That is correct. The Marvell hardware RAID config (MRU) is available only for x86/x64 processors. You can put the card into a Windows or Linux PC, connect the disks you plan to use, configure RAID, and then move the card to your Pi setup. You might even be able to have a 9230-based card in the PC and just move the disks since the RAID config is stored on the drives themselves, not on the card. This is probably fine if you just wanted to use striping or Hyperduo SSD caching, but if you want redundancy you will have no indication that a mirror has failed. |
|
I'm testing an ASM1061R based controller basically identical with these
Issue:
is affecting my controller too and was even confirmed by distributor's technical support in this FAQ Answer: @geerlingguy should such severe limitations not be documented in the "Raspberry Pi PCI Express device compatibility database" ? |
|
Testing on the Raspberry Pi 5: At PCIe Gen 2.0, I'm getting some link errors—but otherwise the card seems to pass through clean-ish at least: I think the PCIe issues are down to the FFC cable and PCIe interference :( |
|
I have some questions in to Raspberry Pi surrounding SATA support, and PCIe link quality. It seems like both cards I've tested run into some errors (more so than I get with NVMe...). |
|
Got my hands on a Pimoroni NVMe base and used a M.2 to PCIe adapter. Please excuse the sketchy setup - just wanted to quickly test. 
And connected 4 drives (these are old HDDs that have various partitions on): And successfully mounted all except Running pibenchmark yields the following in hardware identification: sda results: https://pibenchmarks.com/benchmark/76956/ I then ran two benchmarks in parallel on sda and sdc which both completed (didn't submit results). All this to say that I think this card now works as expected. I was using the default PCIe link speed. |
|
For anyone looking to use this card on a CM4IO, I designed a bracket for it that keeps in in place, so it doesn't get loose and cause a system lockup. https://www.printables.com/model/789947-cm4-si-pex40064-bracket/related |
|
Hi everyone! I just installed a ASMedia ASM1064 4 Port SATA III Card onto a P02 PCIe Adapter Board and it was detected immediately upon boot: pi@NAS-Pi:~ $ lspci However, I can't seem to get the Pi to see the drive that I have plugged into it. Should I recompile the kernel or am I having a different problem? [EDIT] checking my logs, I see messages for the AHCI controller being unavailable: pi@NAS-Pi:~ $ journalctl -p 0..3 -r pi@NAS-Pi:~ $ dmesg | grep -iC 3 "sata" ........... repeated entries cut .................. [ 7.022355] ata1: found unknown device (class 0) ........... repeated entries cut .................. [ 146.630318] EXT4-fs (mmcblk0p2): mounted filesystem 56f80fa2-e005-4cca-86e6-19da1069914d ro with ordered data mode. Quota mode: none. ........... repeated entries cut .................. [ 261.141938] ahci 0000:01:00.0: AHCI controller unavailable! |
I wonder if that might be the drive having trouble powering on? Is it being powered from an external PSU? |
|
Yes, but I used an SSD, so I can't really tell if the drive is powering up. I'll use a normal SATA drive tomorrow, and try again with a different PSU and buck converters. I'll post my results. |
|
I retried today with an ATX PSU to power the drive. I used a normal SATA HDD so that I could feel if it was spinning. The drive was sort of seen: 3.864156] ata24: SATA max UDMA/133 abar m8192@0x1b00082000 port 0x1b00082c80 irq 168 But I think that my problem is here: pi@NAS-Pi:~ $ dmesg | grep -iC 3 "sata" The card that I am using only has 4 ports, yet for some reason the Pi is seeing 24 ports. I'm going to see if there are any drivers or maybe a source code available. |
@geerlingguy I'm trying to compile the kernel as you described. However, when I get to "sudo cp arch/arm/boot/dts/*.dtb /boot/" I get an error: Any suggestions? [EDIT] Here is what's in my arch/arm/boot/dts directory after compiling: |
Docs from Raspberry Pi on building the kernel are at https://www.raspberrypi.com/documentation/computers/linux_kernel.html#native-build I'm fairly certain that the relevant modules for the Marvell SATA cards are enabled in the default Pi kernels, so there is no need to rebuild the kernel yourself for that reason. |
|
Also debugging a similar error over here: #85 The answer according to raspberrypi/linux#6214 is to add the following lines to |
6by9 on the Pi Forums mentioned:
I bought this I/O Crest 4 Port SATA III PCIe card and would like to see if I can get a 4-drive RAID array going:
Relevant Links:
The text was updated successfully, but these errors were encountered: