Raspberry Pi 5 USB 3.0 Write Speed Issue

[schoolpost]

I have conducted a number of tests that demonstrate the Raspberry Pi 5 has a noticeably slower write speed vs that of a Pi 4 when writing to external disks ( SSD's ) via USB 3.0

it also falls well short of the practical speeds that should be achievable via USB 3.0, even accounting for protocol / transfer overheads, 400-450MB/s

My original investigation and findings are detailed here: https://forums.raspberrypi.com/viewtopic.php?t=367370

Pi 5 speeds:

pi@raspberrypi:~ $ sudo ./raw_disk /dev/sda2
Written bytes: 41943040000
Time: 152975439398ns
Speed: 0.274182 GB/s

Pi 4 speeds:

pi@raspberrypi:~ $ sudo ./raw_disk /dev/sda2
Written bytes: 41943040000
Time: 127376266995ns
Speed: 0.329285 GB/s

The benchmark application is compiled from source found here: https://stackoverflow.com/questions/71038719/how-to-reach-10gb-s-writes-in-c-c-to-nvme-disks-under-linux

[ThomasKaiser]

Have you set the cpufreq governor to performance prior to running your benchmark? If you use Raspberry Pi OS the default is ondemand without io_is_busy=1

[ThomasKaiser]

Quick tests using this SSD:

root@rpi5:/mnt/sda1# sbc-bench.sh -S
  * 111.8GB "Samsung SSD 750 EVO 120GB" SSD as /dev/sda [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind ASMedia SATA 6Gb/s bridge (174c:55aa), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 26°C
  * 29.7GB "SanDisk SN32G" UHS SDR104 SD card as /dev/mmcblk0: date 08/2023, manfid/oemid: 0x000003/0x5344, hw/fw rev: 0x8/0x0

Testing methodology: iozone -e -I -a -s 1000M -r 16384k -i 0 -i 1 -i 2 (results KB/sec):

                                                random    random
    reclen    write  rewrite    read    reread    read     write
1)   16384   268674   268322   388941   389261   387650   268278                                                                
2)   16384   268748   268652   389988   390306   388778   268697
3)   16384   323152   323042   390132   390299   389076   323047

• 1. were with OS defaults
• 2. was after echo 1 >/sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
• 3. switching ASPM from OS defaults (powersave) to default: echo default > /sys/module/pcie_aspm/parameters/policy

No idea (so far) how switching ASPM from powersave to default has an impact on idle consumption. BTW: an additional test setting ASPM to performance ended up with same numbers than default.

As reference numbers from my Pi 4 made directly before with same SSD, same cable, same install on same SD card: 1st time ASPM set to powersave (OS default), 2nd to performance:

                                                random    random
    reclen    write  rewrite    read    reread    read     write
1)   16384   326208   319854   357959   358105   356869   319998                                                                
2)   16384   328924   323672   364055   364183   362978   323817

[ThomasKaiser]

the practical speeds that should be achievable via USB 3.0, even accounting for protocol / transfer overheads, 400-450MB/s

Can you show any such numbers achieved over an USB3 SuperSpeed (5 GBps) connection? Me not, the best I achieved were 325-330 MB/sec at 16M blocksize with an EVO840 that shows way higher sequential transfer speeds on a SATA controller.

Forgot that I personally already measured 415/420 MB/s with modern ARM SoCs and USB3 SuperSpeed.

Retested with that EVO840 on two different USB3-to-SATA bridges and same picture: with ASPM set to powersave write performance suffers while when set to default this looks ok-ish to me:

                                                     random    random
         reclen    write  rewrite    read    reread    read     write
ASM1153   16384   322309   321589   389908   390133   389848   321371                                                               
JMS567    16384   313090   312219   372423   372628   372313   312648

And an additional test with a btrfs raid1 out of the two EVOs shows also the expected difference wrt ASPM:

                                                         random    random
             reclen    write  rewrite    read    reread    read     write
powersave     16384   287698   287499   370139   370178   369596   282961
performance   16384   303702   304217   387700   387697   386282   300246

root@rpi5:/mnt/btrfs-raid1# sbc-bench.sh -S
  * 111.8GB "Samsung SSD 750 EVO 120GB" SSD as /dev/sda [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind ASMedia SATA 6Gb/s bridge (174c:55aa), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 25°C
  * 111.8GB "Samsung SSD 840 EVO 120GB" SSD as /dev/sdb [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind JMicron JMS567 SATA 6Gb/s bridge (152d:3562), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 26°C
  * 29.7GB "SanDisk SN32G" UHS SDR104 SD card as /dev/mmcblk0: date 08/2023, manfid/oemid: 0x000003/0x5344, hw/fw rev: 0x8/0x0

root@rpi5:/mnt/btrfs-raid1# grep btrfs /etc/mtab 
/dev/sda /mnt/btrfs-raid1 btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0
    
root@rpi5:/mnt/btrfs-raid1# btrfs device usage /mnt/btrfs-raid1/
/dev/sda, ID: 1
   Device size:           111.79GiB
   Device slack:            8.00KiB
   Data,RAID1:              3.00GiB
   Metadata,RAID1:          1.00GiB
   System,RAID1:           32.00MiB
   Unallocated:           107.76GiB

/dev/sdb, ID: 2
   Device size:           111.79GiB
   Device slack:            8.00KiB
   Data,RAID1:              3.00GiB
   Metadata,RAID1:          1.00GiB
   System,RAID1:           32.00MiB
   Unallocated:           107.76GiB

[P33M]

Two things conspire to give reduced performance here - L1 entry/exit latencies get in the way of DMA, and a devicetree property that should optimise the USB3 controller transmit FIFO threshold wasn't getting applied.

This particularly affects sequential write performance, as you've seen.

What happens if you do (on an otherwise idle system, and without critical data on any attached USB drive)

$ sudo busybox devmem 0x1f0010970c 32 0x232c3000
$ sudo busybox devmem 0x1f0020c108 32 0x22080000
$ sudo busybox devmem 0x1f0030c108 32 0x22080000

And re-run the benchmarks?

[dia3olik]

hey guys! any news on this issue?

[P33M]

Yes. It's fixed.