Apache NuttX RTOS for Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)

Read the articles...

• "Rust Apps on Ox64 BL808 RISC-V SBC and Apache NuttX RTOS"

• "Too many Embedded Logs? PureScript might help (Ox64 BL808 SBC / Apache NuttX RTOS)"

• "(Homage to MakeCode) Coding Ox64 BL808 SBC the Drag-n-Drop Way"

• "QuickJS JavaScript Engine on a Real-Time Operating System (Apache NuttX RTOS)"

• "Zig runs ROM FS Filesystem in the Web Browser (thanks to Apache NuttX RTOS)"

• "TCC RISC-V Compiler runs in the Web Browser (thanks to Zig Compiler)"

• "Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)"

• "Emulate Ox64 BL808 in the Web Browser: Experiments with TinyEMU RISC-V Emulator and Apache NuttX RTOS"

• "Apache NuttX RTOS in a Web Browser? Adventures with TinyEMU and VirtIO"

• "Nim on a Real-Time Operating System: Apache NuttX RTOS + Ox64 BL808 SBC"

• "$8 RISC-V SBC on a Real-Time Operating System: Ox64 + NuttX"

• "Fixed the UART Interrupt and Platform-Level Interrupt Controller (Ox64 BL808)"

• "RISC-V Ox64 BL808 SBC: UART Interrupt and Platform-Level Interrupt Controller (PLIC)"

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

• "RISC-V Ox64 BL808 SBC: Sv39 Memory Management Unit"

• "RISC-V Ox64 BL808 SBC: Starting Apache NuttX RTOS"

• "RISC-V Ox64 BL808 SBC: Booting Linux and (maybe) Apache NuttX RTOS"

What's this BL808? (Datasheet) (Reference Manual)

BL808 is a complex creature with 3 (Asymmetric) RISC-V Cores (linked via IPC)...

1. D0 Core: T-Head C906 64-bit RV64IMAFCV (480 MHz)

   (Multimedia Core with MIPI CSI / DSI, Neural Proc Unit)

   (Memory Mgmt Unit is Sv39, 128/256/512 TLB table entry. Same as Star64)

2. M0 Core: T-Head E907 32-bit RV32IMAFCP (320 MHz)

   (Wireless + Peripherals Core with WiFi, BLE, BT, Zigbee, Audio)

3. LP Core: T-Head E902 32-bit RV32E[M]C (150 MHz)

   (Low Power Core)

   (Upcoming BL606 looks similar, minus the Low Power Core)

Pine64 Ox64 is the dev board for BL808C.

(BL808C supports MIPI CSI Cameras but not MIPI DSI Displays. Maybe someday we'll see BL808D for MIPI DSI Displays)

Is Ox64 BL808 an SBC? Or an MCU Board?

Technically Ox64 BL808 boots 64-bit RISC-V Linux (via MicroSD), so it feels like an SBC...

• "Booting Linux on the Pine64 Ox64 SBC"

• OpenBouffalo Wiki

• Linux Image + OpenSBI + U-Boot Bootloader for BL808

  (Newer version?)

  (OpenSBI is the BIOS for RISC-V SBCs)

• USB-C Port for Camera Module (Dual-Lane MIPI CSI)

  (USB-C is not for Flashing!)

• USB 2.0 support for USB OTG

  (On-The-Go = USB Host + USB Device)

But Ox64 BL808 also feels like an MCU Board...

• Form Factor is similar to MCU Board

• Limited Memory: 64 MB of RAM, 128 Megabits (16 MB) of Flash Memory

• M0 Wireless Core is 32-bit RISC-V MCU

• UART Pins need a USB Serial Adapter for Flashing and Console I/O

• Powered by Micro USB Port

  (Micro USB is not for Flashing either!)

• Super Affordable: $8 for a 64-bit RISC-V Board!

Ox64 BL808 sounds a little tiny for 64-bit Linux?

Yeah 64-bit Linux runs with Limited RAM on the D0 Multimedia Core. But most Peripherals are hosted on the M0 Wireless Core: WiFi, BLE, BT, Zigbee, Audio, ...

So we flash M0 with a simple 32-bit RISC-V Firmware, to forward the Peripheral Interrupts from M0 to D0 Linux.

Here are the binaries loaded into D0 Multimedia Core and M0 Wireless Core, from buildroot_bouffalo...

• d0_lowload_bl808_d0.bin: This is a very basic bootloader that loads opensbi, the kernel and dts files into ram

• m0_lowload_bl808_m0.bin: This firmware runs on M0 and forwards interupts to the D0 for several peripherals

• bl808-firmware.bin: An image containing OpenSBI, Uboot and uboot dtb files.

• sdcard-*.tar.xz: A tarball containing the rootfs for the image to be flashed to the SD card

Perhaps Ox64 BL808 might run more efficiently with a tiny 64-bit RTOS.

Why Apache NuttX RTOS?

It might be interesting to run Apache NuttX RTOS on both the D0 Multimedia Core and the M0 Wireless Core. Then D0 and M0 can talk over OpenAMP (Asymmetric Multi-Processing).

Let's explore...

NuttX Automated Daily Build for Ox64

NuttX for Ox64 is now built automatically every day via GitHub Actions.

The Daily Releases are available here...

• nuttx-ox64/releases

nuttx.hash contains the Commit Hash of the NuttX Kernel and NuttX Apps repos...

NuttX Source: https://github.com/apache/nuttx/tree/31a6ffa15ca4e1604f80a02646a73c16570b8cbb
NuttX Apps: https://github.com/apache/nuttx-apps/tree/dcfb4d066247463daed6c59ed09668004f72e1c8

The GitHub Actions Workflow is here...

• ox64.yml

To run the NuttX Daily Build on Ox64...

1. Download the Image file from the Daily Build
2. Overwrite the Image file in the Ox64 Linux microSD Card
3. Boot Ox64 with the microSD Card

Maybe someday we'll do Daily Automated Testing with the Ox64 Emulator...

• "Emulate Ox64 BL808 SBC with TinyEMU"

Or with a real Ox64 SBC...

1. Download the Daily Build to TFTP Server
2. Power on Ox64 with an IKEA Smart Power Plug via Home Assistant
3. Ox64 boots the Daily Build over TFTP
4. Capture the Automated Testing Log and write to the Release Notes

(Similar to BL602)

Flashing UART vs Serial Console

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

We need to handle TWO UART Ports on Ox64?

Yeah don't confuse the 2 UART Ports on Ox64! Let's give the UART Ports distinctive names (like Migi & Dali)...

1. Ox64 Flashing UART: Used for Flashing Ox64

   • Flashing UART TX is GPIO 14 (Physical Pin 1)
   • Flashing UART RX is GPIO 15 (Physical Pin 2)
   • Remember to connect GND
   • Baud Rate for Normal Mode: 2,000,000 (2 Mbps)
   • Baud Rate for Flashing Mode: 230,400 (230.4 kbps)
   • BL808 UART0 is controlled by the M0 Wireless Core (OpenBouffalo Firmware)

2. Ox64 Serial Console: Used for Linux Serial Console (plus OpenSBI and U-Boot Bootloader)

   • Serial Console TX is GPIO 16 (Physical Pin 32)
   • Serial Console RX is GPIO 17 (Physical Pin 31)
   • Remember to connect GND
   • Baud Rate: 2,000,000 (2 Mbps)
   • BL808 UART3 is controlled by the D0 Multimedia Core (Linux + OpenSBI + U-Boot)
   • Output is totally blank if OpenBouffalo Firmware wasn't flashed correctly, or if OpenSBI / U-Boot / Linux couldn't boot

NEITHER UART Port is accessible over USB-C or Micro USB. So yeah it's totally counterintuitive.

(Maybe someone can create a Stackable HAT or Breadboard, that will expose the 2 UART Ports as USB Dongles? Or a UART Switcher?)

(For Pre-Production Ox64: Physical Pins are different, but GPIOs above are correct)

Why 2 Baud Rates for Flashing UART?

When we power up Ox64 in Normal Mode: (Boot Button NOT pressed)

• Flashing UART Port will show us the OpenBouffalo Firmware running on M0 Wireless Core

• This M0 Firmware will forward Peripheral Interrupts to D0 Multimedia Core

• M0 Firmware is hardcoded for 2 Mbps

• Not really fun to watch. But we use this for testing our 2 Mbps USB Serial Adapter.

When we power up Ox64 in Flashing Mode: (Boot Button pressed)

• Ox64 is ready for Firmware Flashing by the BL DevCube GUI Tool

• Firmware Flashing supports various Baud Rates: 230.4 kbps, 2 Mbps, ...

• But 2 Mbps will fail on macOS. That's why we Flash Firmware at 230.4 kbps.

  (Same problem when flashing BL602)

Serial Console is always 2 Mbps?

Yeah 2 Mbps is hardcoded in Ox64 Linux. Switching to other Baud Rates will show garbled text.

Thus our USB Serial Adapter must connect reliably to Ox64 at 2 Mbps.

Now we flash Ox64 and boot Linux...

Flash OpenSBI and U-Boot Bootloader to Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

Before booting Linux on Ox64, we flash OpenSBI + U-Boot Bootloader to D0 Multimedia Core, and the Peripheral Interrupt Firmware to M0 Wireless Core. From buildroot_bouffalo:

• d0_lowload_bl808_d0.bin: This is a very basic bootloader that loads opensbi, the kernel and dts files into ram

• m0_lowload_bl808_m0.bin: This firmware runs on M0 and forwards interupts to the D0 for several peripherals

• bl808-firmware.bin: An image containing OpenSBI, Uboot and uboot dtb files.

Here are the steps, based on the Official Flashing Instructions...

1. We tested with Pine64 Woodpecker CH340G USB Serial Adapter on macOS x64.

   Warning: Some USB Serial Adapters WON'T WORK!

   Probably because we are connecting at 2 Mbps, which might be too fast for some USB Serial Adapters.

   (Like this CP2102, which shows garbled text at 2 Mbps)

   

2. To Test our USB Serial Adapter: Connect the USB Serial Adapter to Ox64 Flashing UART (pic above)...

   • Flashing UART TX is GPIO 14 (Physical Pin 1)
   • Flashing UART RX is GPIO 15 (Physical Pin 2)
   • Remember to connect GND
   • Baud 2,000,000 (2 Mbps)

   Start the USB Serial Terminal (Flashing UART).

   Power up Ox64 via the Micro USB Port. Ox64 Green LED should light up.

   This Clickety Micro USB Cable is very handy for rebooting Ox64...

   

3. In the USB Serial Terminal (Flashing UART), we should see the Ox64 Factory Test Firmware...

   Build:19:50:39,Nov 20 2022
   Copyright (c) 2022 Bouffalolab team
   dynamic memory init success,heap size = 93 Kbyte 
   sig1:ffff32ff
   sig2:0000ffff
   Pong!
   Ping!
   

   (Source)

   If the text appears garbled: Try a different USB Serial Adapter. (See above)

   My prototype version shows this instead...

   Init CLI with event Driven
   start aos loop... 
   CLI RAW Data, c906
   /romfs/c906.bin not found!
   

   (Source)

4. Pre-Flash Check: Set BL808 board to programming mode

   • Remove the microSD Card
   • Press and Hold BOOT Button
   • Unplug and replug the Micro USB Port
   • Release BOOT button
   • Ox64 Green LED should turn on

   In the USB Serial Terminal (Flashing UART), we should see this...

   .
   

   Yep Ox64 is ready for flashing!

5. Now we prepare to flash:

   Disconnect the USB Serial Terminal (to release the Flashing UART)

   Set BL808 board to programming mode

   • Remove the microSD Card
   • Press and Hold BOOT Button
   • Unplug and replug the Micro USB Port
   • Release BOOT button
   • Ox64 Green LED should turn on

6. We download the Ox64 Binaries...

   • bl808-linux-pine64_ox64_full_defconfig.tar.gz

   From the latest Ox64 Linux Release...

   • openbouffalo/buildroot_bouffalo (Release v1.0.1)

   Unzip the download and we should see this...

   → ls -l firmware
      7340032  bl808-firmware.bin
        31360  d0_lowload_bl808_d0.bin
        65760  m0_lowload_bl808_m0.bin
     43859444  sdcard-pine64_ox64_full_defconfig.img.xz    

7. We'll run BouffaloLab DevCube for Flashing BL808.

   Only Ubuntu x64, macOS and Windows are supported.

   TODO: How to flash BL808 on Arm64 SBCs and Pinebook Pro? Sigh. See bflb-iot-tool / bflb-mcu-tool

8. Download Bouffalo Lab DevCube 1.8.3 from...

   openbouffalo.org/static-assets/bldevcube/BouffaloLabDevCube-v1.8.3.zip

   (1.8.4 and later won't work)

   May need to Grant Execute Permission...

   cd BouffaloLabDevCube-v1.8.3
   chmod +x BLDevCube-macos-x86_64
   ./BLDevCube-macos-x86_64

9. Run DevCube, select "BL808", and switch to "MCU" page

10. M0 Group: Group0

    Image Addr: 0x58000000

    PATH: Select "m0_lowload_bl808_m0.bin"

11. D0 Group: Group0

    Image Addr: 0x58100000

    PATH: Select "d0_lowload_bl808_d0.bin"

12. Set UART Rate to 230400.

    Don't set to 2000000, it will fail on macOS!

    (Same problem when flashing BL602)

13. Click "Create & Download" and wait until it's done

    (See the log)

14. Switch to "IOT" page

15. Enable 'Single Download'

    Set Address to 0x800000

    Select "bl808-firmware.bin"

16. Set UART Rate to 230400.

    Don't set to 2000000, it will fail on macOS!

    (Same problem when flashing BL602)

17. Click "Create & Download" again and wait until it's done

    (See the log)

18. Start the USB Serial Terminal (Flashing UART at 2 Mbps).

    Unplug and replug the Micro USB Port.

    (Don't press the Boot Button!)

19. On the USB Serial Terminal (Flashing UART) we should see...

    [I][] Powered by BouffaloLab
    [I][] Build:11:52:22,Mar  6 2023
    [I][] Copyright (c) 2023 OpenBouffalo team
    [I][] Copyright (c) 2022 Bouffalolab team
    [I][] =========== flash cfg ==============
    [I][] jedec id   0xEF6018
    [I][] mid            0xEF
    [I][] iomode         0x04
    [I][] clk delay      0x01
    [I][] clk invert     0x01
    [I][] read reg cmd0  0x05
    [I][] read reg cmd1  0x35
    [I][] write reg cmd0 0x01
    [I][] write reg cmd1 0x31
    [I][] qe write len   0x01
    [I][] cread support  0x00
    [I][] cread code     0xFF
    [I][] burst wrap cmd 0x77
    [I][] sector size:   0x04
    [I][] =====================================
    [I][] dynamic memory init success,heap size = 156 Kbyte 
    [I][MAIN] Starting Mailbox Handlers
    [I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
    [I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
    [I][MAIN] Running...
    [I][MBOX] Mailbox IRQ Stats:
    [I][MBOX] .Peripheral SDH (33): 0
    [I][MBOX] .Peripheral GPIO (60): 0
    [I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
    

    (Source)

    Yep we have flashed the OpenBouffalo Firmware successfully!

    

20. Connect our USB Serial Adapter to Ox64 Serial Console: (pic above)

    • Serial Console TX is GPIO 16 (Physical Pin 32)
    • Serial Console RX is GPIO 17 (Physical Pin 31)
    • Remember to connect GND
    • Baud 2,000,000 (2 Mbps)

    Start the USB Serial Terminal (Serial Console).

    Unplug and replug the Micro USB Port.

    (Don't press the Boot Button!)

21. On the USB Serial Terminal (Serial Console) we should see...

    U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)
    Card did not respond to voltage select! : -110
    BOOTP broadcast
    Retry time exceeded; starting again
    

    (Source)

    Which is OK because U-Boot Bootloader is waiting for a microSD Card.

22. If nothing appears...

    Check that we are using Bouffalo Lab DevCube 1.8.3

    (1.8.4 and later won't work)

    In BL Dev Cube, UART Rate (for MCU and IoT) should be 230400.

    Don't set to 2000000, it will fail on macOS!

    (Same problem when flashing BL602)

Let's load Ox64 Linux into a microSD Card...

Boot Linux on Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

Now that D0 Multimedia Core is flashed with OpenSBI and U-Boot Bootloader, we're ready to boot Linux on microSD!

Based on the Official Flashing Instructions...

1. Look for the microSD Image that we downloaded earlier...

   sdcard-pine64_ox64_full_defconfig.img.xz
   

   Uncompress the file to get...

   sdcard-pine64_ox64_full_defconfig.img
   

2. Flash the uncompressed image to your microSD card.

   You can use Balena Etcher, GNOME Disks or dd.

3. Insert the microSD Card into Ox64. (Pic above)

   

4. Connect our USB Serial Adapter to Ox64 Flashing UART: (pic above)

   • Flashing UART TX is GPIO 14 (Physical Pin 1)
   • Flashing UART RX is GPIO 15 (Physical Pin 2)
   • Remember to connect GND
   • Baud 2,000,000 (2 Mbps)

   Start the USB Serial Terminal (Flashing UART).

   Unplug and replug the Micro USB Port.

   (Don't press the Boot Button!)

5. On the USB Serial Terminal (Flashing UART) we should see the same thing as earlier...

   [I][MAIN] Starting Mailbox Handlers
   [I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
   [I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
   [I][MAIN] Running...
   [I][MBOX] Mailbox IRQ Stats:
   [I][MBOX] .Peripheral SDH (33): 0
   [I][MBOX] .Peripheral GPIO (60): 0
   [I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
   

   (Source)

   

6. Connect our USB Serial Adapter to Ox64 Serial Console: (pic above)

   • Serial Console TX is GPIO 16 (Physical Pin 32)
   • Serial Console RX is GPIO 17 (Physical Pin 31)
   • Remember to connect GND
   • Baud 2,000,000 (2 Mbps)

   Start the USB Serial Terminal (Serial Console).

   Unplug and replug the Micro USB Port.

   (Don't press the Boot Button!)

7. On the USB Serial Terminal (Serial Console) we should see...

   [I][] Powered by BouffaloLab
   [I][] Build:11:52:04,Mar  6 2023
   [I][] Copyright (c) 2023 OpenBouffalo team
   [I][] Copyright (c) 2022 Bouffalolab team
   [I][] dynamic memory init success,heap s[I][LowLoad] D0 start...
   [I][LowLoad] low_load start... 
   [I][LowLoad] Header at 0x5d5ff000
   [I][LowLoad] Section dtb(1) - Start 0x5d5ff100, Size 14314
   [I][LowLoad] Copying DTB to 0x51ff8000...0x51ffb7ea
   [I][LowLoad] Done!
   [I][LowLoad] Section OpenSBI(2) - Start 0x5d60f100, Size 109864
   [I][LowLoad] Copying OpenSBI to 0x3ef80000...0x3ef9ad28
   [I][LowLoad] Done!
   [I][LowLoad] Section Kernel(3) - Start 0x5d62f100, Size 315597
   [I][LowLoad] Uncompressing Kernel to 0x50000000...
   [I][LowLoad] Done!
   [I][LowLoad] CRC: 00000000
   [I][LowLoad] load time: 61306 us 
   [I][LowLoad] ing PMP
   [I][LowLoad] Booting OpenSBI at 0x000000003ef80000 with DTB at 0x51ff8000
   ...
   OpenSBI v1.2
   Platform Name             : Pine64 Ox64 (D0)
   Platform Features          medeleg
   Platform HART Count       : 1
   Platform IPI Device       : aclint-mswi
   Platform Timer Device     : aclint-mtimer @ 1000000Hz
   Platform Console Device   : bflb_uart
   Platform HSM Device       : ---
   Platform PMU Device       : ---
   Platform Reboot Device    : ---
   Platform Shutdown Device  : ---
   Firmware Base             : 0x3ef80000
   Firmware Size             : 200 KB
   Runtime SBI Version       : 1.0
   ...
   
   U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)
   DRAM:  64 MiB
   Core:  36 devices, 17 uclasses, devicetree: board
   MMC:   mmc@20060000: 0
   Loading Environment from FAT... Unable to read "uboot.env" from mmc0:2... 
   ...
   Starting kernel ...
   Linux version 6.2.0 (runner@fv-az587-938) (riscv64-unknown-linux-gnu-gcc (Xuantie-900 linux-5.10.4 glibc gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 Mon Mar  6 11:17:27 UTC 2023
   ...
   Welcome to Buildroot
   ox64 login: 
   

   (See the Complete Log)

   (Watch the Video on YouTube)

   Yep Linux is running on Ox64 yay! (Pic below)

8. If nothing appears...

   Check that we are using Bouffalo Lab DevCube 1.8.3

   (1.8.4 and later won't work)

   In BL Dev Cube, UART Rate (for MCU and IoT) should be 230400.

   Don't set to 2000000, it will fail on macOS!

   (Same problem when flashing BL602)

9. If we see...

   U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)
   Card did not respond to voltage select! : -110
   BOOTP broadcast
   Retry time exceeded; starting again
   

   (Source)

   Check that the microSD Card is inserted correctly. (Pic above)

10. TODO: TFTP Boot over Ethernet

Comment by @gamelaster...

"This is not hardware specific, but flasher specific. With blisp, I was able to get faster flashing working, but this is Apple's quirk. Or maybe not? Because FreeBSD need same quirks and exact buffer sizes as Apple."

Comment by @madushan1000...

"You can also use u-boot. https://github.com/openbouffalo/u-boot/releases/tag/bl808-2023-02-19 You can also get rid of mailbox, but you will have to build the kernel yourself https://github.com/openbouffalo/linux/tree/bl808/all"

Forward Peripheral Interrupts

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

TODO

[I][MAIN] Starting Mailbox Handlers
[I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
[I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
[I][MAIN] Running...
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX] .Peripheral SDH (33): 0
[I][MBOX] .Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0

(Source)

SDH: SD Card (SDIO) Host Controller (BL808 RM Page 561)

IRQ 60: GPIO_INT0 (IRQ_NUM_BASE+44) GPIO Interrupt (BL808 RM Page 44)

GPIO_INT0_IRQn

SDH is IRQ 33: SDH_IRQn

IRQ_NUM_BASE is 16 (BL808 RM Page 45)

m0_lowload

d0_lowload

Forward GPIO Interrupt

Forward SDH Interrupt

Setup SDH Interrupt

Other Interrupts (unused)

• UART2
• USB
• EMAC

Inspect the Linux Image for Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

Will Apache NuttX RTOS boot on Ox64 BL808?

Let's examine the Linux Kernel Image for Ox64, and we replicate the same format for NuttX. (Which is how we ported NuttX to 64-bit RISC-V Star64 JH7110 SBC)

We download the Ox64 Binaries...

• bl808-linux-pine64_ox64_full_defconfig.tar.gz

From the latest Ox64 Linux Release...

• openbouffalo/buildroot_bouffalo (Release v1.0.1)

Unzip it and mount the SD Card Image...

→ ls -l sdcard-pine64_ox64_full_defconfig     
-  13,154,816  Image
-       4,012  bl808-pine64-ox64.dtb
-       4,106  bl808-sipeed-m1s.dtb
-         350  boot-m1s.scr
-         352  boot-pine64.scr
-         352  boot.scr
d          96  extlinux

Dump the Image as hex...

→ hexdump sdcard-pine64_ox64_full_defconfig/Image
0000000 4d 5a 6f 10 20 08 01 00 00 00 20 00 00 00 00 00
0000010 00 80 cd 00 00 00 00 00 00 00 00 00 00 00 00 00
0000020 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000030 52 49 53 43 56 00 00 00 52 53 43 05 40 00 00 00

The Linux Kernel Image begins with this RISC-V Linux Image Header...

• "Boot Image Header in RISC-V Linux"

Here are the decoded bytes...

1. code0: Executable code

   (4 bytes, offset 0x00)

   4d 5a 6f 10 
   

2. code1: Executable code

   (4 bytes, offset 0x04)

   20 08 01 00 
   

3. text_offset: Image load offset, little endian

   (8 bytes, offset 0x08)

   00 00 20 00 00 00 00 00
   

4. image_size: Effective Image size, little endian

   (8 bytes, offset 0x10)

   00 80 cd 00 00 00 00 00
   

5. flags: Kernel flags, little endian

   (8 bytes, offset 0x18)

   00 00 00 00 00 00 00 00
   

6. version: Version of this header (MinL MinM . MajL MajM)

   (4 bytes, offset 0x20)

   02 00 00 00
   

7. res1: Reserved

   (4 bytes, offset 0x24)

   00 00 00 00
   

8. res2: Reserved

   (8 bytes, offset 0x28)

   00 00 00 00 00 00 00 00
   

9. magic: Magic number, little endian, "RISCV\x00\x00\x00"

   (8 bytes, offset 0x30)

   52 49 53 43 56 00 00 00
   

10. magic2: Magic number 2, little endian, "RSC\x05"

    (4 bytes, offset 0x38)

    52 53 43 05
    

11. res3: Reserved for PE COFF offset

    (4 bytes, offset 0x3C)

    40 00 00 00
    

Our NuttX Kernel shall recreate this RISC-V Linux Image Header. (Total 0x40 bytes)

(Or U-Boot Bootloader might refuse to boot NuttX)

Header Values are exactly the same as Star64. (Except the Image Size and Executable Code, since the Jump Address is different)

Thus we simply reuse the code from NuttX Star64!

Linux Device Tree for Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

TODO: Dump the Device Tree

dtc \
  -o bl808-pine64-ox64.dts \
  -O dts \
  -I dtb \
  bl808-pine64-ox64.dtb

Here's the Decompiled Device Tree: bl808-pine64-ox64.dts

TODO: Transmit to UART3 at 0x30002000. Reuse the BL602 UART Driver for NuttX.

serial@30002000 {
  compatible = "bflb,bl808-uart";
  reg = <0x30002000 0x1000>;
  interrupts = <0x14 0x04>;
  clocks = <0x04>;
  status = "okay";
  phandle = <0x0a>;
};

(Source)

TODO: Forward the Interrupts from M0 Wireless Core to D0 Multimedia Core via Mailbox / IPC (Where are the addresses documented?)

mailbox@30005000 {
  compatible = "bflb,bl808-ipc";
  reg = <
    0x30005000 0x20 
    0x30005020 0x20 
    0x2000a800 0x20 
    0x2000a820 0x20
  >;
  interrupts = <0x36 0x04>;
  interrupt-controller;
  #interrupt-cells = <0x03>;
  #mbox-cells = <0x02>;
  status = "okay";
  phandle = <0x03>;
};

(Source)

TODO: Print Debug Logs with OpenSBI

Boot Apache NuttX RTOS on Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

What happens if we boot Star64 NuttX on Ox64 BL808?

Let's find out!

## Download and build NuttX for Star64
git clone --branch ox64 https://github.com/lupyuen2/wip-pinephone-nuttx nuttx
git clone --branch ox64 https://github.com/lupyuen2/wip-pinephone-nuttx-apps apps
cd nuttx
tools/configure.sh star64:nsh
make

## Export the Binary Image to nuttx.bin
riscv64-unknown-elf-objcopy \
  -O binary \
  nuttx \
  nuttx.bin

## TODO: Prepare the microSD for Ox64 Linux
## https://lupyuen.github.io/articles/ox64#boot-linux-on-ox64

## Copy and overwrite the `Image` file on the microSD for Ox64 Linux
cp nuttx.bin Image
cp Image "/Volumes/NO NAME"
diskutil unmountDisk /dev/disk2

We boot NuttX on Ox64 via microSD... But Ox64 shows absolutely nothing!

Retrieving file: /extlinux/../Image
append: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
Retrieving file: /extlinux/../bl808-pine64-ox64.dtb
## Flattened Device Tree blob at 51ff8000
   Booting using the fdt blob at 0x51ff8000
Working FDT set to 51ff8000
   Loading Device Tree to 0000000053f22000, end 0000000053f25fab ... OK
Working FDT set to 53f22000
Starting kernel ...

(Source)

We're hoping that NuttX would crash and OpenSBI could print a meaningful Stack Trace. But nope! NuttX was probably stuck in a loop waiting for Star64 UART.

Let's print to the Ox64 Serial Console in the NuttX Boot Code (in RISC-V Assembly)...

Print to Ox64 Serial Console in NuttX Boot Code

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

How to print to the Ox64 Serial Console in the NuttX Boot Code? (RISC-V Assembly)

When we compare the BL808 and BL602 Reference Manuals, we discover that BL808 UART works the same way as BL602.

This is how the BL602 UART Driver prints to the Serial Console: bl602_serial.c

#define BL602_UART_FIFO_WDATA_OFFSET 0x000088  /* uart_fifo_wdata */
#define BL602_UART_FIFO_WDATA(n) (BL602_UART_BASE(n) + BL602_UART_FIFO_WDATA_OFFSET)

static void bl602_send(struct uart_dev_s *dev, int ch) {
  ...
  // Wait for FIFO to be empty
  while ((getreg32(BL602_UART_FIFO_CONFIG_1(uart_idx)) & \
    UART_FIFO_CONFIG_1_TX_CNT_MASK) == 0);
  // Write output to FIFO
  putreg32(ch, BL602_UART_FIFO_WDATA(uart_idx));
}

So for BL808, we simply write the character to...

• UART3 Base Address: 0x30002000 (from the Linux Device Tree earlier)

• Offset: 0x88

Based on Star64 Debug Code, we code this in RISC-V Assembly...

/* Load UART3 Base Address to Register t0 */
li  t0, 0x30002000

/* Load `1` to Register t1 */
li  t1, 0x31
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

/* Load `2` to Register t1 */
li  t1, 0x32
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

/* Load `3` to Register t1 */
li  t1, 0x33
/* Store byte from Register t1 to UART3 Base Address, Offset 0x88 */
sb  t1, 0x88(t0)

We insert the above code into the NuttX Boot Code: jh7110_head.S

Now NuttX prints to the Serial Console yay! (Pic below)

Starting kernel ...
123

(Source)

OpenSBI boots on Ox64 with Hart ID 0 (instead of 1), so we remove this code...

 /* We assume that OpenSBI has passed Hart ID (value 1) in Register a0.
   * But NuttX expects Hart ID to start at 0, so we subtract 1.
   */
  /* Previously: addi a0, a0, -1 */

(Source)

Update the NuttX Boot Address for Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

What is the Linux Boot Address for Ox64 BL808?

From the U-Boot Settings...

kernel_addr_r=0x50200000

Let's update the Boot Address in NuttX: ld.script

MEMORY
{
  kflash (rx) : ORIGIN = 0x50200000, LENGTH = 2048K   /* w/ cache */
  ksram (rwx) : ORIGIN = 0x50400000, LENGTH = 2048K   /* w/ cache */
  pgram (rwx) : ORIGIN = 0x50600000, LENGTH = 4096K   /* w/ cache */
  ramdisk (rwx) : ORIGIN = 0x50A00000, LENGTH = 6M   /* w/ cache */
}

TODO: Use up to 64 MB, the total RAM Size on Ox64

We make the same changes to the NuttX Config: nsh/defconfig

CONFIG_RAM_START=0x50200000
CONFIG_RAM_SIZE=1048576
CONFIG_ARCH_PGPOOL_PBASE=0x50600000
CONFIG_ARCH_PGPOOL_VBASE=0x50600000
CONFIG_ARCH_PGPOOL_SIZE=4194304

And the Memory Mapping: jh7110_mm_init.c

/* Map the whole I/O memory with vaddr = paddr mappings */
#define MMU_IO_BASE     (0x00000000)
#define MMU_IO_SIZE     (0x50000000)

TODO: What's the RAM Disk Address? It's missing from U-Boot Settings

/* Ramdisk Load Address from U-Boot */
#define RAMDISK_ADDR_R  (0x46100000)

(Source)

NuttX shows the same output as earlier, no change...

Starting kernel ...
123

(Source)

Let's fix the NuttX UART Driver...

Fix the NuttX UART Driver for Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

NuttX on Ox64 has been awfully quiet. How to fix the UART Driver so that NuttX can print things?

Ox64 is still running on the JH7110 UART Driver (16550). Let's make a quick patch so that we will see something on the Ox64 Serial Console...

We hardcode the UART3 Base Address (from above) and FIFO Offset for now: uart_16550.c

// Write one character to the UART (polled)
static void u16550_putc(FAR struct u16550_s *priv, int ch) {

  // Hardcode the UART3 Base Address and FIFO Offset
  *(volatile uint8_t *) 0x30002088 = ch; ////

  // Previously:
  // while ((u16550_serialin(priv, UART_LSR_OFFSET) & UART_LSR_THRE) == 0);
  // u16550_serialout(priv, UART_THR_OFFSET, (uart_datawidth_t)ch);
}

(Yeah the UART Buffer might overflow, we'll fix later)

We skip the reading and writing of other UART Registers, because we'll patch them later: uart_16550.c

// Read UART Register
static inline uart_datawidth_t u16550_serialin(FAR struct u16550_s *priv, int offset) {
  return 0; ////
  // Commented out the rest
}

// Write UART Register
static inline void u16550_serialout(FAR struct u16550_s *priv, int offset, uart_datawidth_t value) {
  // Commented out the rest
}

And we won't wait for UART Ready, since we're not accessing the Line Control Register: uart_16550.c

// Wait until UART is not busy. This is needed before writing to Line Control Register.
// Otherwise we will get spurious interrupts on Synopsys DesignWare 8250.
static int u16550_wait(FAR struct u16550_s *priv) {
  // Nopez! No waiting for now
  return OK; ////
}

Now NuttX prints our very first Stack Dump on Ox64 yay!

Starting kernel ...
123
ABC
riscv_exception: EXCEPTION: Load access fault. MCAUSE: 0000000000000005, EPC: 0000000050208086, MTVAL: 000000000c002104
riscv_exception: PANIC!!! Exception = 0000000000000005
_assert: Current Version: NuttX  12.0.3 93a92a7-dirty Nov  5 2023 11:27:46 risc-v
_assert: Assertion failed panic: at file: common/riscv_exception.c:85 task: Idle_Task process: Kernel 0x50200e28
up_dump_register: EPC: 0000000050208086
up_dump_register: A0: 000000000c002104 A1: ffffffffffffffff A2: 0000000000000001 A3: 0000000000000003
up_dump_register: A4: ffffffffffffffff A5: 8000000200046000 A6: 0000000000000000 A7: fffffffffffffff8
up_dump_register: T0: 00000000502000a8 T1: 0000000000000007 T2: 656d616e2d64746d T3: 0000000050407b10
up_dump_register: T4: 0000000050407b08 T5: 0000000053f23fff T6: 0000000053f33870
up_dump_register: S0: 0000000000000000 S1: 0000000050400140 S2: 0000000000000001 S3: 8000000200046002
up_dump_register: S4: 0000000050400070 S5: 00000000000001b6 S6: 0000000000000000 S7: 0000000000000000
up_dump_register: S8: 0000000053f7a15c S9: 0000000053fcf2e0 S10: 0000000000000001 S11: 0000000000000003
up_dump_register: SP: 0000000050407a00 FP: 0000000000000000 TP: 0000000000000000 RA: 0000000050204064

(Source)

Let's look up the RISC-V Exception Code Address 0x50208086 in our RISC-V Disassembly...

EXCEPTION: Load access fault 
MCAUSE: 0000000000000005 
EPC:    0000000050208086 
MTVAL:  000000000c002104

And the offending Data Address 0xc002104. (Which looks very familiar!)

Platform-Level Interrupt Controller for Ox64 BL808

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

Why did NuttX crash with this RISC-V Exception?

EXCEPTION: Load access fault 
MCAUSE: 0000000000000005 
EPC:    0000000050208086 
MTVAL:  000000000c002104

NuttX crashed when it tried to access invalid Data Address 0xc002104 from Code Address 0x50208086.

We look up Code Address 0x50208086 in our NuttX Disassembly...

000000005020807a <modifyreg32>:
up_irq_save():
nuttx/include/arch/irq.h:689
    5020807a:	4789                	li	a5,2
    5020807c:	1007b7f3          	csrrc	a5,sstatus,a5
modifyreg32():
nuttx/arch/risc-v/src/common/riscv_modifyreg32.c:52
{
  irqstate_t flags;
  uint32_t   regval;

  flags   = spin_lock_irqsave(NULL);
  regval  = getreg32(addr);
    50208080:	4118                	lw	a4,0(a0)
nuttx/arch/risc-v/src/common/riscv_modifyreg32.c:53
  regval &= ~clearbits;
    50208082:	fff5c593          	not	a1,a1
nuttx/arch/risc-v/src/common/riscv_modifyreg32.c:52
  regval  = getreg32(addr);
    50208086:	2701                	sext.w	a4,a4

Which comes from here: riscv_modifyreg32.c

 // Atomically modify the specified bits in a memory mapped register
void modifyreg32(uintptr_t addr, uint32_t clearbits, uint32_t setbits) {
  irqstate_t flags;
  uint32_t   regval;

  flags   = spin_lock_irqsave(NULL);
  // Crashes here because `addr` is invalid...
  regval  = getreg32(addr);
  regval &= ~clearbits;
  regval |= setbits;
  putreg32(regval, addr);
  spin_unlock_irqrestore(NULL, flags);
}

It's trying to modify a Memory-Mapped Register, and crashed.

But what Memory-Mapped Register?

The offending Data Address 0xc002104 actually comes from Star64 PLIC! (Platform-Level Interrupt Controller)

// From https://github.com/lupyuen2/wip-pinephone-nuttx/blob/ox64/arch/risc-v/src/jh7110/hardware/jh7110_memorymap.h#L30
#define JH7110_PLIC_BASE    0x0c000000

// From https://github.com/lupyuen2/wip-pinephone-nuttx/blob/ox64/arch/risc-v/src/jh7110/hardware/jh7110_plic.h#L34-L49
/* Interrupt Priority */
#define JH7110_PLIC_PRIORITY  (JH7110_PLIC_BASE + 0x000000)

/* Hart 1 S-Mode Interrupt Enable */
#define JH7110_PLIC_ENABLE1   (JH7110_PLIC_BASE + 0x002100)
#define JH7110_PLIC_ENABLE2   (JH7110_PLIC_BASE + 0x002104)

/* Hart 1 S-Mode Priority Threshold */
#define JH7110_PLIC_THRESHOLD (JH7110_PLIC_BASE + 0x202000)

/* Hart 1 S-Mode Claim / Complete */
#define JH7110_PLIC_CLAIM     (JH7110_PLIC_BASE + 0x202004)

The PLIC Base Address is different for BL808, let's change it.

What's the PLIC Base Address in Ox64 BL808?

PLIC Base Address is 0xe0000000, according to the Linux Device Tree: bl808-pine64-ox64.dts

interrupt-controller@e0000000 {
  compatible = "thead,c900-plic";
  reg = <0xe0000000 0x4000000>;
  interrupts-extended = <0x06 0xffffffff 0x06 0x09>;
  interrupt-controller;
  #address-cells = <0x00>;
  #interrupt-cells = <0x02>;
  riscv,ndev = <0x40>;
  phandle = <0x01>;
};

TODO: Why isn't this documented in XuanTie OpenC906 User Manual?

So we change the PLIC Base Address for Ox64: jh7110_memorymap.h

#define JH7110_PLIC_BASE    0xe0000000

TODO: Enable Scheduler Debug

Handle RISC-V Exceptions in NuttX

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

Now NuttX crashes at a different place, with IRQ 15...

123ABC
nx_start: Entry
up_irqinitialize: a
up_irqinitialize: b
up_irqinitialize: c
riscv_dispatch_irq: irq=15
irq_unexpected_isr: ERROR irq: 15
_assert: Current Version: NuttX  12.0.3 910bfca-dirty Nov  6 2023 15:23:11 risc-v
_assert: Assertion failed panic: at file: irq/irq_unexpectedisr.c:54 task: Idle_Task process: Kernel 0x50200e50

(Source)

What's IRQ 15?

From XuanTie OpenC906 User Manual (Page 21):

"Exception Vector ID 15: A store/atomic instruction page error exception."

This RISC-V Exception says that we tried to write to an invalid Data Address. And failed.

Where did it crash?

Based on our log, NuttX crashes before setting the PLIC!

From jh7110_irq.c

// Init the IRQs
void up_irqinitialize(void) {
  _info("a\n");////

  /* Disable S-Mode interrupts */
  _info("b\n");////
  up_irq_save();

  /* Disable all global interrupts */
  _info("c\n");////
  // Crashes here!
  putreg32(0x0, JH7110_PLIC_ENABLE1);
  putreg32(0x0, JH7110_PLIC_ENABLE2);

  /* Colorize the interrupt stack for debug purposes */
  ...

  /* Set irq threshold to 0 (permits all global interrupts) */
  _info("e\n");////
  putreg32(0, JH7110_PLIC_THRESHOLD);

  /* Attach the common interrupt handler */
  _info("f\n");////
  riscv_exception_attach();

But it's a RISC-V Exception! Shouldn't NuttX dump this as a proper exception?

See the riscv_exception_attach() above? It happens AFTER the crash! This means NuttX hasn't properly initialised the Exception Handlers, when the crash happened.

Let's init the Exception Handlers earlier: jh7110_irq.c

// Init the IRQs
void up_irqinitialize(void) {
  _info("a\n");////

  /* Disable S-Mode interrupts */
  _info("b\n");////
  up_irq_save();

  /* Attach the common interrupt handler */
  _info("f\n");////
  // Init the Exception Handlers here
  riscv_exception_attach();

  /* Disable all global interrupts */
  _info("c\n");////
  // Crashes here!
  putreg32(0x0, JH7110_PLIC_ENABLE1);
  putreg32(0x0, JH7110_PLIC_ENABLE2);

riscv_exception_attach() will handle all RISC-V Exceptions, including Store/AMO Page Fault (IRQ 15): riscv_exception.c

// Attach standard exception with suitable handler
void riscv_exception_attach(void) {
  // Handle Store/AMO Page Fault (IRQ 15)
  irq_attach(RISCV_IRQ_STOREPF, riscv_exception, NULL);

Now we see the Store/AMO Page Fault Exception!

up_irqinitialize: c
riscv_dispatch_irq: irq=15
riscv_exception: 
EXCEPTION: Store/AMO page fault
MCAUSE: 000000000000000f
EPC:    0000000050207e6a
MTVAL:  00000000e0002100

(Source)

Code Address is 0x50207e6a, from our PLIC Code...

nuttx/arch/risc-v/src/chip/jh7110_irq.c:62
  putreg32(0x0, JH7110_PLIC_ENABLE1);
    50207e64:	700017b7          	lui	a5,0x70001
    50207e68:	0786                	slli	a5,a5,0x1
    50207e6a:	1007a023          	sw	zero,256(a5) # 70001100 <__ramdisk_end+0x1e601100>

The offending Data Address is 0xe0002100. Which is our BL808 PLIC!

Add PLIC to I/O Memory Map

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

But is 0xe0002100 accessible?

Ah we forgot to add it to the I/O Memory Map! Let's fix it: jh7110_mm_init.c

/* Map the whole I/O memory with vaddr = paddr mappings */
#define MMU_IO_BASE     (0x00000000)
#define MMU_IO_SIZE     (0xf0000000)

(Doesn't look right, but we'll fix later)

Now NuttX boots further! And tries to register IRQ 57 for the Star64 UART Interrupt...

up_irqinitialize: c
up_irqinitialize: d
up_irqinitialize: e
up_irqinitialize: g
irq_attach: irq=17, isr=0x50207eee
up_enable_irq: irq=17
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
irq_attach: irq=57, isr=0x502041fe
up_enable_irq: irq=57
riscv_dispatch_irq: irq=5
riscv_exception: 
EXCEPTION: Load access fault
MCAUSE: 0000000000000005
EPC:    0000000050208342
MTVAL:  00000000e0002104

(Source)

But it crashes while accessing the PLIC at another address: 0xe0002104.

Are we tired of PLIC yet?

Yeah let's fix PLIC later. The entire UART Driver will be revamped anyway, including the UART Interrupt.

Let's disable the UART Interrupt for now: uart_16550.c

// Attach the UART Interrupt for Star64
static int u16550_attach(struct uart_dev_s *dev) {
  // Don't attach the interrupt
  // Previously: ret = irq_attach(priv->irq, u16550_interrupt, dev);

  // Don't enable the interrupt
  // Previously: up_enable_irq(priv->irq);

Fail to Load Initial RAM Disk

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

Now NuttX boots even further yay! But crashes in the NuttX Bringup...

up_irqinitialize: c
up_irqinitialize: d
up_irqinitialize: e
up_irqinitialize: g
irq_attach: irq=17, isr=0x50207e64
up_enable_irq: irq=17
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_lowpri: Starting low-priority kernel worker thread(s)
_assert: Current Version: NuttX  12.0.3 b244f85-dirty Nov  6 2023 17:35:34 risc-v
_assert: Assertion failed ret >= 0: at file: init/nx_bringup.c:283 task: AppBringUp process: Kernel 0x5020107e

(Source)

Because it couldn't map the Initial RAM Disk: nx_bringup.c

/* Mount the file system containing the init program. */
ret = nx_mount(CONFIG_INIT_MOUNT_SOURCE, CONFIG_INIT_MOUNT_TARGET,
  CONFIG_INIT_MOUNT_FSTYPE, CONFIG_INIT_MOUNT_FLAGS,
  CONFIG_INIT_MOUNT_DATA);
DEBUGASSERT(ret >= 0);

That's because we haven't loaded the Initial RAM Disk! Let's fix this later.

NuttX Boot Flow for Ox64 BL808

(To see the NuttX Source Code: Right-click the Node and select "Open Link")

flowchart TD
START --> bl808_head["NuttX Boot Code: \n bl808_head \n (Prints 123)"]
click bl808_head href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_head.S#L41-L156" "arch/risc-v/src/bl808/bl808_head.S" _blank

bl808_head --> bl808_start["NuttX Start Code: \n bl808_start"]
click bl808_start href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L254-L297" "arch/risc-v/src/bl808/bl808_start.c" _blank

bl808_start --> bl808_start_s["Start Supervisor Mode: \n bl808_start_s \n (Prints ABC)"]
click bl808_start_s href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L200-L254" "arch/risc-v/src/bl808/bl808_start.c" _blank

bl808_start_s --> riscv_earlyserialinit["Early Serial Init: \n riscv_earlyserialinit"]
click riscv_earlyserialinit href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L297-L314" "arch/risc-v/src/bl808/bl808_start.c" _blank

bl808_start_s --> bl808_mm_init["Memory Mgmt Init: \n bl808_mm_init \n (Map the Memory Mgmt Unit)"]
click bl808_mm_init href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_mm_init.c#L278-L298" "arch/risc-v/src/bl808/bl808_mm_init.c" _blank

bl808_start_s --> nx_start["Start NuttX: \n nx_start \n (Starts many things)"]
click nx_start href "https://github.com/apache/nuttx/blob/master/sched/init/nx_start.c#L298-L713" "sched/init/nx_start.c" _blank

riscv_earlyserialinit --> bl808_earlyserialinit["UART Early Init: \n bl808_earlyserialinit \n (Setup the UART)"]
click bl808_earlyserialinit href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_serial.c#L700-L720" "drivers/serial/uart_16550.c" _blank

nx_start --> up_irqinitialize["IRQ Init: \n up_irqinitialize"]
click up_irqinitialize href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_irq.c#L41-L103" "arch/risc-v/src/bl808/bl808_irq.c#" _blank

nx_start --> nx_bringup["Bringup NuttX: \n nx_bringup"]
click nx_bringup href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L379-L468" "sched/init/nx_bringup.c" _blank

up_irqinitialize --> riscv_exception_attach["Attach RISC-V Exceptions: \n riscv_exception_attach \n (Attach the RISC-V Exception Handlers)"]
click riscv_exception_attach href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_exception.c#L89-L142" "arch/risc-v/src/common/riscv_exception.c" _blank

up_irqinitialize --> up_initialize["Init NuttX: \n up_initialize"]
click up_initialize href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_initialize.c#L70-L132" "arch/risc-v/src/common/riscv_initialize.c" _blank

up_initialize --> riscv_serialinit["Serial Init: \n riscv_serialinit"]
click riscv_serialinit href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L314-L327" "arch/risc-v/src/bl808/bl808_start.c" _blank

riscv_serialinit --> bl808_serialinit["UART Init: \n bl808_serialinit \n (Register /dev/console)"]
click bl808_serialinit href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_serial.c#L720-L762" "drivers/serial/uart_16550.c" _blank

nx_bringup --> nx_create_initthread["Create Init Thread: \n nx_create_initthread \n (Create AppBringUp thread)"]
click nx_create_initthread href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L336-L375" "sched/init/nx_bringup.c" _blank

nx_create_initthread --> nx_start_task["Start NuttX Task: \n nx_start_task"]
click nx_start_task href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L310-L336" "sched/init/nx_bringup.c" _blank

nx_start_task --> nx_start_application["Start Application: \n nx_start_application"]
click nx_start_application href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L213-L310" "sched/init/nx_bringup.c" _blank

nx_start_application --> board_late_initialize["Board Late Init: board_late_initialize \n (Mount Initial RAM Disk)"]
click board_late_initialize href "https://github.com/apache/nuttx/blob/master/boards/risc-v/bl808/ox64/src/bl808_appinit.c#L134-L167" "boards/risc-v/bl808/ox64/src/bl808_appinit.c" _blank

board_late_initialize --> nx_mount["Mount RAM Disk: nx_mount \n (Fails because Initial RAM Disk is missing)"]
click nx_mount href "https://github.com/apache/nuttx/blob/master/fs/mount/fs_mount.c#L260-L514" "fs/mount/fs_mount.c" _blank

Loading

Read the article...

• "Ox64 BL808 RISC-V SBC: Starting Apache NuttX RTOS"

What happens exactly when NuttX boots on Ox64?

In this article, NuttX has booted plenty of code on Ox64. Here's the flow of the NuttX Code that boots on Ox64...

NuttX Boot Code: bl808_head calls...

• NuttX Start Code: bl808_start which calls...

• Start Supervisor Mode: bl808_start_s which prints "ABC" and calls...

• Early Serial Init: riscv_earlyserialinit (see below) and...

  Memory Mgmt Init: bl808_mm_init (to init the Memory Mgmt Unit) and...

  Start NuttX: nx_start (see below)

Early Serial Init: riscv_earlyserialinit calls...

• UART Early Init: bl808_earlyserialinit

  (To setup the UART)

Memory Mgmt Init: bl808_mm_init inits the Memory Mgmt Unit by calling...

• MMU Map Region: mmu_ln_map_region (to map a Memory Region) and...

  MMU Set Entry: mmu_ln_setentry (to set a Page Table Entry)

  (More about the Memory Mgmt Unit)

Start NuttX: nx_start does many things and calls...

• IRQ Init: up_irqinitialize (see below) and...

  Bringup NuttX: nx_bringup (see below)

IRQ Init: up_irqinitialize calls...

• Attach RISC-V Exceptions: riscv_exception_attach (to attach the RISC-V Exception Handlers) and...

  Init NuttX: up_initialize (see below)

Init NuttX: up_initialize calls...

• Serial Init: riscv_serialinit which calls...

• UART Init: bl808_serialinit

  (To register "/dev/console")

Bringup NuttX: nx_bringup calls...

• Create Init Thread: nx_create_initthread (to create "AppBringUp" thread) which calls...

• Start NuttX Task: nx_start_task (to start NuttX Task) which calls...

• Start Application: nx_start_application which calls...

• Board Late Init: board_late_initialize (to mount Initial RAM Disk) and...

  Mount RAM Disk: nx_mount (to mount ROM File System from Initial RAM Disk)

  (Which fails because our Initial RAM Disk is missing)

  (Which prevents NuttX Shell from starting)

  (How NuttX Shell should be started)

  (How NuttX Apps are started)

Therefore we expect NuttX to boot completely on Ox64 when we've implemented...

• Initial RAM Disk for Ox64

• UART Driver and UART Interrupts

• Memory Map might need fixing too

NuttX UART Driver for Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: UART Interrupt and Platform-Level Interrupt Controller (PLIC)"

BL808 UART is mostly identical to BL602 UART, so we ported the NuttX BL602 UART Driver to BL808.

Here's the UART Driver ported to BL808: bl602_serial.c

We hardcoded the UART3 Base Address: bl602_uart.h

#define BL602_UART0_BASE 0x30002000
#define BL602_UART_BASE(n) (BL602_UART0_BASE)
// Previously: #define BL602_UART_BASE(n)    (BL602_UART0_BASE + (n * (BL602_UART1_BASE - BL602_UART0_BASE)))

We fixed the NuttX Start Code to call our new UART Driver: jh7110_start.c

void riscv_earlyserialinit(void) {
  bl602_earlyserialinit();
}

void riscv_serialinit(void) {
  bl602_serialinit();
}

We disabled UART Interrupts for now: bl602_attach and bl602_detach

And the UART Driver works! (See the log)

TODO: /dev/ttyS0 is missing

TODO: Enable UART Interrupts

Initial RAM Disk for Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

Two ways we can load the Initial RAM Disk...

1. Load the Initial RAM Disk from a Separate File: initrd (similar to Star64)

   This means we need to modify the U-Boot Script: boot-pine64.scr

   And make it load the initrd file into RAM.

   (Which is good for separating the NuttX Kernel and NuttX Apps)

   OR...

2. Append the Initial RAM Disk to the NuttX Kernel Image

   So the U-Boot Bootloader will load (one-shot into RAM) the NuttX Kernel + Initial RAM Disk.

   And we reuse the existing U-Boot Config on the microSD Card: extlinux/extlinux.conf

   (Which might be more efficient for our Limited RAM)

   (See the U-Boot Boot Flow)

   TODO: Can we mount the File System directly from the NuttX Kernel Image in RAM? Without copying to the RAM Disk Memory Region?

We'll do the Second Method, since we are low on RAM. Like this...

## Export the Binary Image to `nuttx.bin`
riscv64-unknown-elf-objcopy \
  -O binary \
  nuttx \
  nuttx.bin

## Insert 64 KB of zeroes after Binary Image for Kernel Stack
head -c 65536 /dev/zero >/tmp/nuttx.zero

## Append Initial RAM Disk to Binary Image
cat nuttx.bin /tmp/nuttx.zero initrd \
  >Image

## Overwrite the Linux Image on Ox64 microSD
cp Image "/Volumes/NO NAME/"

This is how we copy the initrd in RAM to the Memory Region for the RAM Disk: jh7110_start.c

static void jh7110_copy_ramdisk(void) {
  // Based on ROM FS Format: https://docs.kernel.org/filesystems/romfs.html
  // After _edata, search for "-rom1fs-". This is the RAM Disk Address.
  // Stop searching after 64 KB.
  extern uint8_t _edata[];
  extern uint8_t _sbss[];
  extern uint8_t _ebss[];
  const char *header = "-rom1fs-";
  uint8_t *ramdisk_addr = NULL;
  for (uint8_t *addr = _edata; addr < (uint8_t *)JH7110_IDLESTACK_TOP + (65 * 1024); addr++) {
    if (memcmp(addr, header, strlen(header)) == 0) {
      ramdisk_addr = addr;
      break;
    }
  }
  // Check for Missing RAM Disk
  if (ramdisk_addr == NULL) { _info("Missing RAM Disk"); }
  DEBUGASSERT(ramdisk_addr != NULL); 

  // RAM Disk must be after Idle Stack
  if (ramdisk_addr <= (uint8_t *)JH7110_IDLESTACK_TOP) { _info("RAM Disk must be after Idle Stack"); }
  DEBUGASSERT(ramdisk_addr > (uint8_t *)JH7110_IDLESTACK_TOP);

  // Read the Filesystem Size from the next 4 bytes, in Big Endian
  // Add 0x1F0 to Filesystem Size
  const uint32_t size =
    (ramdisk_addr[8] << 24) + 
    (ramdisk_addr[9] << 16) + 
    (ramdisk_addr[10] << 8) + 
    ramdisk_addr[11] + 
    0x1F0;
  _info("size=%d\n", size);

  // Filesystem Size must be less than RAM Disk Memory Region
  DEBUGASSERT(size <= (size_t)__ramdisk_size);

  // Before Copy: Verify the RAM Disk Image to be copied
  verify_image(ramdisk_addr);

  // Copy the Filesystem Size to RAM Disk Start
  // Warning: __ramdisk_start overlaps with ramdisk_addr + size
  // memmove is aliased to memcpy, so we implement memmove ourselves
  local_memmove((void *)__ramdisk_start, ramdisk_addr, size);

  // Before Copy: Verify the copied RAM Disk Image
  verify_image(__ramdisk_start);
}

We copy the initrd at the very top of our NuttX Start Code, before erasing the BSS (in case it corrupts our RAM Disk, but actually it shouldn't): jh7110_start.c

// NuttX Start Code
void jh7110_start(int mhartid) {
  DEBUGASSERT(mhartid == 0); /* Only Hart 0 supported for now */
  if (0 == mhartid) {
    /* Copy the RAM Disk */
    jh7110_copy_ramdisk();

    /* Clear the BSS */
    jh7110_clear_bss();

NuttX mounts the RAM Disk from the Memory Region later during startup: jh7110_appinit.c

// After NuttX has booted...
void board_late_initialize(void) {
  // Mount the RAM Disk
  mount_ramdisk();
}

// Mount the RAM Disk
int mount_ramdisk(void) {
  desc.minor    = RAMDISK_DEVICE_MINOR;
  desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
  desc.sectsize = SECTORSIZE;
  desc.image    = __ramdisk_start;
  ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&desc);

And NuttX mounts our RAM Disk successfully!

jh7110_copy_ramdisk: _edata=0x50400258, _sbss=0x50400290, _ebss=0x50407000, JH7110_IDLESTACK_TOP=0x50407c00
jh7110_copy_ramdisk: ramdisk_addr=0x50408288
jh7110_copy_ramdisk: size=8192016
jh7110_copy_ramdisk: Before Copy: ramdisk_addr=0x50408288
jh7110_copy_ramdisk: After Copy: __ramdisk_start=0x50a00000
...
elf_initialize: Registering ELF
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x5040c618
elf_read: Read 64 bytes from offset 0

(Source)

Why did we insert 64 KB of zeroes after the NuttX Binary Image, before the initrd Initial RAM Disk?

## Insert 64 KB of zeroes after Binary Image for Kernel Stack
head -c 65536 /dev/zero >/tmp/nuttx.zero

## Append Initial RAM Disk to Binary Image
cat nuttx.bin /tmp/nuttx.zero initrd \
  >Image

When we refer to the NuttX Log and the NuttX Linker Script...

// End of Data Section
_edata=0x50400258

// Start of BSS Section
_sbss=0x50400290

// End of BSS Section
_ebss=0x50407000

// Top of Idle Stack
JH7110_IDLESTACK_TOP=0x50407c00

// We located the initd after the Top of Idle Stack
ramdisk_addr=0x50408288, size=8192016

// And we copied initrd to the Memory Region for the RAM Disk
__ramdisk_start=0x50a00000

Which says...

1. The NuttX Binary Image nuttx.bin terminates at _edata. (End of Data Section)

2. If we append initrd directly to the end of nuttx.bin, it will collide with the BSS Section and the Idle Stack. And initrd will get overwritten by NuttX.

3. Best place to append initrd is after the Top of Idle Stack. Which is located 32 KB after _edata. (End of Data Section)

4. That's why we inserted a padding of 64 KB between nuttx.bin and initrd. So it won't collide with BSS and Idle Stack.

5. Our code locates initrd (searching by Magic Number "-rom1fs-"). And copies initrd to __ramdisk_start. (Memory Region for the RAM Disk)

6. NuttX mounts the RAM Disk from __ramdisk_start. (Memory Region for the RAM Disk)

But 64 KB sounds so arbitrary. What if the parameters change?

That's why we have a Runtime Check: jh7110_start.c

  // RAM Disk must be after Idle Stack
  if (ramdisk_addr <= (uint8_t *)JH7110_IDLESTACK_TOP) { _info("RAM Disk must be after Idle Stack"); }
  DEBUGASSERT(ramdisk_addr > (uint8_t *)JH7110_IDLESTACK_TOP);

Why did we call local_memmove to copy initrd to __ramdisk_start? Why not memcpy?

That's because initrd overlaps with __ramdisk_start!

ramdisk_addr = 0x50408288, size = 8192016
ramdisk_addr + size = 0x50bd8298
Which is AFTER __ramdisk_start (0x50a00000)

memcpy won't work with Overlapping Memory Regions. So we wrote our own: jh7110_start.c

// From libs/libc/string/lib_memmove.c
static FAR void *local_memmove(FAR void *dest, FAR const void *src, size_t count) {
  FAR char *d;
  FAR char *s;
  DEBUGASSERT(dest > src);
  d = (FAR char *) dest + count;
  s = (FAR char *) src + count;

  while (count--) {
    d -= 1;
    s -= 1;
    // TODO: Very strange. This needs to be volatile or C Compiler will replace this by memcpy.
    volatile char c = *s;
    *d = c;
  }
  return dest;
}

We're sure that it works?

That's why we called verify_image to do a simple integrity check on initrd, before and after copying. And that's how we discovered that memcpy doesn't work. From jh7110_start.c

// Verify that image is correct
static void verify_image(uint8_t *addr) {
  // Verify that the Byte Positions below (offset by 1) contain 0x0A
  for (int i = 0; i < sizeof(search_addr) / sizeof(search_addr[0]); i++) {
    const uint8_t *p = addr + search_addr[i] - 1;
    if (*p != 0x0A) { _info("No Match: %p\n", p); }
  }
}

// Byte Positions (offset by 1) of 0x0A in initrd. Extracted from:
// grep --binary-files=text -b -o A initrd
const uint32_t search_addr[] =
{
76654,
78005,
79250,
...
7988897,
7992714,
};

But NuttX fails to start our NuttX Shell (NSH) ELF Executable from "/system/bin/init"...

elf_read: Read 3392 bytes from offset 3385080
elf_addrenv_select: ERROR: up_addrenv_text_enable_write failed: -22
elf_load: ERROR: elf_addrenv_select() failed: -22
...
elf_loadbinary: Failed to load ELF program binary: -22
exec_internal: ERROR: Failed to load program '/system/bin/init': -22
_assert: Current Version: NuttX  12.0.3 8017bd9-dirty Nov 10 2023 22:50:07 risc-v
_assert: Assertion failed ret > 0: at file: init/nx_bringup.c:302 task: AppBringUp process: Kernel 0x502014ea

(Source)

Maybe because NuttX is trying to map the User Address Space 0xC000 0000: nsh/defconfig

CONFIG_ARCH_TEXT_VBASE=0xC0000000
CONFIG_ARCH_TEXT_NPAGES=128
CONFIG_ARCH_DATA_VBASE=0xC0100000
CONFIG_ARCH_DATA_NPAGES=128
CONFIG_ARCH_HEAP_VBASE=0xC0200000
CONFIG_ARCH_HEAP_NPAGES=128

But our Kernel Memory Space already extends to 0xF000 0000? (Because of the PLIC at 0xE000 0000)

From jh7110_mm_init.c:

/* Map the whole I/O memory with vaddr = paddr mappings */
#define MMU_IO_BASE     (0x00000000)
#define MMU_IO_SIZE     (0xf0000000)

Let's disable PLIC, and exclude PLIC from Memory Map. Will the NuttX Shell start?

Yep it does! (See the log)

Now we fix the Memory Map...

NuttX Memory Map for Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: Sv39 Memory Management Unit"

To fix the NuttX Memory Map for Ox64, let's trace the MMU Page Table Entries. From the MMU Log...

Map the I/O Region (Level 1)

From the MMU Log...

mmu_ln_map_region: 
  ptlevel=1, lnvaddr=0x50407000, paddr=0, vaddr=0, size=0x40000000, mmuflags=0x26

mmu_ln_setentry: 
  ptlevel=1, lnvaddr=0x50407000, paddr=0, vaddr=0, mmuflags=0x26
mmu_ln_setentry: 
  index=0, paddr=0, mmuflags=0xe7, pte_addr=0x50407000, pte_val=0xe7

mmuflags=0x26 means Read + Write + Global

Map the PLIC (Level 2)

From the MMU Log...

mmu_ln_map_region: 
  ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, size=0x10000000, mmuflags=0x26

mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, mmuflags=0x26
mmu_ln_setentry: index=0x100, paddr=0xe0000000, mmuflags=0xe7, pte_addr=0x50403800, pte_val=0x380000e7
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0200000, vaddr=0xe0200000, mmuflags=0x26
mmu_ln_setentry: index=0x101, paddr=0xe0200000, mmuflags=0xe7, pte_addr=0x50403808, pte_val=0x380800e7
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0400000, vaddr=0xe0400000, mmuflags=0x26
mmu_ln_setentry: index=0x102, paddr=0xe0400000, mmuflags=0xe7, pte_addr=0x50403810, pte_val=0x381000e7
...
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xefe00000, vaddr=0xefe00000, mmuflags=0x26
mmu_ln_setentry: index=0x17f, paddr=0xefe00000, mmuflags=0xe7, pte_addr=0x50403bf8, pte_val=0x3bf800e7

mmuflags=0x26 means Read + Write + Global

Connect the Level 1 and Level 2 Page Tables for PLIC

From the MMU Log...

mmu_ln_setentry: 
  ptlevel=1, lnvaddr=0x50407000, paddr=0x50403000, vaddr=0xe0000000, mmuflags=0x20

mmu_ln_setentry: 
  index=0x3, paddr=0x50403000, mmuflags=0x21, pte_addr=0x50407018, pte_val=0x14100c21

mmuflags=0x20 means PTE_G: Global Mapping.

Map the Kernel Text (Levels 2 & 3)

From the MMU Log...

// Level 2
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x50404000, vaddr=0x50200000, mmuflags=0x0
mmu_ln_setentry: index=0x81, paddr=0x50404000, mmuflags=0x1, pte_addr=0x50406408, pte_val=0x14101001

// Level 3
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50404000, paddr=0x50200000, vaddr=0x50200000, mmuflags=0x2a
mmu_ln_setentry: index=0, paddr=0x50200000, mmuflags=0xeb, pte_addr=0x50404000, pte_val=0x140800eb

mmu_ln_setentry: ptlevel=3, lnvaddr=0x50404000, paddr=0x5020100, vaddr=0x50201000, mmuflags=0x2a
mmu_ln_setentry: index=0x1, paddr=0x50201000, mmuflags=0xeb, pte_addr=0x50404008, pte_val=0x140804eb

mmu_ln_setentry: ptlevel=3, lnvaddr=0x50404000, paddr=0x50202000, vaddr=0x50202000, mmuflags=0x2a
mmu_ln_setentry: index=0x2, paddr=0x50202000, mmuflags=0xeb, pte_addr=0x50404010, pte_val=0x140808eb
...
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50404000, paddr=0x503ff000, vaddr=0x503ff000, mmuflags=0x2a
mmu_ln_setentry: index=0x1ff, paddr=0x503ff000, mmuflags=0xeb, pte_addr=0x50404ff8, pte_val=0x140ffceb

mmuflags=0x0 means PTE is a pointer to the next level of the page table

mmuflags=0x2a means Read + Execute + Global

Map the Kernel Data (Levels 2 & 3)

From the MMU Log...

// Level 2
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x50405000, vaddr=0x50400000, mmuflags=0x0
mmu_ln_setentry: index=0x82, paddr=0x50405000, mmuflags=0x1, pte_addr=0x50406410, pte_val=0x14101401

// Level 3
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50405000, paddr=0x50400000, vaddr=0x5040000, mmuflags=0x26
mmu_ln_setentry: index=0, paddr=0x50400000, mmuflags=0xe7, pte_addr=0x50405000, pte_val=0x141000e7
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50405000, paddr=0x50401000, vaddr=0x50401000, mmuflags=0x26
mmu_ln_setentry: index=0x1, paddr=0x50401000, mmuflags=0xe7, pte_addr=0x50405008, pte_val=0x141004e7
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50405000, paddr=0x50402000, vaddr=0x50402000, mmuflags=0x26
mmu_ln_setentry: index=0x2, paddr=0x50402000, mmuflags=0xe7, pte_addr=0x50405010, pte_val=0x141008e7
...
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50405000, paddr=0x505ff000, vaddr=0x505ff000, mmuflags=0x26
mmu_ln_setentry: index=0x1ff, paddr=0x505ff000, mmuflags=0xe7, pte_addr=0x50405ff8, pte_val=0x1417fce7

mmuflags=0x0 means PTE is a pointer to the next level of the page table

mmuflags=0x26 means Read + Write + Global

Connect the Level 1 and Level 2 Page Tables

From the MMU Log...

mmu_ln_setentry: 
  ptlevel=1, lnvaddr=0x50407000, paddr=0x50406000, vaddr=0x50200000, mmuflags=0x20

mmu_ln_setentry: 
  index=0x1, paddr=0x50406000, mmuflags=0x21, pte_addr=0x50407008, pte_val=0x14101821

mmuflags=0x20 means PTE_G: Global Mapping.

And PTE is a pointer to the next level of the page table.

Which means that Virtual Address 0x5020 0000 points to the L2 Page Table 0x5040 6000

Map the Page Pool (Level 2)

From the MMU Log...

mmu_ln_map_region: ptlevel=2, lnvaddr=0x50406000, paddr=0x50600000, vaddr=0x50600000, size=0x1400000, mmuflags=0x26

mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x50600000, vaddr=0x50600000, mmuflags=0x26
mmu_ln_setentry: index=0x83, paddr=0x50600000, mmuflags=0xe7, pte_addr=0x50406418, pte_val=0x141800e7

mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x50800000, vaddr=0x50800000, mmuflags=0x26
mmu_ln_setentry: index=0x84, paddr=0x50800000, mmuflags=0xe7, pte_addr=0x50406420, pte_val=0x142000e7

mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x50a00000, vaddr=0x50a00000, mmuflags=0x26
mmu_ln_setentry: index=0x85, paddr=0x50a00000, mmuflags=0xe7, pte_addr=0x50406428, pte_val=0x142800e7
...
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50406000, paddr=0x51800000, vaddr=0x51800000, mmuflags=0x26
mmu_ln_setentry: index=0x8c, paddr=0x51800000, mmuflags=0xe7, pte_addr=0x50406460, pte_val=0x146000e7

mmuflags=0x26 means Read + Write + Global

Map the User Code, Data and Heap (Levels 1, 2, 3)

From the MMU Log...

nx_start_application: Starting init task: /system/bin/init
// Level 1 (Code, Data, Heap)
// `mmuflags=0x0` means it's a link to a PTE
// `vaddr=0x80100000` will be aligned to 0x80000000
mmu_ln_setentry: ptlevel=1, lnvaddr=0x50600000, paddr=0x50601000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: index=0x2, paddr=0x50601000, mmuflags=0x1, pte_addr=0x50600010, pte_val=0x14180401

// Level 2 (Code, Data)
// `mmuflags=0x0` means it's a link to a PTE
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x50602000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: index=0, paddr=0x50602000, mmuflags=0x1, pte_addr=0x50601000, pte_val=0x14180801

// Level 3 (Data)
// `mmuflags=0x16` means Read + Write + User
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50602000, paddr=0x50603000, vaddr=0x80100000, mmuflags=0x16
mmu_ln_setentry: index=0x100, paddr=0x50603000, mmuflags=0xd7, pte_addr=0x50602800, pte_val=0x14180cd7

// Level 3 (Code)
// `mmuflags=0x1a` means Read + Execute + User
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50602000, paddr=0x50604000, vaddr=0x80000000, mmuflags=0x1a
mmu_ln_setentry: index=0, paddr=0x50604000, mmuflags=0xdb, pte_addr=0x50602000, pte_val=0x141810db

mmu_ln_setentry: ptlevel=3, lnvaddr=0x50602000, paddr=0x50605000, vaddr=0x80001000, mmuflags=0x1a
mmu_ln_setentry: index=0x1, paddr=0x50605000, mmuflags=0xdb, pte_addr=0x50602008, pte_val=0x141814db
...
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50602000, paddr=0x50619000, vaddr=0x80015000, mmuflags=0x1a
mmu_ln_setentry: index=0x15, paddr=0x50619000, mmuflags=0xdb, pte_addr=0x506020a8, pte_val=0x141864db

// Level 3 (Data)
// `mmuflags=0x16` means Read + Write + User
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50602000, paddr=0x5061a000, vaddr=0x80101000, mmuflags=0x16
mmu_ln_setentry: index=0x101, paddr=0x5061a000, mmuflags=0xd7, pte_addr=0x50602808, pte_val=0x141868d7

// Level 2 (Heap)
// `mmuflags=0x0` means it's a link to a PTE
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x5061b000, vaddr=0x80200000, mmuflags=0x0
mmu_ln_setentry: index=0x1, paddr=0x5061b000, mmuflags=0x1, pte_addr=0x50601008, pte_val=0x14186c01

// Level 3 (Heap)
// `mmuflags=0x16` means Read + Write + User
mmu_ln_setentry: ptlevel=3, lnvaddr=0x5061b000, paddr=0x5061c000, vaddr=0x80200000, mmuflags=0x16
mmu_ln_setentry: index=0, paddr=0x5061c000, mmuflags=0xd7, pte_addr=0x5061b000, pte_val=0x141870d7

mmu_ln_setentry: ptlevel=3, lnvaddr=0x5061b000, paddr=0x5061d000, vaddr=0x80201000, mmuflags=0x16
mmu_ln_setentry: index=0x1, paddr=0x5061d000, mmuflags=0xd7, pte_addr=0x5061b008, pte_val=0x141874d7
...
mmu_ln_setentry: ptlevel=3, lnvaddr=0x5061b000, paddr=0x5069c000, vaddr=0x80280000, mmuflags=0x16
mmu_ln_setentry: index=0x80, paddr=0x5069c000, mmuflags=0xd7, pte_addr=0x5061b400, pte_val=0x141a70d7

Page Tables

What are the 3 Levels of Page Tables?

From the MMU Log...

lnvaddr=0x50407000 is m_l1_pgtable (Level 1 Page Table for Kernel)

lnvaddr=0x50406000 is m_l2_pgtable (Level 2 Page Table for Kernel)

lnvaddr=0x50404000 is m_l3_pgtable (Level 3 Page Table for Kernel)

lnvaddr=0x50403000 is m_int_l3_pgtable (Level 3 Page Table for Kernel PLIC)

lnvaddr=0x5061b000 is User L1 Page Table for Code, Data, Heap

lnvaddr=0x50601000 is User L2 Page Table for Code, Data

lnvaddr=0x50602000 is User L3 Page Table for Code, Data

lnvaddr=0x50601000 is User L2 Page Table for Heap

lnvaddr=0x5061b000 is User L3 Page Table for Heap

Fix the NuttX Memory Map for PLIC

Read the article...

• "RISC-V Ox64 BL808 SBC: Sv39 Memory Management Unit"

Can we add L1 for 0xE000 0000 to access PLIC?

Nope it fails. Because 0xE000 0000 is not aligned to 0x4000 0000 for L1.

But when we add L1 for 0xC000 0000, it works!

  // Map PLIC
  mmu_ln_map_region(1, PGT_L1_VBASE, 0xC0000000, 0xC0000000,
                    0x40000000, MMU_IO_FLAGS);

But NuttX Apps need to access 0xC000 0000. So NuttX Apps will fail...

nx_start_application: Starting init task: /system/bin/init
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x50602000, vaddr=0xc0100000, mmuflags=0x0
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x5061b000, vaddr=0xc0200000, mmuflags=0x0
elf_addrenv_select: ERROR: up_addrenv_text_enable_write failed: -22
elf_load: ERROR: elf_addrenv_select() failed: -22
elf_loadbinary: Failed to load ELF program binary: -22
exec_internal: ERROR: Failed to load program '/system/bin/init': -22
_assert: Current Version: NuttX  12.0.3 47ec850-dirty Nov 11 2023 20:24:22 risc-v
_assert: Assertion failed ret > 0: at file: init/nx_bringup.c:302 task: AppBringUp process: Kernel 0x502014b0

(Source)

What if we map PLIC as L2 at 0xE000 0000? By reusing the existing L2 Page Tables?

  // Map PLIC as L2, by reusing the existing L2 Page Tables
  map_region(0xE0000000, 0xE0000000, 0x10000000, MMU_IO_FLAGS);

This fails with IRQ 5. The existing L2 Page Tables can't accommodate PLIC at 0xE000 0000.

What if we add L1 for 0xC000 0000? And move apps to 0x8000 0000?

From jh7110_mm_init.c:

  // Map PLIC as L1
  // This will waste a whole chunk of L1 Addresses (Size 0x4000 0000) just for PLIC:
  mmu_ln_map_region(1, PGT_L1_VBASE, 0xC0000000, 0xC0000000,
                    0x40000000, MMU_IO_FLAGS);

NSH starts OK! Though we wasted a whole chunk of L1 Addresses (Size 0x4000 0000) just for PLIC...

jh7110_kernel_mappings: map PLIC as L1
mmu_ln_map_region: ptlevel=1, lnvaddr=0x50406000, paddr=0xc0000000, vaddr=0xc0000000, size=0x40000000, mmuflags=0x26
mmu_ln_setentry: ptlevel=1, lnvaddr=0x50406000, paddr=0xc0000000, vaddr=0xc0000000, mmuflags=0x26
...
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
mmu_ln_setentry: ptlevel=1, lnvaddr=0x50600000, paddr=0x50601000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x50602000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x5061b000, vaddr=0x80200000, mmuflags=0x0
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
up_exit: TCB=0x50409900 exiting
riscv_dispatch_irq: irq=8

NuttShell (NSH) NuttX-12.0.3
riscv_dispatch_irq: irq=8
nsh> riscv_dispatch_irq: irq=8
riscv_dispatch_irq: irq=8
nx_start: CPU0: Beginning Idle Loop

(Source)

What if we map PLIC as a new L2 Page Table: Interrupt L2?

From jh7110_mm_init.c:

// Interrupt L2 Table Table
#define PGT_INT_L2_PBASE (uintptr_t)&m_int_l2_pgtable
#define PGT_INT_L2_VBASE PGT_INT_L2_PBASE
#define PGT_INT_L2_SIZE (512)  /* Enough to map 1 GiB */
static size_t m_int_l2_pgtable[PGT_INT_L2_SIZE] locate_data(".pgtables");

// Map PLIC as Interrupt L2 Table Table
mmu_ln_map_region(2, PGT_INT_L2_PBASE, 0xE0000000, 0xE0000000, 0x10000000,
                  MMU_IO_FLAGS);

// Connect the L1 and Interrupt L2 Page Tables for PLIC
mmu_ln_setentry(1, PGT_L1_VBASE, PGT_INT_L2_PBASE, 0xE0000000, PTE_G);

NSH starts OK yay!

jh7110_kernel_mappings: map PLIC as Interrupt L2
mmu_ln_map_region: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, size=0x10000000, mmuflags=0x26
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, mmuflags=0x26
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0200000, vaddr=0xe0200000, mmuflags=0x26
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xe0400000, vaddr=0xe0400000, mmuflags=0x26
...

mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xefa00000, vaddr=0xefa00000, mmuflags=0x26
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xefc00000, vaddr=0xefc00000, mmuflags=0x26
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50403000, paddr=0xefe00000, vaddr=0xefe00000, mmuflags=0x26
jh7110_kernel_mappings: connect the L1 and Interrupt L2 page tables for PLIC
mmu_ln_setentry: ptlevel=1, lnvaddr=0x50407000, paddr=0x50403000, vaddr=0xe0000000, mmuflags=0x20
...
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
mmu_ln_setentry: ptlevel=1, lnvaddr=0x50600000, paddr=0x50601000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x50602000, vaddr=0x80100000, mmuflags=0x0
mmu_ln_setentry: ptlevel=2, lnvaddr=0x50601000, paddr=0x5061b000, vaddr=0x80200000, mmuflags=0x0
elf_symname: Symbol has no name
elf_symvalue: SHN_UNDEF: Failed to get symbol name: -3
elf_relocateadd: Section 2 reloc 2: Undefined symbol[0] has no name: -3
up_exit: TCB=0x5040a900 exiting
riscv_dispatch_irq: irq=8

NuttShell (NSH) NuttX-12.0.3
riscv_dispatch_irq: irq=8
nsh> riscv_dispatch_irq: irq=8
riscv_dispatch_irq: irq=8
nx_start: CPU0: Beginning Idle Loop

(Source)

TODO: Who maps the User Memory for lnvaddr=0x50600000?

Ox64 boots to NuttX Shell

Compute Level 1 PTE:

Based the Updated MMU Log with PTE...

connect the L1 and Interrupt L2 page tables for PLIC
mmu_ln_setentry:
  ptlevel=1, lnvaddr=0x50407000, paddr=0x50403000, vaddr=0xe0000000, mmuflags=0x20
mmu_ln_setentry: 
  index=0x3, 
  paddr=0x50403000, 
  mmuflags=0x21, 
  pte_addr=0x50407018, 
  pte_val=0x14100c21

To compute the Address of Level 1 PTE:

• pte_addr = lnvaddr + (index * 8) = 0x50407018

  (8 bytes per PTE)

• index = vpn >> 18 = 3

  (Extract Bits 18 to 26 to get Level 1 Index)

• vpn = vaddr >> 12 = 0xe0000

  (4096 bytes per Memory Page)

To compute the Value of Level 1 PTE:

• pte_val = (ppn << 10) | mmuflags = 0x14100c21

  (Shift 10 bits to accommodate MMU Flags)

• ppn = paddr >> 12 = 0x50403

  (1<<12 bits per Memory Page)

mmuflags=0x21 means 0b100001:

• NOT Dirty

• NOT Accessed

• Global Mapping

• NOT User-Accessible

• NOT eXecutable

• NOT Writeable

• NOT Readable

• Valid

Compute Level 2 PTE:

Based the Updated MMU Log with PTE...

map PLIC as Interrupt L2
mmu_ln_map_region:
  ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, size=0x10000000, mmuflags=0x26
mmu_ln_setentry:
  ptlevel=2, lnvaddr=0x50403000, paddr=0xe0000000, vaddr=0xe0000000, mmuflags=0x26
mmu_ln_setentry:
  index=0x100,
  paddr=0xe0000000,
  mmuflags=0xe7,
  pte_addr=0x50403800,
  pte_val=0x380000e7

To compute the Address of Level 2 PTE:

• pte_addr = lnvaddr + (index * 8) = 0x50403800

  (8 bytes per PTE)

• index = (vpn >> 9) & 0b111111111 = 0x100

  (Extract Bits 9 to 17 to get Level 2 Index)

• vpn = vaddr >> 12 = 0xe0000

  (4096 bytes per Memory Page)

To compute the Value of Level 2 PTE:

• pte_val = (ppn << 10) | mmuflags = 0x380000e7

  (Shift 10 bits to accommodate MMU Flags)

• ppn = paddr >> 12 = 0xe0000

  (4096 bytes per Memory Page)

mmuflags=0xe7 means 0b11100111:

• Dirty

• Accessed

• Global Mapping

• NOT User-Accessible

• NOT eXecutable

• Writeable

• Readable

• Valid

Compute Level 3 PTE:

Based the Updated MMU Log with PTE...

map kernel text
mmu_ln_setentry: 
  ptlevel=2, lnvaddr=0x50406000, paddr=0x50404000, vaddr=0x50200000, mmuflags=0x0
mmu_ln_setentry: 
  index=0x81, paddr=0x50404000, mmuflags=0x1, pte_addr=0x50406408, pte_val=0x14101001
...
mmu_ln_setentry: ptlevel=3, lnvaddr=0x50404000, paddr=0x5020100, vaddr=0x50201000, mmuflags=0x2a
mmu_ln_setentry: 
  index=0x1, 
  paddr=0x50201000, 
  mmuflags=0xeb, 
  pte_addr=0x50404008, 
  pte_val=0x140804eb

To compute the Address of Level 3 PTE:

• pte_addr = lnvaddr + (index * 8) = 0x50404008

  (8 bytes per PTE)

• index = vpn & 0b111111111 = 0x1

  (Extract Bits 0 to 8 to get Level 3 Index)

• vpn = vaddr >> 12 = 0x50201

  (4096 bytes per Memory Page)

To compute the Value of Level 3 PTE:

• pte_val = (ppn << 10) | mmuflags = 0x140804eb

  (Shift 10 bits to accommodate MMU Flags)

• ppn = paddr >> 12 = 0x50201

  (4096 bytes per Memory Page)

mmuflags=0xeb means 0b11101011:

• Dirty

• Accessed

• Global Mapping

• NOT User-Accessible

• eXecutable

• NOT Writeable

• Readable

• Valid

TODO: G Bit

"The G bit designates a global mapping. Global mappings are those that exist in all address spaces. For non-leaf PTEs, the global setting implies that all mappings in the subsequent levels of the page table are global. Note that failing to mark a global mapping as global merely reduces performance, whereas marking a non-global mapping as global is a software bug that, after switching to an address space with a different non-global mapping for that address range, can unpredictably result in either mapping being used."

TODO: SATP Log

NuttX swaps the SATP Register

Start NuttX Apps on Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

NuttX Kernel starts a NuttX App (in ELF Format) by calling...

• ELF Loader: g_elfbinfmt, which calls...

• elf_loadbinary, which calls...

• elf_load, which calls...

• elf_addrenv_alloc, which calls...

• up_addrenv_create, which calls...

  (Also calls mmu_satp_reg to set SATP Register)

• create_region, which calls...

• mmu_ln_setentry to populate the Page Table Entries

Who calls ELF Loader g_elfbinfmt to start the NuttX App?

Earlier we stepped through the Boot Sequence for NuttX...

• "NuttX Boot Flow"

Right after that, nx_bringup calls...

• nx_create_initthread, which calls...

• nx_start_application, which calls...

• exec_spawn, which calls...

• exec_internal, which calls...

• load_module and...

  exec_module

load_module calls...

• load_absmodule, which calls...

• binfmt_s.load, which calls...

• ELF Loader: g_elfbinfmt to load the ELF File (explained above)

Simple NuttX App for Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

What's inside the simplest app for NuttX?

From hello_main.c

int main(int argc, FAR char *argv[]) {
  printf("Hello, World!!\n");
  return 0;
}

Here's the RISC-V Disassembly: hello.S

000000000000003e <main>:
main():
apps/examples/hello/hello_main.c:37
/****************************************************************************
 * hello_main
 ****************************************************************************/

int main(int argc, FAR char *argv[])
{
  3e:	1141                	addi	sp,sp,-16
apps/examples/hello/hello_main.c:38
  printf("Hello, World!!\n");
  40:	00000517          	auipc	a0,0x0	40: R_RISCV_PCREL_HI20	.LC0
	40: R_RISCV_RELAX	*ABS*
  44:	00050513          	mv	a0,a0	44: R_RISCV_PCREL_LO12_I	.L0 
	44: R_RISCV_RELAX	*ABS*

0000000000000048 <.LVL1>:
apps/examples/hello/hello_main.c:37
{
  48:	e406                	sd	ra,8(sp)
apps/examples/hello/hello_main.c:38
  printf("Hello, World!!\n");
  4a:	00000097          	auipc	ra,0x0	4a: R_RISCV_CALL	puts
	4a: R_RISCV_RELAX	*ABS*
  4e:	000080e7          	jalr	ra # 4a <.LVL1+0x2>

0000000000000052 <.LVL2>:
apps/examples/hello/hello_main.c:40
  return 0;
}
  52:	60a2                	ld	ra,8(sp)
  54:	4501                	li	a0,0
  56:	0141                	addi	sp,sp,16
  58:	8082                	ret

We see that main calls...

• puts, which calls...

• lib_fwrite_unlocked, which calls...

• stream->fs_iofunc.write OR...

  write (See below)

TODO: Which one?

TODO: _start prepares sig_trampoline and calls main

This code doesn't look right...

apps/examples/hello/hello_main.c:38
  printf("Hello, World!!\n");
  4a:	00000097          	auipc	ra,0x0
  4e:	000080e7          	jalr	ra

That's because this is Relocatable Code. The auipc offset will be fixed up by the NuttX ELF Loader when it loads this code into User Memory.

The Relocation Info shows that 0x0 will be replaced by the address of puts...

  4a:	00000097          	auipc	ra,0x0
  4a: R_RISCV_CALL	puts
  4e:	000080e7          	jalr	ra

Why puts instead of printf?

The GCC Compiler has cleverly optimised away printf to become puts.

If we do this...

  printf("Hello, World %s!!\n", "Luppy");

Then printf will appear in our disassembly.

NuttX App calls NuttX Kernel

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

How does a NuttX App make a System Call to NuttX Kernel?

Our NuttX App calls write, which is a Proxy Version...

// From nuttx/syscall/proxies/PROXY_write.c
/* Auto-generated write proxy file -- do not edit */
#include <nuttx/config.h>
#include <unistd.h>
#include <syscall.h>

ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {
  return (ssize_t)sys_call3((unsigned int)SYS_write, (uintptr_t)parm1, (uintptr_t)parm2, (uintptr_t)parm3);
}

Proxy for write calls sys_call3, which makes an ecall to NuttX Kernel...

// Make an `ecall` to NuttX Kernel
static inline uintptr_t sys_call3(unsigned int nbr, uintptr_t parm1,
                                  uintptr_t parm2, uintptr_t parm3) {
  register long r0 asm("a0") = (long)(nbr);
  register long r1 asm("a1") = (long)(parm1);
  register long r2 asm("a2") = (long)(parm2);
  register long r3 asm("a3") = (long)(parm3);
  asm volatile
    (
     "ecall"
     :: "r"(r0), "r"(r1), "r"(r2), "r"(r3)
     : "memory"
     );
  asm volatile("nop" : "=r"(r0));
  return r0;
}

Here's how it works...

We can see the list of proxies in the RISC-V Disassembly of the NuttX Apps...

## Hello App
→ grep PROXY hello.S
PROXY__assert.c
PROXY__exit.c
PROXY_clock_gettime.c
PROXY_gettid.c
PROXY_lseek.c
PROXY_nxsem_wait.c
PROXY_sem_clockwait.c
PROXY_sem_destroy.c
PROXY_sem_post.c
PROXY_sem_trywait.c
PROXY_task_setcancelstate.c
PROXY_write.c

## NuttX Shell NSH
→ grep PROXY init.S
PROXY__assert.c
PROXY__exit.c
PROXY_clock_gettime.c
PROXY_gettid.c
PROXY_nxsem_wait.c
PROXY_sched_getparam.c
PROXY_sched_setparam.c
PROXY_sem_clockwait.c
PROXY_sem_destroy.c
PROXY_sem_post.c
PROXY_sem_trywait.c
PROXY_task_setcancelstate.c
PROXY_write.c
PROXY_boardctl.c
PROXY_clock_nanosleep.c
PROXY_close.c
PROXY_ftruncate.c
PROXY_get_environ_ptr.c
PROXY_getenv.c
PROXY_gethostname.c
PROXY_ioctl.c
PROXY_kill.c
PROXY_lseek.c
PROXY_lstat.c
PROXY_mkdir.c
PROXY_mount.c
PROXY_nx_pthread_create.c
PROXY_nx_pthread_exit.c
PROXY_nx_vsyslog.c
PROXY_open.c
PROXY_pgalloc.c
PROXY_posix_spawn.c
PROXY_pthread_detach.c
PROXY_read.c
PROXY_rename.c
PROXY_rmdir.c
PROXY_sched_getscheduler.c
PROXY_sched_lock.c
PROXY_sched_unlock.c
PROXY_setenv.c
PROXY_stat.c
PROXY_sysinfo.c
PROXY_umount2.c
PROXY_unlink.c
PROXY_unsetenv.c
PROXY_waitpid.c

Kernel Handles App Call

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

Our App makes an ecall to jump to NuttX Kernel...

What happens on the other side?

Remember the Proxy Function from earlier? Now we do the exact opposite in our Stub Function (that runs in the Kernel)...

// From nuttx/syscall/stubs/STUB_write.c
/* Auto-generated write stub file -- do not edit */
#include <nuttx/config.h>
#include <stdint.h>
#include <unistd.h>

uintptr_t STUB_write(int nbr, uintptr_t parm1, uintptr_t parm2, uintptr_t parm3) {
  return (uintptr_t)write((int)parm1, (FAR const void *)parm2, (size_t)parm3);
}

To handle IRQ 8 (RISCV_IRQ_ECALLU), NuttX does...

• Attach RISCV_IRQ_ECALLU, which calls...

• riscv_swint, which calls...

• dispatch_syscall, which calls Kernel Function Stub and...

• sys_call2 with A0=SYS_syscall_return (3), which calls...

• riscv_perform_syscall, which calls...

• riscv_swint with IRQ 0, to return from Syscall

How did we figure out that 63 is the System Call Number for "write"?

OK this part gets tricky. Below is the Enum that defines all System Call Numbers: syscall.h and syscall_lookup.h

// System Call Enum sequentially assigns
// all System Call Numbers (8 to 147-ish)
enum {
  ...
  SYSCALL_LOOKUP(close, 1)
  SYSCALL_LOOKUP(ioctl, 3)
  SYSCALL_LOOKUP(read,  3)
  SYSCALL_LOOKUP(write, 3)
  ...
};

But it's an Enum, numbered sequentially from 8 to 147-ish. We won't actually see 63 in the NuttX Source Code.

When we look at the RISC-V Disassembly hello.S:

ssize_t write(int parm1, FAR const void * parm2, size_t parm3)
{
 dcc:	872a                	mv	a4,a0

0000000000000dce <.LVL1>:
 dce:	87ae                	mv	a5,a1

0000000000000dd0 <.LVL2>:
 dd0:	86b2                	mv	a3,a2

0000000000000dd2 <.LBB4>:
sys_call3():
/Users/Luppy/ox64/nuttx/include/arch/syscall.h:252
  register long r0 asm("a0") = (long)(nbr);
 dd2:	03f00513          	li	a0,63

We see that SYS_write = 63. Also from hello.S:

 <2><66e7>: Abbrev Number: 6 (DW_TAG_enumerator)
    <66e8>   DW_AT_name        : (indirect string, offset: 0x4b98): SYS_write
    <66ec>   DW_AT_const_value : 63

What happens when we run this?

In make menuconfig, enable CONFIG_DEBUG_SYSCALL_INFO: Build Setup > Debug Options > Syscall Debug Features > Syscall Warning / Error / Info

The NuttX Log shows the System Calls from NuttX App to NuttX Kernel...

riscv_dispatch_irq: irq=8
riscv_swint: Entry: regs: 0x5040bcb0 cmd: 63
up_dump_register: EPC: 00000000800019b2
up_dump_register: A0: 000000000000003f A1: 0000000000000001 A2: 000000008000ad00 A3: 000000000000001e
up_dump_register: A4: 0000000000000001 A5: 000000008000ad00 A6: 0000000000000000 A7: fffffffffffffff8
up_dump_register: T0: 0000000050212a20 T1: 0000000000000007 T2: 0000000000000000 T3: 0000000080200908
up_dump_register: T4: 0000000080200900 T5: 0000000000000000 T6: 0000000000000000
up_dump_register: S0: 00000000802005c0 S1: 0000000080202010 S2: 0000000080202010 S3: 0000000000000000
up_dump_register: S4: 0000000000000001 S5: 0000000000000000 S6: 0000000000000000 S7: 0000000000000000
up_dump_register: S8: 0000000000000000 S9: 0000000000000000 S10: 0000000000000000 S11: 0000000000000000
up_dump_register: SP: 0000000080202b70 FP: 00000000802005c0 TP: 0000000000000000 RA: 0000000080001a6a
riscv_swint: SWInt Return: 37
STUB_write: nbr=440, parm1=1, parm2=8000ad00, parm3=1e

NuttShell (NSH) NuttX-12.0.3
riscv_swint: Entry: regs: 0x5040baa0 cmd: 3
up_dump_register: EPC: 0000000080001a6a
up_dump_register: A0: 0000000000000003 A1: 000000005040bbec A2: 000000000000001e A3: 0000000000000000
up_dump_register: A4: 0000000000007fff A5: 0000000000000001 A6: 0000000000000009 A7: fffffffffffffff8
up_dump_register: T0: 000000000000002e T1: 000000000000006a T2: 00000000000001ff T3: 000000000000006c
up_dump_register: T4: 0000000000000068 T5: 0000000000000009 T6: 000000000000002a
up_dump_register: S0: 00000000802005c0 S1: 0000000080202010 S2: 0000000080202010 S3: 0000000000000000
up_dump_register: S4: 0000000000000001 S5: 0000000000000000 S6: 0000000000000000 S7: 0000000000000000
up_dump_register: S8: 0000000000000000 S9: 0000000000000000 S10: 0000000000000000 S11: 0000000000000000
up_dump_register: SP: 000000005040bcb0 FP: 00000000802005c0 TP: 0000000000000000 RA: 0000000080001a6a
riscv_swint: SWInt Return: 1e

What are the Register Values?

Before System Call:

• A0 = 0x3f (SYS_write)

• A1 = 1 (stdout)

• A2 = 0x8000ad00 (g_nshgreeting)

• A3 = 0x1e (length)

• nbr = 440 (Offset for the stub lookup table, g_stublookup)

After System Call:

• A0 = 3 (SYS_syscall_return)

• Returns 0x1E = 30 chars, including linefeeds before and after

Kernel Starts a NuttX App

(To see the NuttX Source Code: Right-click the Node and select "Open Link")

flowchart TD
START --> nx_bringup["NuttX Bringup: \n nx_bringup"]
click nx_bringup href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L373-L458" "sched/init/nx_bringup.c" _blank

nx_bringup --> nx_create_initthread["Create Init Thread: \n nx_create_initthread \n (Create the Init Thread)"]
click nx_create_initthread href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L330-L367" "sched/init/nx_bringup.c" _blank

nx_create_initthread --> nx_start_application["Start App: \n nx_start_application \n (Start NuttX Shell)"]
click nx_start_application href "https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L212C1-L302" "sched/init/nx_bringup.c" _blank

nx_start_application --> exec_spawn["Exec Spawn: \n exec_spawn \n (Start the app)"]
click exec_spawn href "https://github.com/apache/nuttx/blob/master/binfmt/binfmt_exec.c#L183-L223" "binfmt/binfmt_exec.c" _blank

exec_spawn --> exec_internal["Exec Internal: \n exec_internal \n (Start the app)"]
click exec_internal href "https://github.com/apache/nuttx/blob/master/binfmt/binfmt_exec.c#L42-L179" "binfmt/binfmt_exec.c" _blank

exec_internal --> load_module["Load Module: \n load_module \n (Load the app)"]
click load_module href "https://github.com/apache/nuttx/blob/master/binfmt/binfmt_loadmodule.c#L136-L225" "binfmt/binfmt_loadmodule.c" _blank

exec_internal --> exec_module["Execute Module: \n exec_module \n (Execute the app)"]
click exec_module href "https://github.com/apache/nuttx/blob/master/binfmt/binfmt_execmodule.c#L190-L450" "binfmt/binfmt_execmodule.c" _blank

load_module --> load_absmodule["Load Absolute Module: \n load_absmodule \n (Load an absolute path)"]
click load_absmodule href "https://github.com/apache/nuttx/blob/master/binfmt/binfmt_loadmodule.c#L83-L132" "binfmt/binfmt_loadmodule.c" _blank

load_absmodule --> binfmt_s.load["Load Binary Format: \n binfmt_s.load \n (Load a binary module)"]
click binfmt_s.load href "https://github.com/apache/nuttx/blob/master/include/nuttx/binfmt/binfmt.h#L122-L148" "include/nuttx/binfmt/binfmt.h" _blank

binfmt_s.load --> g_elfbinfmt["ELF Loader: \n g_elfbinfmt \n (Load the ELF File)"]
click g_elfbinfmt href "https://github.com/apache/nuttx/blob/master/binfmt/elf.c#L84-L94" "binfmt/elf.c" _blank

g_elfbinfmt --> elf_loadbinary["Load ELF Binary: \n elf_loadbinary \n (Load the ELF Binary)"]
click elf_loadbinary href "https://github.com/apache/nuttx/blob/master/binfmt/elf.c#L225-L355" "binfmt/elf.c" _blank

elf_loadbinary --> elf_load["Load ELF: \n elf_load \n (Load the ELF Binary)"]
click elf_load href "https://github.com/apache/nuttx/blob/master/binfmt/libelf/libelf_load.c#L297-L445" "binfmt/libelf/libelf_load.c" _blank

elf_load --> elf_addrenv_alloc["Allocate Address Env: \n elf_addrenv_alloc \n (Allocate the Address Env)"]
click elf_addrenv_alloc href "https://github.com/apache/nuttx/blob/master/binfmt/libelf/libelf_addrenv.c#L56-L178" "binfmt/libelf/libelf_addrenv.c" _blank

elf_addrenv_alloc --> up_addrenv_create["Create Address Env: \n up_addrenv_create \n (Create the Address Env)"]
click up_addrenv_create href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_addrenv.c#L339-L490" "arch/risc-v/src/common/riscv_addrenv.c" _blank

elf_addrenv_alloc --> mmu_satp_reg["Set SATP Register: \n mmu_satp_reg \n (Set SATP Register)"]
click mmu_satp_reg href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_mmu.h#L152-L176" "arch/risc-v/src/common/riscv_mmu.h" _blank

up_addrenv_create --> create_region["Create MMU Region: \n create_region \n (Create the MMU Region)"]
click create_region href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_addrenv.c#L213-L310" "arch/risc-v/src/common/riscv_addrenv.c" _blank

create_region --> mmu_ln_setentry["Set MMU Page Table Entry: \n mmu_ln_setentry \n (Populate the Page Table Entries)"]
click mmu_ln_setentry href "https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_mmu.c#L62-L109" "arch/risc-v/src/common/riscv_mmu.c" _blank

Loading

Read the article...

• "RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"

Phew so NuttX Apps can call NuttX Kernel...

But how does NuttX Kernel start a NuttX App?

Earlier we stepped through the Boot Sequence for NuttX...

• "NuttX Boot Flow"

Right after that, NuttX Bringup (nx_bringup) calls...

• Create Init Thread: nx_create_initthread (to create the Init Thread), which calls...

• Start App: nx_start_application (to start NuttX Shell), which calls...

• Exec Spawn: exec_spawn (to start the app), which calls...

• Exec Internal: exec_internal (to start the app), which calls...

• Load Module: load_module (to load the app, see below) and...

  Execute Module: exec_module (to execute the app)

To load a NuttX App module: load_module calls...

• Load Absolute Module: load_absmodule (to load an absolute path), which calls...

• Load Binary Format: binfmt_s.load (to load a binary module), which calls...

• ELF Loader: g_elfbinfmt (to load the ELF File, see below)

To load the ELF File: ELF Loader g_elfbinfmt calls...

• Load ELF Binary: elf_loadbinary (to load the ELF Binary), which calls...

• Load ELF: elf_load (to load the ELF Binary), which calls...

• Allocate Address Env: elf_addrenv_alloc (to allocate the Address Env), which calls...

• Create Address Env: up_addrenv_create (to create the Address Env), which calls...

  (Also calls mmu_satp_reg to set SATP Register)

• Create MMU Region: create_region (to create the MMU Region), which calls...

• Set MMU Page Table Entry: mmu_ln_setentry (to populate the Page Table Entries)

There's plenty happening inside Execute Module: exec_module. But we won't explore today.

UART Interrupt for Ox64 BL808

Read the article...

• "RISC-V Ox64 BL808 SBC: UART Interrupt and Platform-Level Interrupt Controller (PLIC)"

Let's fix the UART Interrupts for NuttX on Ox64 BL808!

We fix the PLIC Offsets according to C906 User Manual (Page 77)

What's the UART3 IRQ?

From the Linux Device Tree...

serial@30002000 {
  compatible = "bflb,bl808-uart";
  reg = <0x30002000 0x1000>;
  interrupts = <0x14 0x04>;
  clocks = <0x04>;
  status = "okay";
  phandle = <0x0a>;
};

Thus...

• RISC-V IRQ = 0x14 = 20

• UART3 Int = (IRQ_NUM_BASE + 4)

• IRQ_NUM_BASE = 16

• NuttX IRQ = 45 (Offset by RISCV_IRQ_EXT)

• RISCV_IRQ_EXT = 25

We show the UART Interrupt Status...

bl602_attach: BL602_UART_INT_STS=0x84
bl602_attach: BL602_UART_INT_MASK=0xfff
bl602_attach: BL602_UART_INT_CLEAR=0x0
bl602_attach: BL602_UART_INT_EN=0xfff

(Source)

"urx_fer_int = 1" means "UART RX FIFO error interrupt, auto-cleared when FIFO overflow/underflow error flag is cleared"

We clear the RX FIFO Underflow, but still no UART Interrupts...

bl602_attach: BL602_UART_FIFO_CONFIG_0=0x80
bl602_attach: BL602_UART_FIFO_CONFIG_0=0x8

We dump the PLIC and UART Registers in U-Boot...

## UART Registers
=> md 0x30002000 0x36
30002000: 00001705 00000701 00130013 00000000  ................
30002010: 009f0070 0000006f 0000000f 00000000  p...o..........
30002020: 00000012 00000fff 00000000 00000fff  ................
30002030: 00000001 00000000 00000000 00000000  ................
30002040: 00000000 00000000 00000003 00000000  ................
30002050: 0026ffff 00000002 00000000 00000000  ..&.............
30002060: 00000000 00000000 00000000 00000000  ................
30002070: 00000000 00000000 00000000 00000000  ................
30002080: 00000000 07070000 0000000a 00000078  ............x...
30002090: 00000000 00000000 00000000 00000000  ................
300020a0: 00000000 00000000 00000000 00000000  ................
300020b0: 00000000 00000000 00000000 00000000  ................
300020c0: 00000000 00000000 00000000 00000000  ................
300020d0: 00000000 00000000                    ........

## PLIC Interrupt Priority
=> md 0xe0000004 0x50
e0000004: 00000000 00000000 00000000 00000000  ................
e0000014: 00000000 00000000 00000000 00000000  ................
e0000024: 00000000 00000000 00000000 00000000  ................
e0000034: 00000000 00000000 00000000 00000000  ................
e0000044: 00000000 00000000 00000000 00000000  ................
e0000054: 00000000 00000000 00000000 00000000  ................
e0000064: 00000000 00000000 00000000 00000000  ................
e0000074: 00000000 00000000 00000000 00000000  ................
e0000084: 00000000 00000000 00000000 00000000  ................
e0000094: 00000000 00000000 00000000 00000000  ................
e00000a4: 00000000 00000000 00000000 00000000  ................
e00000b4: 00000000 00000000 00000000 00000000  ................
e00000c4: 00000000 00000000 00000000 00000000  ................
e00000d4: 00000000 00000000 00000000 00000000  ................
e00000e4: 00000000 00000000 00000000 00000000  ................
e00000f4: 00000000 00000000 00000000 00000000  ................
e0000104: 00000000 00000000 00000000 00000000  ................
e0000114: 00000000 00000000 00000000 00000000  ................
e0000124: 00000000 00000000 00000000 00000000  ................
e0000134: 00000000 00000000 00000000 00000000  ................

## PLIC Hart 0 S-Mode Interrupt Enable
=> md 0xe0002080 2
e0002080: 00000000 00000000                    ........

## PLIC Hart 0 S-Mode Priority Threshold
=> md 0xe0201000 2
e0201000: 00000007 00000000                    ........

## PLIC Hart 0 S-Mode Claim / Complete
=> md 0xe0201004 1
e0201004: 00000000                             ....

## Interrupt Pending
=> md 0xe0001000 2
e0001000: 00000000 00000000                    ........

## PLIC Hart 0 M-Mode Interrupt Enable
=> md 0xe0002000 2
e0002000: 00000000 00000000                    ........

## PLIC Hart 0 M-Mode Priority Threshold
=> md 0xe0200000 2
e0200000: 00000007 00000000                    ........

## PLIC Hart 0 M-Mode Claim / Complete
=> md 0xe0200004 1
e0200004: 00000000                             ....

## Doesn't work: PLIC permission control register
## md 0xe01ffffc 1

Claim / Complete M-Mode seems OK in U-Boot. Can we Claim / Complete M-Mode in NuttX S-Mode?

=> md 0xe0200004 1
e0200004: 00000000                             ....
=> mw 0xe0200004 0x00 1

Yes we can Claim / Complete M-Mode in NuttX S-Mode. But S-Mode Claim is still 0...

PLIC Hart 0 M-Mode Claim / Complete (0xe0200004):
0000  07 00 00 00                                      ....            
bl602_attach: Claim / Complete M-Mode: claim=7
PLIC Hart 0 M-Mode Claim / Complete (0xe0200004):
0000  00 00 00 00                                      ....            
...
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0

TODO: Why UART Interrupt not enabled? U-Boot and OpenSBI don't use UART Interrupts?

TODO: What is Priority Threshold 7?

But after enabling UART IRQ, PLIC Interrupt Priority is 0 and Invalid!

PLIC Interrupt Priority (0xe0000004):
0000  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0040  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

We set PLIC Interrupt Priority to 1 ourselves...

bl602_attach: Set PLIC Interrupt Priority to 1
PLIC Interrupt Priority (0xe0000004):
0000  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0040  00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
0050  01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00  ................
0060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00a0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ...............
00e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0100  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0110  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0120  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0130  00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................

Then IRQ 25 is OK yay!

riscv_dispatch_irq: irq=8
riscv_dispatch_irq: irq=8

NuttShell (NSH) NuttX-12.0.3
riscv_dispatch_irq: irq=25
riscv_dispatch_irq: irq=25
riscv_dispatch_irq: irq=25

But UART Interrupt can't be handled because PLIC Claim is 0...

riscv_dispatch_irq: Do irq=8
NuttShell (NSH) NuttX-12.0.3
riscv_dispatch_irq: irq=25, claim=0
riscv_dispatch_irq: *0xe0201004=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        

Why is PLIC Claim = 0?

This means that there are no External Interrupts triggered. So NuttX does nothing, again and again, till it becomes too busy to respond.

But Interrupt Pending is actually set for 2 External Interrupts!

What are the 2 Interrupts Pending?

PLIC Interrupt Pending (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        

• (0xe0001000) PLIC_IP0: Interrupt Pending for interrupts 1 to 31

  0x100000 = (1 << 20) = IRQ 20 (UART3)

• (0xe0001004) PLIC_IP1: Interrupt Pending for interrupts 32 to 63

  0x100000 = (1 << 20) = IRQ 52 (EMAC2)

  TODO: Why EMAC2? We didn't enable the interrupt

We handle Interrupt Pending ourselves...

riscv_dispatch_irq: irq=25, claim=0
riscv_dispatch_irq: *0xe0201004=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        
riscv_dispatch_irq: Do irq=45
After Claim (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        

riscv_dispatch_irq: irq=25, claim=0
riscv_dispatch_irq: *0xe0201004=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        
riscv_dispatch_irq: Do irq=45
After Claim (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........

"Do IRQ" now works! But Interrupt Pending is not cleared, after we Claimed the interrupt.

Doesn't NuttX already implement C906 PLIC?

Yep but for Machine Mode only...

• c906_irq.c

• c906_irq_dispatch.c

What if we copy this code into Ox64 PLIC?

// From arch/risc-v/src/c906/c906_irq.c
/* Clear pendings in PLIC */
uintptr_t val = getreg32(JH7110_PLIC_CLAIM);
putreg32(val, JH7110_PLIC_CLAIM);

Still the same, Claim = 0...

riscv_dispatch_irq: irq=25, claim=0
riscv_dispatch_irq: *0xe0201004=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 10 00 00 00 10 00                          ........        

Something special about T-Head C906 PLIC?

According to the Linux Device Tree, Ox64 uses this PLIC Driver: "thead,c900-plic"

interrupt-controller@e0000000 {
  compatible = "thead,c900-plic";

Then from this Linux Patch

"The T-HEAD C9xx SoC implements a modified/custom T-HEAD PLIC specification which will mask current IRQ upon read to CLAIM register and will unmask the IRQ upon write to CLAIM register. The thead,c900-plic compatible string represents the custom T-HEAD PLIC specification."

"thead,c900-plic" is implemented in Linux here: irq-sifive-plic.c

Which sets PLIC_QUIRK_EDGE_INTERRUPT, which is used by...

• plic_irq_set_type

• plic_irq_domain_translate

  Which calls irq_domain_translate_twocell

  (Instead of the normal irq_domain_translate_onecell)

What does it do?

From another Linux Patch

The Renesas RZ/Five SoC has a RISC-V AX45MP AndesCore with NCEPLIC100. The NCEPLIC100 supports both edge-triggered and level-triggered interrupts. In case of edge-triggered interrupts NCEPLIC100 ignores the next interrupt edge until the previous completion message has been received and NCEPLIC100 doesn't support pending interrupt counter, hence losing the interrupts if not acknowledged in time.

So the workaround for edge-triggered interrupts to be handled correctly and without losing is that it needs to be acknowledged first and then handler must be run so that we don't miss on the next edge-triggered interrupt.

TODO: Fix this for Ox64 NuttX

After Clearing the Pending Interrupts, NuttX responds to Key Presses yay!

NuttShell (NSH) NuttX-12.0.3
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
nsh> riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
nx_start: CPU0: Beginning Idle Loop
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
ˇriscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........ 

But Claim is still 0 though.

Does it work for UART Input?

Since we've correctly identified the IRQ Number, riscv_dispatch_irq will (eventually) call bl602_receive to read the UART Input (pic below)...

bl602_receive: rxdata=-1
bl602_receive: rxdata=0x0

But the UART Input is empty! We need to troubleshoot our UART Driver some more.

(See the Complete Log)

(Watch the Demo on YouTube)

Strangeness in Ox64 BL808 PLIC

Read the article...

• "RISC-V Ox64 BL808 SBC: UART Interrupt and Platform-Level Interrupt Controller (PLIC)"

PLIC in Ox64 BL808 is acting really strange...

Why is Interrupt Priority set for 4 Interrupts, when we only set 1 (for UART)?

bl602_attach: Set PLIC Interrupt Priority to 1
PLIC Interrupt Priority (0xe0000004):
...
0040  00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
0050  01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00  ................

Maybe it's a problem with the RISC-V Code generated by GCC?

Let's do a simple test: bl602_serial.c

// Test the setting of PLIC Interrupt Priority
void test_interrupt_priority(void) {
  static uint32_t before1 = 0xFF;
  static uint32_t before2 = 0xFF;
  static uint32_t after1 = 0xFF;
  static uint32_t after2 = 0xFF;

  // Read the values before setting Interrupt Priority
  before1 = *(volatile uint32_t *) 0xe0000050UL;
  before2 = *(volatile uint32_t *) 0xe0000054UL;

  // Set the Interrupt Priority
  *(volatile uint32_t *) 0xe0000050UL = 1;

  // Read the values after setting Interrupt Priority
  after1 = *(volatile uint32_t *) 0xe0000050UL;
  after2 = *(volatile uint32_t *) 0xe0000054UL;
  _info("before1=%u, before2=%u, after1=%u, after2=%u\n", before1, before2, after1, after2);
}

The Interrupt Priority wasn't be set correctly. Why did 0xe0000054 change from 0 to 1?

bl602_attach: Test Interrupt Priority
test_interrupt_priority: before1=0, before2=0, after1=1, after2=1

Here's the Disassembly, which looks OK...

0000000050200daa <test_interrupt_priority>:
test_interrupt_priority():
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:451
  uint32_t before1 = *(volatile uint32_t *) 0xe0000050;
    50200daa:	461d                	li	a2,7
    50200dac:	0676                	slli	a2,a2,0x1d

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:454
  *(volatile uint32_t *) 0xe0000050 = 1;
    50200dae:	4785                	li	a5,1

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:451
  uint32_t before1 = *(volatile uint32_t *) 0xe0000050;
    50200db0:	4a34                	lw	a3,80(a2)

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:452
  uint32_t before2 = *(volatile uint32_t *) 0xe0000054;
    50200db2:	4a78                	lw	a4,84(a2)

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:454
  *(volatile uint32_t *) 0xe0000050 = 1;
    50200db4:	ca3c                	sw	a5,80(a2)

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:457
  uint32_t after1 = *(volatile uint32_t *) 0xe0000050;
    50200db6:	4a3c                	lw	a5,80(a2)

/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:458
  uint32_t after2 = *(volatile uint32_t *) 0xe0000054;
    50200db8:	05462803          	lw	a6,84(a2)

Maybe we need to flush the CPU Cache?

Nope sfence doesn't help...

0000000050200daa <test_interrupt_priority>:
test_interrupt_priority():
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:451
  uint32_t before1 = *(volatile uint32_t *) 0xe0000050;
    50200daa:	461d                	li	a2,7
    50200dac:	0676                	slli	a2,a2,0x1d
    50200dae:	4a34                	lw	a3,80(a2)
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:452
  uint32_t before2 = *(volatile uint32_t *) 0xe0000054;
    50200db0:	4a78                	lw	a4,84(a2)
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:455
  *(volatile uint32_t *) 0xe0000050 = 1;
    50200db2:	4785                	li	a5,1
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:451
  uint32_t before1 = *(volatile uint32_t *) 0xe0000050;
    50200db4:	2681                	sext.w	a3,a3
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:452
  uint32_t before2 = *(volatile uint32_t *) 0xe0000054;
    50200db6:	2701                	sext.w	a4,a4
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:455
  *(volatile uint32_t *) 0xe0000050 = 1;
    50200db8:	ca3c                	sw	a5,80(a2)
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:458
  __asm__ __volatile__
    50200dba:	12000073          	sfence.vma
    50200dbe:	0330000f          	fence	rw,rw
    50200dc2:	0000100f          	fence.i
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:467
  uint32_t after1 = *(volatile uint32_t *) 0xe0000050;
    50200dc6:	4a3c                	lw	a5,80(a2)
    50200dc8:	2781                	sext.w	a5,a5
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:469
  __asm__ __volatile__
    50200dca:	12000073          	sfence.vma
    50200dce:	0330000f          	fence	rw,rw
    50200dd2:	0000100f          	fence.i
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:477
  uint32_t after2 = *(volatile uint32_t *) 0xe0000054;
    50200dd6:	05462803          	lw	a6,84(a2)
    50200dda:	2801                	sext.w	a6,a6
/Users/Luppy/ox64/nuttx/arch/risc-v/src/chip/bl602_serial.c:479
  __asm__ __volatile__
    50200ddc:	12000073          	sfence.vma
    50200de0:	0330000f          	fence	rw,rw
    50200de4:	0000100f          	fence.i

Let's do the same with U-Boot Bootloader. It looks OK, doesn't have the same problem...

## Read the values before setting Interrupt Priority
=> md 0xe0000050 1
e0000050: 00000000                             ....
=> md 0xe0000054 1
e0000054: 00000000                             ....

## Set the Interrupt Priority
=> mw 0xe0000050 0x01 1

## Read the values after setting Interrupt Priority
=> md 0xe0000050 1
e0000050: 00000001                             ....
=> md 0xe0000054 1
e0000054: 00000000                             ....

And U-Boot doesn't use MMU.

Could it be caused by MMU?

Let's try setting Interrupt Priority before MMU Init. It works OK!

123jh7110_copy_ramdisk: _edata=0x50400258, _sbss=0x504002a0, _ebss=0x50408000, JH7110_IDLESTACK_TOP=0x50408c00
jh7110_copy_ramdisk: ramdisk_addr=0x50410291
jh7110_copy_ramdisk: size=8192000
ABCjh7110_mm_init: Test Interrupt Priority
test_interrupt_priority: before1=0, before2=0, after1=1, after2=0
jh7110_kernel_mappings: map I/O regions

When we set Interrupt Priority after MMU Init, it looks incorrect...

123jh7110_copy_ramdisk: _edata=0x50400258, _sbss=0x504002a0, _ebss=0x50408000, JH7110_IDLESTACK_TOP=0x50408c00
jh7110_copy_ramdisk: ramdisk_addr=0x50410291
jh7110_copy_ramdisk: size=8192000
ABCjh7110_kernel_mappings: map I/O regions
jh7110_kernel_mappings: map PLIC as Interrupt L2
jh7110_kernel_mappings: connect the L1 and Interrupt L2 page tables for PLIC
jh7110_kernel_mappings: map kernel text
jh7110_kernel_mappings: map kernel data
jh7110_kernel_mappings: connect the L1 and L2 page tables
jh7110_kernel_mappings: map the page pool
mmu_satp_reg: pgbase=0x50407000, asid=0x0, reg=0x8000000000050407
mmu_write_satp: reg=0x8000000000050407
jh7110_mm_init: Test Interrupt Priority
test_interrupt_priority: before1=0, before2=0, after1=0, after2=0
nx_start: Entry

So it's an MMU problem!

Why is MMU messing up our updates to Ox64 BL808 PLIC?

We might have missed something specific to C906 MMU. Here are the Extended Page Attributes, from C906 User Manual (Page 53)

• SO – Strong order (Bit 63)

  Indicates the access order required by memory.

  0: no strong order (Normal-memory)

  1: strong order (Device)

  The default value is no strong order.

• C – Cacheable (Bit 62)

  0: Non-cacheable

  1: Cacheable

  The default value is Non-cacheable.

• B – Buffer (Bit 61)

  0: Non-bufferable

  1: Bufferable

  The default value is Non-bufferable

Also...

"C906 extended page attributes exist only when the MAEE bit in the MXSTATUS register is 1."

So Beware: T-Head MMU Flags (Strong Order / Shareable) are available only if OpenSBI has set the MAEE Bit in the MXSTATUS Register to 1. Otherwise the MMU will crash when we set the flags!

For Ox64 (T-Head C906): MAEE Bit in MXSTATUS Register is set to 1. So T-Head MMU Flags are allowed. (But d0_lowload Boot Code doesn't set MXSTATUS?)

For CanMV-k230 (T-Head C908): MAEE Bit in MXSTATUS Register is (probably) set to 0. So T-Head MMU Flags are NOT ALLOWED. MMU will crash if we set the T-Head MMU Flags! (Reported by yf13)

TODO: Does T-Head C908 support T-Head MMU Flags or RISC-V Svpbmt Extension? We can't find the C908 Manual, and this Linux Patch says...

In an ideal world those would be handled as extensions as well - T-Head fixed their vectors with the C908 so they might do standards-compliant Svpbmt and Zicbom in the future.

This is how we set the T-Head MMU Flags (Strong Order / Shareable)...

• "Fixed the UART Interrupt and Platform-Level Interrupt Controller (Ox64 BL808)"

What is ERRATA_THEAD_* in Linux Kernel?

• riscv/errata/thead/errata.c

• riscv/include/asm/errata_list.h

• riscv/include/asm/pgtable-64.h

• drivers/perf/riscv_pmu_sbi.c

Check out the answer here...

• "Fixed the UART Interrupt and Platform-Level Interrupt Controller (Ox64 BL808)"

What if we disable and re-enable MMU, while setting PLIC Interrupt Priority?

Yep seems to work...

jh7110_mm_init: Disable MMU
mmu_write_satp: reg=0
jh7110_mm_init: Test Interrupt Priority
test_interrupt_priority: before1=0, before2=0, after1=1, after2=0
jh7110_mm_init: Enable MMU

Compare Ox64 BL808 UART Registers

Read the article...

• "Fixed the UART Interrupt and Platform-Level Interrupt Controller (Ox64 BL808)"

To fix the null UART Input, let's compare the UART Registers from NuttX vs U-Boot Bootloader

UART Registers from NuttX UART Driver...

// UART Registers from NuttX
bl602_receive: rxdata=-1
bl602_receive: rxdata=0x0
UART Registers (0x30002000):
0000  05 17 00 00 | 01 07 00 00 | 13 00 13 00 | 00 00 00 00  ................
0010  70 00 9f 00 | 6f 00 00 00 | 0f 00 00 00 | 00 00 00 00  p...o...........
0020 [94 00 00 00]|[f5 0f 00 00]| 00 00 00 00 | ff 0f 00 00  ................
0030  01 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
0040  00 00 00 00 | 00 00 00 00 | 03 00 00 00 | 00 00 00 00  ................
0050 [ff ff 1c 00]| 02 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
0060  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ...............
0070  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
0080 [80 00 00 00]|[18 00 07 07]| 0a 00 00 00 |[00 00 00 00] ................
0090  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
00a0  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
00b0  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
00c0  00 00 00 00 | 00 00 00 00 | 00 00 00 00 | 00 00 00 00  ................
00d0  00 00 00 00 | 00 00 00 00 |             |              ........        

UART Registers from U-Boot Bootloader...

## UART Registers from U-Boot
=> md 0x30002000 0x36
30002000: 00001705  00000701 00130013 00000000  ................
30002010: 009f0070  0000006f 0000000f 00000000  p...o..........
30002020:[00000012][00000fff]00000000 00000fff  ................
30002030: 00000001  00000000 00000000 00000000  ................
30002040: 00000000  00000000 00000003 00000000  ................
30002050:[0026ffff] 00000002 00000000 00000000  ..&.............
30002060: 00000000  00000000 00000000 00000000  ................
30002070: 00000000  00000000 00000000 00000000  ................
30002080:[00000000][07070000]0000000a[00000078]  ............x...
30002090: 00000000  00000000 00000000 00000000  ................
300020a0: 00000000  00000000 00000000 00000000  ................
300020b0: 00000000  00000000 00000000 00000000  ................
300020c0: 00000000  00000000 00000000 00000000  ................
300020d0: 00000000  00000000                    ........

Here are the differences (marked above)...

Offset 20: uart_int_sts (Interrupt Status)

00000094 = 0b10010100
Bit 7 urx_fer_int: UART RX FIFO error interrupt, auto-cleared when FIFO overflow/underflow error flag is cleared
Bit 4 urx_rto_int: UART RX Time-out interrupt
Bit 2 utx_frdy_int: UART TX FIFO ready (tx_fifo_cnt > tx_fifo_th) interrupt, auto-cleared when data is pushed

00000012 = 0b00010010
Bit 4 urx_rto_int: UART RX Time-out interrupt
Bit 1 urx_end_int: UART RX transfer end interrupt (set according to cr_urx_-len)

Offset 24: uart_int_mask (Interrupt Mask)
00000ff5
00000fff
TODO: Set to 0xfff

Offset 50: urx_bcr_int_cfg (Receive Byte Count)
001cffff
0026ffff
Number of bytes received. OK to ignore this.

Offset 80: uart_fifo_config_0 (FIFO Config 0)
00000080
00000000
Bit 7 rx_fifo_underflow: Underflow flag of RX FIFO
Can be cleared by rx_fifo_clr.
TODO: Set Bit 3 rx_fifo_clr: Clear signal of RX FIFO

Offset 84: uart_fifo_config_1 (FIFO Config 1)
07070018
07070000
rx_fifo_cnt = 1 (RX FIFO available count)
tx_fifo_cnt = 8 (TX FIFO available count)
Let's ignore this.

Offset 8c: uart_fifo_rdata (Receive Data)
00000000
00000078
RX FIFO. OK to ignore this.

Nope still the same.

Fix the UART Interrupt for Ox64 BL808

Read the article...

• "Fixed the UART Interrupt and Platform-Level Interrupt Controller (Ox64 BL808)"

Let's fix the UART Interrupt for Ox64 BL808!

UART Input is strangely null, so we tried printing the UART Input just before reading it: bl602_serial.c

/****************************************************************************
 * Name: bl602_rxavailable
 *
 * Description:
 *   Return true if the receive register is not empty
 *
 ****************************************************************************/

static bool bl602_rxavailable(struct uart_dev_s *dev)
{
  struct bl602_uart_s *priv = (struct bl602_uart_s *)dev->priv;
  uint8_t uart_idx          = priv->config.idx;

  /* Return true is data is available in the receive data buffer */

  uintptr_t rx = getreg32(0x3000208c); _info("rx=%p\n", rx); ////
  return (getreg32(BL602_UART_FIFO_CONFIG_1(uart_idx)) & \
          UART_FIFO_CONFIG_1_RX_CNT_MASK) != 0;
}

Yes UART Input is correct!

nx_start: CPU0: Beginning Idle Loop
bl602_rxavailable: rx=0x31
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrut Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
bl602_rxavailable: rx=0x32
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........  

But somehow UART Input is erased when we read BL602_UART_FIFO_CONFIG_1 (Offset 0x84)...

  uintptr_t fifo = getreg32(0x30002084);
  uintptr_t rx = getreg32(0x3000208c);
  _info("fifo=%p, rx=%p\n", fifo, rx);

Which shows...

nx_start: CPU0: Beginning Idle Loop
bl602_rxavailable: fifo=0x7070120, rx=0
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........

Why is C906 messing up the memory access?

From Linux Kernel we see the MMU Flags (Strong Order + Shareable) for I/O Memory: pgtable-64.h

Which is used by this T-Head Errata: errata_list.h

We do the same to Enable Strong Order + Shareable in NuttX Page Table Entries: riscv_mmu.c

  // Save the Page Table Entry
  lntable[index] = (paddr | mmuflags);

  //// Begin Test
  // From https://github.com/torvalds/linux/blob/master/arch/riscv/include/asm/pgtable-64.h#L126-L142
  /*
  * [63:59] T-Head Memory Type definitions:
  * bit[63] SO - Strong Order
  * bit[60] SH - Shareable
  * 10010 - IO   Strongly-ordered, Non-cacheable, Non-bufferable, Shareable, Non-trustable
  */
  #define _PAGE_IO_THEAD		((1UL << 63) | (1UL << 60))
  if ((mmuflags & PTE_R) &&  // Leaf Page Table Entry
    (vaddr < 0x40000000UL || vaddr >= 0xe0000000UL))  // I/O or PLIC Memory
    {
      // Enable Strong Order and Shareable
      lntable[index] = lntable[index] | _PAGE_IO_THEAD;
      _info("vaddr=%p, lntable[index]=%p\n", vaddr, lntable[index]);
    }
  //// End Test

Yep UART Input works OK yay!

nx_start: CPU0: Beginning Idle Loop
bl602_receive: rxdata=0x31
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
1riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
bl602_receive: rxdata=0x32
riscv_dispatch_irq: Clear Pending Interrupts, irq=45, claim=0
PLIC Interrupt Pending (0xe0001000):
0000  00 00 00 00 00 00 00 00                          ........        
2

Finally UART Input and PLIC are both OK yay!

Starting kernel ...
123jh7110_copy_ramdisk: _edata=0x50400258, _sbss=0x50400290, _ebss=0x50110_IDLESTACK_TOP=0x50407c00
jh7110_copy_ramdisk: ramdisk_addr=0x50410281
jh7110_copy_ramdisk: size=8192000
ABCjh7110_kernel_mappings: map I/O regions
mmu_ln_setentry: vaddr=0, lntable[index]=0x90000000000000e7
jh7110_kernel_mappings: map PLIC as Interrupt L2
mmu_ln_setentry: vaddr=0xe0000000, lntable[index]=0x90000000380000e7
mmu_ln_setentry: vaddr=0xe0200000, lntable[index]=0x90000000380800e7
mmu_ln_setentry: vaddr=0xe0400000, lntable[index]=0x90000000381000e7
mmu_ln_setentry: vaddr=0xe0600000, lntable[index]=0x90000000381800e7
...
mmu_ln_setentry: vaddr=0xefc00000, lntable[index]=0x900000003bf000e7
mmu_ln_setentry: vaddr=0xefe00000, lntable[index]=0x900000003bf800e7
jh7110_kernel_mappings: connect the L1 and Interrupt L2 page tables for PLIC
jh7110_kernel_mappings: map kernel text
jh7110_kernel_mappings: map kernel data
jh7110_kernel_mappings: connect the L1 and L2 page tables
jh7110_kernel_mapmap the page pool
up_irqinitialize: 

NuttShell (NSH) NuttX-12.0.3
nsh> uname -a
NuttX 12.0.3 fd05b07 Nov 24 2023 07:42:54 risc-v star64
nsh> 
nsh> ls /dev
/dev:
 console
 null
 ram0
 zero
nsh> 
nsh> hello
Hello, World!!

(Watch the Demo on YouTube)

C906 MMU is actually explained in C906 Integration Manual (Chinese), Page 9.

And here's the RTL Code for C906 MMU.

Fix the RISC-V Timer with OpenSBI

Read the article...

• "Nim on a Real-Time Operating System: Apache NuttX RTOS + Ox64 BL808 SBC"

The sleep command hangs in NuttX Shell. How to fix it?

That's because we haven't implemented the RISC-V Timer for Ox64! We should call OpenSBI to handle the Timer, here's the fix.

(Ignore riscv_mtimer.c, we were verifying that mtime and mtimecmp are unused in Kernel Mode)

We only need to change arch/risc-v/src/bl808/bl808_timerisr.c

// Timer Frequency
#define MTIMER_FREQ 1000000

// This function is called during start-up to initialize the timer interrupt.
void up_timer_initialize(void) {
  struct oneshot_lowerhalf_s *lower = riscv_mtimer_initialize(
    0, 0, RISCV_IRQ_STIMER, MTIMER_FREQ);
  DEBUGASSERT(lower);
  up_alarm_set_lowerhalf(lower);
}

How it works: At startup, up_timer_initialize (above) calls...

• riscv_mtimer_initialize which calls...

• riscv_mtimer_set_mtimecmp which calls...

• riscv_sbi_set_timer which calls...

• sbi_ecall which makes an ecall to OpenSBI

• Which accesses the System Timer

Originally we set MTIMER_FREQ to 10000000: bl808_timerisr.c

#define MTIMER_FREQ 10000000

But this causes the command sleep 1 to pause for 10 seconds. So we divide the frequency by 10: bl808_timerisr.c

#define MTIMER_FREQ 1000000

Now the sleep command works correctly in NuttX Shell!

Here's the log (ignore the errors)

LED Driver for Ox64 BL808

Read the article...

• "Nim on a Real-Time Operating System: Apache NuttX RTOS + Ox64 BL808 SBC"

We wish to blink an LED with Nim on Ox64...

• "Blink an LED with Nim"

But first we need a barebones NuttX LED Driver for Ox64.

How to create the NuttX LED Driver?

We assume LED is connected to GPIO 29, Pin 21. (See the Pinout)

(With a 47 Ohm Resistor, yellow-purple-black-gold)

How do we flip a BL808 GPIO High and Low?

From BL808 Reference Manual Page 56, "Normal GPIO Output Mode"...

• Set reg_gpio_xx_oe (Bit 6) to 1 to enable the GPIO output mode
  = (1 << 6)

• Set reg_gpio_xx_func_sel (Bits 8 to 12) to 11 to enter the SWGPIO mode
  = (11 << 8)

• Set reg_gpio_xx_mode (Bits 30 to 31) to 0 to enable the normal output function of I/O
  = (0 << 30)

• Set reg_gpio_xx_pu (Bit 4) and reg_gpio_xx_pd (Bit 5) to 0 to disable the internal pull-up and pull-down functions
  = (0 << 4)

• Set the level of I/O pin through reg_gpio_xx_o (Bit 24)
  = Either (0 << 24) Or (1 << 24)

(GPIO Bit Definitions are below)

Which means...

• Set GPIO Output to 0
  = (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (0 << 24)
  = 0xb40

• Set GPIO Output to 1
  = (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (1 << 24)
  = 0x1000b40

How to test this?

GPIO 29 Base Address gpio_cfg29 is 0x20000938.

For testing, we run U-Boot Bootloader Commands to set GPIO 29 to High and Low...

## Dump gpio_cfg29 at 0x20000938
$ md 0x20000938 1
20000938: 00400803                             ..@.

## Set GPIO Output to 0: (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (0 << 24)
## = 0xb40
$ mw 0x20000938 0xb40 1
$ md 0x20000938 1
20000938: 00000b40                             @...

## Set GPIO Output to 1: (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (1 << 24)
## = 0x1000b40
$ mw 0x20000938 0x1000b40 1
$ md 020000938 1
20000938: 01000b40                             @...

And U-Boot switches the LED On and Off correctly yay!

How to flip the GPIO in our NuttX LED Driver?

This is how we flip the GPIO in our NuttX LED Driver: bl808_userleds.c

// Switch the LEDs On and Off according to the LED Set
// (Bit 0 = LED 0)
void board_userled_all(uint32_t ledset)
{
  _info("ledset=0x%x\n", ledset);////
  int i;

  // For LED 0 to 2...
  for (i = 0; i < BOARD_LEDS; i++)
    {
      // Get the desired state of the LED
      bool val = ((ledset & g_led_setmap[i]) != 0);
      _info("led=%d, val=%d\n", i, val);////

      // If this is LED 0...
      if (i == 0)
        {
          // Switch it On or Off?
          if (val)
            {
              // Switch LED 0 (GPIO 29) to On:
              // Set gpio_cfg29 to (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (1 << 24)
              // mw 0x20000938 0x1000b40 1
              *(volatile uint32_t *) 0x20000938 = 0x1000b40;
            }
          else
            {
              // Switch LED 0 (GPIO 29) to Off:
              // Set gpio_cfg29 to (1 << 6) | (11 << 8) | (0 << 30) | (0 << 4) | (0 << 24)
              // mw 0x20000938 0xb40 1
              *(volatile uint32_t *) 0x20000938 = 0xb40;
            }
        }
      ////TODO: a64_pio_write(g_led_map[i], (ledset & g_led_setmap[i]) != 0);
    }
}

And our LED Driver works OK with Nim: It blinks our LED on Ox64 BL808 SBC!

• Watch the Demo on YouTube

• See the Log

• "Blink an LED with Nim"

Later we'll replace the (awful) code above by the BL808 GPIO Driver. Which we'll copy from NuttX for BL602.

How did we get the GPIO Bit Definitions?

From BL808 Reference Manual Page 119...

4.8.30 gpio_cfg29
Base Address：0x20000938

Bits Name Type Reset Description

31:30 reg_gpio_29_mode r/w 0 When GPIO Function Selected to SWGPIO
00 (Output Value Mode): GPIO Output by reg_gpio_x_o
Value
01 (Set/Celar Mode ) :GPIO Output set by reg_gpio_x_set
and clear by reg_gpio_x_clr
10 : SWGPIO Source comes from GPIO DMA (GPIO DMA
Mode), GPIO Output value by gpio_dma_o
11: SWGPIO Source comes from GPIO DMA (GPIO DMA
Mode), GPIO Outout value by gpio_dma_set/gpio_dma_clr

29 RSVD

28 reg_gpio_29_i r 0

27 RSVD

26 reg_gpio_29_clr w1p 0 When SWGPIO @ Set/Clear Mode
Set this bit will clear GPIO output value to 0,when set/clr at
the same time, only set take effect

25 reg_gpio_29_set w1p 0 When SWGPIO @ Set/Clear Mode
Set this bit will set GPIO output value to 1,when set/clr at
the same time, only set take effect

24 reg_gpio_29_o r/w 0 When SWGPIO @ Output Value Mode
00 : GPIO Value changes according to this value
01 : GPIO Value Set by this register and clr by clr_reg

23 RSVD

22 reg_gpio_29_int_mask r/w 1 mask interrupt (1)

21 gpio_29_int_stat r 0 interrupt status

20 reg_gpio_29_int_clr r/w 0 clear interrupt

19:16 reg_gpio_29_int_mode_set r/w 0 0000 : sync falling edge trigger
0001 : sync rising edge trigger
0010 : sync low level trigger
0011 : sync high level trigger
01xx : sync rising & falling edge trigger
1000 : async falling edge trigger
1001 : async rising edge trigger
1010 : async low level trigger
1011 : async high level trigger

15:13 RSVD

12:8 reg_gpio_29_func_sel r/w 5’hB GPIO Function Select (Default : SW-GPIO)

7 RSVD

6 reg_gpio_29_oe r/w 0 Register Controlled GPIO Output Enable (Used when GPIO
Function select to Register Control GPIO)

5 reg_gpio_29_pd r/w 0 GPIO Pull Down Control

4 reg_gpio_29_pu r/w 0 GPIO Pull Up Control

3:2 reg_gpio_29_drv r/w 0 GPIO Driving Control

1 reg_gpio_29_smt r/w 1 GPIO SMT Control

0 reg_gpio_29_ie r/w 0 GPIO Input Enable

Emulate Ox64 BL808 SBC with TinyEMU

Read the articles...

• "Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)"

• "Emulate Ox64 BL808 in the Web Browser: Experiments with TinyEMU RISC-V Emulator and Apache NuttX RTOS"

Objective: Take the NuttX Kernel built for Ox64 BL808 SBC. And boot it on TinyEMU RISC-V Emulator in the Web Browser!

Check out the details here...

• "Emulate Ox64 BL808 SBC with TinyEMU"

(Live Demo of Ox64 BL808 Emulator)

Build and Test NuttX Apps in the Web Browser

Ox64 Emulator is now integrated with TCC RISC-V Compiler in WebAssembly. So we can Build and Test NuttX Apps in the Web Browser!

Read the article...

• "Zig runs ROM FS Filesystem in the Web Browser (thanks to Apache NuttX RTOS)"

QuickJS JavaScript Engine on Ox64 BL808 SBC

QuickJS JavaScript Engine now runs on Ox64 BL808 SBC, and in the Ox64 Web Emulator!

Read the article...

• "QuickJS JavaScript Engine on a Real-Time Operating System (Apache NuttX RTOS)"

Documentation for Ox64 BL808

Pine64 Ox64 64-bit RISC-V SBC (Sorry for my substandard soldering)

• "Ox64 BL808 RISC-V SBC: Booting Linux and (maybe) Apache NuttX RTOS"

• "Booting Linux on the Pine64 Ox64 SBC"

• Pine64 Ox64 Wiki

• Pine64 Ox64 Schematic

• OpenBouffalo Wiki

• Linux Image + OpenSBI + U-Boot for BL808

  (Newer version?)

• BL808 Datasheet

• BL808 Reference Manual

• XuanTie OpenC906 User Manual

• BL808 D0 Core: T-Head C906 480MHz 64-bit RISC-V CPU

  (Multimedia Core: MIPI CSI / DSI, Neural Proc Unit)

  Memory Mgmt Unit is Sv39, 128/256/512 TLB table entry. (Same as Star64?)

• BL808 M0 Core: T-Head E907 320MHz 32-bit RISC-V CPU

  (Wireless + Peripherals Core: WiFi, BLE, BT, Zigbee, Audio)

• BL808 LP Core: T-Head E902 150MHz 32-bit RISC-V CPU

  (Low Power Core)