6. Linux Kernel & Driver Development
在学习NVDLA的UMD和KMD代码的时候，特别是在进行系统级调试的时候，发现对于Linux内核和驱动的相关知识相对欠缺，很多过程和原理都不熟悉，需要进行补充；本章节有对NVDLA代码的学习，也有对Linux内核和驱动开发本身的学习。
当系统首次启动或复位时，处理器会从默认位置处开始执行代码。在个人电脑（PC）中，该位置位于基本输入/输出系统（BIOS）中，该系统存储在主板上的闪存中。嵌入式系统中的中央处理单元（CPU）调用复位向量以在闪存/ ROM中的默认地址处启动程序。无论哪种情况，结果都是一样的。由于PC提供了很大的灵活性，因此BIOS必须确定哪些设备可以启动。
当找到引导设备时，第一阶段引导加载程序被加载到RAM中并执行。该引导加载程序的长度小于512个字节（单个扇区），其任务是加载第二阶段引导加载程序。
当第二阶段引导加载程序在RAM中并且正在执行时，通常会显示启动画面，并将Linux和可选的初始RAM磁盘（临时根文件系统）加载到内存中。加载镜像时，第二阶段引导加载程序将控制权交给内核映像，并对内核进行解压缩和初始化。在此阶段，第二阶段引导加载程序检查系统硬件，枚举附加的硬件设备，安装根设备，然后加载必要的内核模块。完成后，将启动第一个用户空间程序（init），并执行高级系统初始化。
具体的细节，可以参考IBM的技术博客：Inside the Linux boot process
在VP的仿真中，需要Aarch64访问Ubuntu的文件系统，加载编译好的nvdla_runtime可执行文件，这就需要借助于VIRT FS功能。
- add QEMU command line:
-fsdev local,id=r,path=.,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r
- run following command on the guest
mount -t 9p -o trans=virtio r /mnt
目前公开的KMD是针对AArch64架构处理器的，使用的Linux Kernel是4.13.3。针对DMA传输，使用的协议是DRM和GEM PRIME。
在进行KMD编译的时候，需要准备好Linux Kernel源码和交叉编译器。在这里就是
- 交叉编译器：https://releases.linaro.org/components/toolchain/binaries/latest/aarch64-linux-gnu/gcc-linaro-7.2.1-2017.11-i686_aarch64-linux-gnu.tar.xz
- Linux Kernel源码：4.13.3
cp arch/arm64/configs/defconfig .config
make ARCH=arm64 CROSS_COMPILE=~/Downloads/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- oldconfig
make ARCH=arm64 CROSS_COMPILE=~/Downloads/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- -k
在kmd目录下，编译KMD的命令如下：
make KDIR=<path_to_Linux_source> ARCH=arm64 CROSS_COMPILE=<path_to_toolchain>
示例：
make KDIR=your path/linux-4.13.3 ARCH=arm64 CROSS_COMPILE=your path/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin/
编译之后，将会得到kmd/port/linux/opendla.ko文件
而编译UMD就方便很多，命令如下：
export TOP=<path_to_umd>
export TOOL_CHAIN_PREFIX=$(CROSS_COMPILE)/aarch64-linux-gnu-
make
编译之后，将会得到umd/out/runtime/nvdla_runtime/nvdla_runtime和umd/out/runtime/libnvdla_runtime/libnvdla_runtime.so。
当进行NV_SMALL编译的时候，需要在脚本中添加CFLAG_MODULE='-DDLA_SMALL_CONFIG'的选项.
Fedora/RISC-V的kernel编译需要在RISC-V的vm上面直接编译，不能使用交叉编译。因此，需要借助于Fedora提供的vm。如果只是尝鲜，可以下载提供的编译好的disk-image：stage4 disk image
现在我们需要先明确一下使用Qemu或者硬件boot Fedora/RISC-V Linux需要的文件，主要包括三个：vmlinux，bbl，disk.img。因此，我们需要依次clone相关的仓库并生成所需的文件。
Linux kernel build scripts and patches for Fedora/RISC-V
Used to build Fedora/RISC-V stage4 disk images
具体操作步骤可以参考链接：Architectures/RISC-V/Installing 需要下载后缀为-sda.raw.xz的镜像文件，解压，并使用镜像管理工具guestfish（Guestfish使用说明）生成ext4格式的文件系统。
下载bbl（bbl下载链接），并启动qemu虚拟机。
qemu-system-riscv64 \
-nographic \
-machine virt \
-smp 4 \
-m 2G \
-kernel bbl \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-device,rng=rng0 \
-append "console=ttyS0 ro root=/dev/vda" \
-device virtio-blk-device,drive=hd0 \
-drive file=Fedora-Developer-Rawhide-20180808.n.0-sda1.raw,format=raw,id=hd0 \
-device virtio-net-device,netdev=usernet \
-netdev user,id=usernet,hostfwd=tcp::10000-:22
vmlinux是未压缩的内核，vmlinux 是ELF文件，即编译出来的最原始的文件。用于kernel-debug，产生system.map符号表，不能用于直接加载，不可以作为启动内核。只是启动过程中的中间媒体。“vm”代表“Virtual Memory”。Linux 支持虚拟内存，不像老的操作系统比如DOS有640KB内存的限制
克隆源代码riscv-linux
截至到编辑日期，已经支持到4.18-rc6.我们这里选择4.15版本进行编译。
在host端克隆源代码fedora-riscv-kernel 在Fedora/RISC-V的vm端，借助于ssh将文件拷贝到虚拟机。在进行编译之前，还需要安装几个必须的库，包括bc/openssl/zlib-dev，需要从koji/packages下载所需的RPM包，注意需要与系统版本相对应，例如bc-1.07.1-6.fc29.riscv64.rpm。
安装完成依赖库以后，直接make，就可以编译生成所需的bbl文件。
可以选择克隆源代码fedora-riscv-stage4，也可以选择下载提供的最新的img文件stage4 disk image。
退出Fedora/RISC-V虚拟机，采用命令启动新的qemu，并测试。
qemu-system-riscv64 \
-nographic \
-machine virt \
-smp 4 \
-m 2G \
-kernel bbl \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-device,rng=rng0 \
-append "console=ttyS0 ro root=/dev/vda" \
-device virtio-blk-device,drive=hd0 \
-drive file=stage4-disk.img,format=raw,id=hd0 \
-device virtio-net-device,netdev=usernet \
-netdev user,id=usernet,hostfwd=tcp::10000-:22
In order to maintain the same kernel verison with RISC-V, we need to port nvdla VP(including linux kernel, umd, kmd) from default linux-4.13.3 to newer linux-4.15.
Download the buildroot from https://buildroot.org/download.html. We use the version buildroot-2018.08-rc2.
Use qemu_aarch64_virt_defconfig as base config, then set the customized configurations:
$ make qemu_aarch64_virt_defconfig
$ make menuconfig
* Target Options -> Target Architecture -> AArch64 (little endian)
* Target Options -> Target Architecture Variant -> cortex-A57
* Toolchain -> Toolchain type -> External toolchain
* Toolchain -> Toolchain -> Linaro AArch64 2018.05
* Toolchain -> Toolchain origin -> Toolchain to be downloaded and installed
* Kernel -> () Kernel version -> 4.15
* Kernel -> Kernel configuration -> Use the architecture default configuration
* System configuration -> Enable root login with password -> Y
* System configuration -> Root password -> nvdla
* Target Packages -> Show packages that are also provided by busybox -> Y
* Target Packages -> Networking applications -> openssh -> Y
Then build. When it's done, we get the kernel image and rootfs in output/images like this.
make
The make progress may take vary time according to your machine.
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Copying files into the device: done
Writing superblocks and filesystem accounting information: done
ln -sf rootfs.ext2 /home/nvdla/NVDLA/BuildRoot/buildroot-2018.08-rc2/output/images/rootfs.ext4
nvdla@nvdla:~/NVDLA/BuildRoot/buildroot-2018.08-rc2$ ls output/images/
**Image** **rootfs.ext2** **rootfs.ext4**
And we get the drm.ko file in the following directory:
~/buildroot-2018.08-rc2/output/build/linux-4.15/drivers/gpu/drm/drm.ko
Building KMD, and we get the opendla.ko:
make KDIR=~/buildroot-2018.08-rc2/output/build/linux-4.15 ARCH=arm64 CROSS_COMPILE=~/buildroot-2018.08-rc2/output/host/bin/aarch64-linux-gnu-
Building UMD, and we get the nvdla_runtime & libnvdla_runtime.so:
export TOP=<path to umd>
export CROSS_COMPILE=~/buildroot-2018.08-rc2/output/host/bin
make
Using the FAMOUS command line to launch the VP:
export SC_SIGNAL_WRITE_CHECK=DISABLE
./build/bin/aarch64_toplevel -c conf/aarch64_nvdla_4.15.lua
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Welcome to Buildroot
buildroot login: root
Password:
# uname -a
Linux buildroot 4.15.0 #1 SMP PREEMPT Wed Aug 22 14:45:31 CST 2018 aarch64 GNU/Linux
Inserting drm.ko is OK. But there are some errors when inserting opendla.ko.
# ls
Image drm.ko rootfs.ext2 rootfs.ext4
# insmod drm.ko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# insmod ../../images/linux-4.15/opendla.ko
[ 1747.174820] opendla: loading out-of-tree module taints kernel.
[ 1747.182053] reset engine done
[ 1747.193011] [drm] Initialized nvdla 0.0.0 20171017 for 10200000.nvdla on minor 0
[ 1747.200831] NVDLA 10200000.nvdla: failed to register drm device
[ 1747.202097] NVDLA: probe of 10200000.nvdla failed with error -12
NEED Some Help??????
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
OK. With the help from giusecesa4 Error -12 refers to the Linux constant ENOMEM., the problem is solved, and the lenet-5 test is completed gracefully.
# insmod ../../images/linux-4.15/drm.ko
# insmod ../../images/linux-4.15/opendla.ko
[ 70.413868] opendla: loading out-of-tree module taints kernel.
[ 70.417904] reset engine done
[ 70.464807] [drm] Initialized nvdla 0.0.0 20171017 for 10200000.nvdla on minor 0
# ls
basic.nvdla libnvdla_runtime.so
img_1.jpg nvdla_runtime
# ./nvdla_runtime --loadable basic.nvdla --image img_1.jpg --rawdump
creating new runtime context...
Emulator starting
surface format: 0
dlaimg height: 28 x 28 x 1: LS: 32 SS: 0 Size: 896
submitting tasks...
[ 169.761591] Enter:dla_read_network_config
And the output.dimg is like this:
[ 214.048938] 11 HWLs done, totally 11 layers
[ 214.049072] Enter: dla_free_op_desc op desc index 9 ROI 0
[ 214.049544] Exit: dla_free_op_desc
[ 214.050007] Enter: dla_free_op_desc op desc index 10 ROI 0
[ 214.050250] Exit: dla_free_op_desc
[ 214.050531] Exit:dla_op_completion processor SDP group0 status=0
[ 214.050706] Exit:dla_handle_events, ret:0
[ 214.050945] Enter:dla_handle_events, processor:PDP
[ 214.051087] Exit:dla_handle_events, ret:0
[ 214.051208] Enter:dla_handle_events, processor:CDP
[ 214.051424] Exit:dla_handle_events, ret:0
[ 214.051549] Enter:dla_handle_events, processor:RUBIK
[ 214.051692] Exit:dla_handle_events, ret:0
[ 214.055361] reset engine done
Work Found!
Work Done
Shutdown signal received, exiting
Test pass
# cat output.dimg
**0 0 1 0 0 0 0 0 0 0 **
#