Using Linux Perf Tool For Nuclei RISC-V Processor

[fanghuaqi]

Linux Perf Tool is an outstanding tool that serves the significant purpose of measuring CPU performance and offers valuable assistance to developers in optimizing software.

Linux Kernel For RISC-V improved perf support for RISC-V using SBI PMU extension and Sscofpmf extension, which are both supported by Nuclei RISC-V Processor now.

• SBI PMU extension is an interface for supervisor-mode to configure and use the RISC-V hardware performance counters with assistance from the machine-mode (or hypervisor-mode)
• Sscofpmf risc-v extension is used to add an counter overflow interrupt and Privilege Mode counter Filtering

By default, now in dev_nuclei_6.1_v2/dev_nuclei_6.6_v2/dev_nuclei_6.9_v2 branches, we enabled CONFIG_PERF_EVENTS=y for Linux Kernel, which means by PMU support is enabled in Linux Kernel, if your CPU is configured with PMU, you will be able to see following output during boot up, see fd3dc31 commit for changes we made to support PMU.

We tested Linux Kernel 6.9 with PMU.

# In OpenSBI
Boot HART ISA Extensions  : sscofpmf,time,sstc
... ...
Boot HART MHPM Count      : 4
# In Linux Kernel
[   17.066711] riscv-pmu-sbi: SBI PMU extension is available
[   17.071960] riscv-pmu-sbi: 16 firmware and 6 hardware counters

But when CONFIG_PERF_EVENTS=y enabled, you will not be able to read cycle/instret/time csr, you need to execute echo 2 > /proc/sys/kernel/perf_user_access to allow user mode access these counters, and now the cycle/instret counter by default is not counted, it is controlled by PMU now, you need to use linux perf tool to enable it.

Here is a sample usage for coremark program ported by us using perf tool.

# echo 2 > /proc/sys/kernel/perf_user_access
# perf stat -e cycles -e instructions -e cache-misses -e branches -e branch-misses coremark_1core
Start to run CoreMark
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 14788
Total time (secs): 14.788000
Iterations/Sec   : 270.489586
Iterations       : 4000
Compiler version : GCC10.2.0
Compiler flags   : -Ofast -mbranch-cost=1 -mstrict-align -funroll-all-loops -finline-limit=1000 -ftree-dominator-opts -fselective-scheduling -funroll-loops -finline-functions -falign-functions=4 -falign-jumps=4 -falign-loops=4 -fipa-pta -fno-code-hoisting -fno-common -fno-if-conversion -fno-if-conversion2 -fno-tree-loop-distribute-patterns -fno-tree-vectorize -fno-tree-loop-ivcanon -fno-tree-vrp -fgcse-las --param=max-loop-header-insns=4 --param loop-max-datarefs-for-datadeps=0 --param=unroll-jam-min-percent=0 --param=max-goto-duplication-insns=0 -DMULTITHREAD=1 -DUSE_PTHREAD  -lrt -lpthread -march=rv64imafdc -mabi=lp64d
Memory location  : Please put data memory location here
                        (e.g. code in flash, data on heap etc)
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0x65c5
Correct operation validated. See README.md for run and reporting rules.
CoreMark 1.0 : 270.489586 / GCC10.2.0 -Ofast -mbranch-cost=1 -mstrict-align -funroll-all-loops -finline-limit=1000 -ftree-dominator-opts -fselective-scheduling -funroll-loops -finline-functions -falign-functions=4 -falign-jumps=4 -falign-loops=4 -fipa-pta -fno-code-hoisting -fno-common -fno-if-conversion -fno-if-conversion2 -fno-tree-loop-distribute-patterns -fno-tree-vectorize -fno-tree-loop-ivcanon -fno-tree-vrp -fgcse-las --param=max-loop-header-insns=4 --param loop-max-datarefs-for-datadeps=0 --param=unroll-jam-min-percent=0 --param=max-goto-duplication-insns=0 -DMULTITHREAD=1 -DUSE_PTHREAD  -lrt -lpthread -march=rv64imafdc -mabi=lp64d / Heap
Begin_Cycle 9223372037062820250, End_Cycle 9223372037805837188, User_Cycle 743016938 cycles
Begin_Instret 9223372037119765118, End_Instret 9223372038073768277, User_Instret 954003159 Instrets
CoreMark/MHz calc via cycle: 5.383457 CoreMark/MHz
IPC:  1.284

 Performance counter stats for 'coremark_1core':

         951534830      cycles                                                  
        1219293100      instructions                     #    1.28  insn per cycle
             13957      cache-misses                                            
         264065366      branches                                                
           6903336      branch-misses                    #    2.61% of all branches

      18.942352295 seconds time elapsed

      18.884026000 seconds user
       0.039923000 seconds sys

About Linux Perf Tool, you need to cross build it using Linux Kernel, you will need at least to cross build zlib, elfutils and libtraceevent and then build linux perf tool

• zlib 1.3.1: github.com/madler/zlib.git
• elfutils 0.191: https://sourceware.org/git/elfutils.git
• libtraceevent 1.8.3: https://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git

We provide sample script to cross build perf tool for linux kernel 6.9 for rv64imafdc:

• https://github.com/Nuclei-Software/linux/blob/dev_nuclei_6.9.y-perf/tools/perf/pre_perf.sh : build perf required library
• https://github.com/Nuclei-Software/linux/blob/dev_nuclei_6.9.y-perf/tools/perf/do_perf.sh: build perf tool itself

Tools directories must be like this:

├── linux
├── elfutils
├── libtraceevent
└── zlib

Download prebuilt Nuclei RISC-V Toolchain ( Linux/Glibc )(gcc13) from https://nucleisys.com/download.php

# Make sure above directory include source code are setup
# Make sure toolchain PATH is already setup
$ which riscv64-unknown-linux-gnu-gcc
/path/to/your/gcc/bin/riscv64-unknown-linux-gnu-gcc
$ cd linux/tools/perf
$ ls *_perf.sh
do_perf.sh  pre_perf.sh
# Build perf required third party libraries
./pre_perf.sh riscv64-unknown-linux-gnu rv64imafdc lp64d install
# Build perf tools
./do_perf.sh riscv64-unknown-linux-gnu rv64imafdc lp64d install

# Then you will be able to see perf tools in `install` folder
$ ls install/
bin/  env.sh  etc/  include/  lib/  libexec/  share/

# WARNING: Below is in a riscv linux environment, not x86 linux environment
# You just need to copy these install folder in your riscv linux environment and execute following command
# you will be able to use perf in your riscv linux
source /path/to/install/env.sh
perf --version