Skip to content
This repository has been archived by the owner on Jan 28, 2022. It is now read-only.

ldd3 project

Jump to bottom
wangqi edited this page May 9, 2021 · 12 revisions

Here we will use ldd3 project to verify the gccIDE. We have three goals.

  • Compile ldd3 project on alpine 3.13.
  • Edit ldd3 project with gccIDE.
  • Install/remove compiled module on alpine.

For those who wants to compile ldd3 project

If you just want to compile the ldd3 project, please follow these steps.

Install the linux kernel source

You need either alpine 3.13 docker container or gccIDE docker container before the following steps.

% docker pull alpine:3.13

Please run the following commands with the root privilege in the new container.

  • LDD3 project need kernel source tree to compile, on alpine, the package is linux-lts-dev.
  • musl-dev and linux-headers package provides header files required by compilation process.
# apk add linux-lts-dev musl-dev linux-headers

Get the ldd3 project source

You need a normal user/group for the following commands. Using root is not a good idea.

% cd ~/proj
% git clone https://github.com/martinezjavier/ldd3.git
% cd ldd3

Export the KERNELDIR

% export KERNELDIR=/lib/modules/5.10.29-0-lts/build/

Why we need the above KERNELDIR? If you check the Makefile in misc-modules directory. In the following Makefile, you will find how the script setups the KERNELDIR. On some system the Makefile is smart to determine the kernel location. On my Docker desktop for Mac, uname -r returns 5.10.25-linuxkit, while the package linux-lts-dev believes it's 5.10.29-0-lts. Amazing?

# To build modules outside of the kernel tree, we run "make"
# in the kernel source tree; the Makefile these then includes this
# Makefile once again.
# This conditional selects whether we are being included from the
# kernel Makefile or not.

LDDINC=$(PWD)/../include
EXTRA_CFLAGS += -I$(LDDINC)

ifeq ($(KERNELRELEASE),)

    # Assume the source tree is where the running kernel was built
    # You should set KERNELDIR in the environment if it's elsewhere
    KERNELDIR ?= /lib/modules/$(shell uname -r)/build
    # The current directory is passed to sub-makes as argument
    PWD := $(shell pwd)

modules:
        $(MAKE) -C $(KERNELDIR) M=$(PWD) modules

modules_install:
        $(MAKE) -C $(KERNELDIR) M=$(PWD) modules_install

clean:
        rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions *.mod modules.order *.symvers

.PHONY: modules modules_install clean

else
    # called from kernel build system: just declare what our modules are
    obj-m := hello.o hellop.o seq.o jiq.o sleepy.o complete.o \
             silly.o faulty.o kdatasize.o kdataalign.o jit.o
endif

Compile ldd3 project

Now you can compile the ldd3 project as a whole project. During the compilation process, some sub-project may warn you. You can ignore them.

% cd ~/proj/ldd3
% make

Or you can just compile sub-project in the sub-directory.

$ cd ~/proj/ldd3/misc-modules
% make

For those who wants to edit ldd3 project

If you want to edit the ldd3 project, please follow these steps. Remember the previous steps is also needed, edit/compile is a cycle.

Get the gccIDE

Follow the instructions in Quick Guide section to set up the gccIDE. Don't forget to create the proj-vol volume.

Make with bear

The following is the example to generate compile_commands.json for scullc. Here, bear will run make as usual and generate compile_commands.json in the current directory. See here to know more about compile_commands.json.

We also run make clean to clean the result. Here we only concern the generated compile_commands.json file.

% cd ~/proj/ldd3/scullc
% bear -- make
make -C /lib/modules/5.10.29-0-lts/build M=/home/ide/proj/ldd3/scullc modules
make[1]: Entering directory '/usr/src/linux-headers-5.10.29-0-lts'
  CC [M]  /home/ide/proj/ldd3/scullc/main.o
  CC [M]  /home/ide/proj/ldd3/scullc/scull-shared/scull-async.o
  LD [M]  /home/ide/proj/ldd3/scullc/scullc.o
  MODPOST /home/ide/proj/ldd3/scullc/Module.symvers
  CC [M]  /home/ide/proj/ldd3/scullc/scullc.mod.o
  LD [M]  /home/ide/proj/ldd3/scullc/scullc.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.10.29-0-lts'
% make clean
$ ls 
Makefile               main.c                 scull-shared           scullc_load
compile_commands.json  mmap.c                 scullc.h               scullc_unload

Edit source file

Now you can edit any source file under scullc directory. You can also verify the Code completion, Find definition/declaration and find references. Please repeat the above process for other sub-directory to taste the support provided by gccIDE.

You can see generating the compile_commands.json file with bear is easy. CMake also supports generating the compile_commands.json.

The compile_commands.json should be enough, In case you need, please copy and paste the following content into .ccls file in project root directory ~/proj/ldd3. Actually, the following parameters were extracted from compile_commands.json by myself. Strictly speaking, the following step is not necessary if you successfully generate the compile_commands.json. It's just the last remedy in case you need it.

gcc

%h
%c -std=gnu89
-DMODULE
-DCC_USING_FENTRY
-D__KERNEL__
-isystem
/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1/include
-I
/lib/modules/5.10.29-0-lts/build/arch/x86/include
-I
/lib/modules/5.10.29-0-lts/build/arch/x86/include/generated
-I
/lib/modules/5.10.29-0-lts/build/include
-I
/lib/modules/5.10.29-0-lts/build/arch/x86/include/uapi
-I
/lib/modules/5.10.29-0-lts/build/arch/x86/include/generated/uapi
-I
/lib/modules/5.10.29-0-lts/build/include/uapi
-I
/lib/modules/5.10.29-0-lts/build/include/generated/uapi
--gcc-toolchain=/usr

For those who want to install/remove the developed module

If you want to insmod / rmmod the developed module in docker environment you will fail. You can edit/compile the kernel module in docker environment. But you can't install/remove modules in docker environment. Even run docker with docker run --privileged options, you will fail again.

When I tried to use Docker desktop for Mac to perform the insmod command, the docker kernel refused with the "Invalid module format". The root cause is: the Kernel from which you build your kernel module and to which you are inserting module should be of the same version. See LDD3 Page 26 for more detail.

Docker desktop for Mac use kernel 5.10.25-linuxkit, gccIDE is based on alpine 3.13, the kernel version is 5.10.29-0-lts for package linux-lts-dev. Even you can build the kernel module in gccIDE. You can't install the new module in the docker environment. Their kernel version is simply different.

The quick solution is:

  • Create an alpine VM in virtualbox.
  • For development job, use gccIDE to access the source code.
  • For kernel install and remove job, use the alpine VM to access the kernel module.
  • ldd3 project on GitHub can help us to share the code between alpine VM and gccIDE.

The idea behind this solution is to change the target kernel version to pass the kernel version dependency check. Now you can insmod and rmmod the kernel module you just developed.

Clone this wiki locally