bpf tc can't load by tc or cargo bpf load

[Sherlock-Holo]

I'm using Archlinux

Linux sherlockholo-pc 5.14.14-arch1-1 #1 SMP PREEMPT Wed, 20 Oct 2021 21:35:18 +0000 x86_64 GNU/Linux

I try this codes

#![no_std]
#![no_main]

use redbpf_probes::tc::prelude::*;
// use redbpf_probes::xdp::prelude::*;

program!(0xFFFFFFFE, "GPL");

#[tc_action]
fn test(skb: SkBuff) -> TcActionResult {
    Ok(TcAction::Ok)
}

and build with cargo bpf build

When I try to load it by sudo tc filter add ingress bpf da obj target/bpf/programs/learn_bpf/learn_bpf.elf sec tc_action/test

it reports

Error fetching program/map!
Unable to load program

also I try to use cargo-bpf, sudo -E cargo bpf load -i lo target/bpf/programs/learn_bpf/learn_bpf.elf, it just stuck without any output.

How should I do to load the bpf tc program?

[rhdxmr]

Hello @Sherlock-Holo

Did you check this example?

redbpf/examples/example-userspace/examples/tc-map-share.rs

Line 55 in a9b98c2

for sec in &[".BTF.ext", ".eh_frame", ".text", ".BTF"] {

Some sections are needed to be removed to load tc program

$ llvm-strip --no-strip-all --remove-section .BTF.ext target/bpf/programs/learn_bpf/learn_bpf.elf
... and so on...

and then create qdisc if not exists

$ sudo tc qdisc add dev lo clsact

and try loading it

$ sudo tc filter add ingress dev lo bpf da obj target/bpf/programs/learn_bpf/learn_bpf.elf sec tc_action/test

[Sherlock-Holo]

@rhdxmr

I try llvm-strip --no-strip-all --remove-section .BTF.ext --remove-section .eh_frame --remove-section .text --remove-section .BTF learn_bpf.elf

but it reports

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_loc+0x8) has relocation against symbol ''

even I split the --remove-section arguments, strip 1 section at at a time, it will still failed in remove the .text

[Sherlock-Holo]

my llvm-strip version is

llvm-strip --version
llvm-strip, compatible with GNU strip
LLVM (http://llvm.org/):
LLVM version 12.0.1
Optimized build.
Default target: x86_64-pc-linux-gnu
Host CPU: znver3

[rhdxmr]

Then you sholud remove .debug_loc first.
$ llvm-strip --remove-section .debug_loc ....

And then try removing .text section.
$ llvm-strip --remove-section .text ...

[Sherlock-Holo]

llvm-strip --no-strip-all --remove-section .debug_loc learn_bpf.elf success
llvm-strip --no-strip-all --remove-section .text learn_bpf.elf failed, report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_info+0x5a) has relocation against symbol ''

so try llvm-strip --no-strip-all --remove-section .debug_info learn_bpf.elf success
but remove .text still failed,

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_ranges+0x0) has relocation against symbol ''

so keep trying llvm-strip --no-strip-all --remove-section .debug_ranges learn_bpf.elf, remove .text now report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_frame+0x18) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_frame learn_bpf.elf, then remove .text report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_line+0x30) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_line learn_bpf.elf, now can remove .text

now can load by tc

[Sherlock-Holo]

@rhdxmr well... that's so hard to try to remove the .text section :(

I have try similar bpf tc codes, written by c, and compile by
clang -I ./headers -Wall -O3 -target bpf -c main.c -o main.o (please ignore the -I ./headers, actually there's nothing there)

and load by tc success directly, no need to remove any section

is a plan to let redbpf can load by tc directly without removing some section?

[rhdxmr]

I understand your feeling because I also thought that running those commands are unintitive and bothering steps.
But I couldn't solve this problem when I faced it before. Because I was not sure why .text section is generated during compiling bpf program and I couldn't find how to prevent this.

And there is also another problem. .BTF section is unnecessary and problematic when running tc utility but it is useful in many cases other than tc case. So .BTF related sections are needed to be included to bpf obj optionally.

I hope to handle these problems and help users but I guess it takes time.

[Sherlock-Holo]

for now the problem of tc can be bypassed, but cargo bpf load is still sticking

It seems cargo-bpf doesn't handle bpf tc?

https://github.com/foniod/redbpf/blob/main/cargo-bpf/src/load.rs#L35

[Sherlock-Holo]

llvm-strip --no-strip-all --remove-section .debug_loc learn_bpf.elf success llvm-strip --no-strip-all --remove-section .text learn_bpf.elf failed, report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_info+0x5a) has relocation against symbol ''

so try llvm-strip --no-strip-all --remove-section .debug_info learn_bpf.elf success but remove .text still failed,

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_ranges+0x0) has relocation against symbol ''

so keep trying llvm-strip --no-strip-all --remove-section .debug_ranges learn_bpf.elf, remove .text now report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_frame+0x18) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_frame learn_bpf.elf, then remove .text report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_line+0x30) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_line learn_bpf.elf, now can remove .text

now can load by tc

@rhdxmr though strip these sections can let tc to load the bpf tc, but it seems it lost a lot of debug info, I am writing a TCP port block bpf tc, and it denied by bpf Verifier, without any analysis info

[rhdxmr]

for now the problem of tc can be bypassed, but cargo bpf load is still sticking

It seems cargo-bpf doesn't handle bpf tc?

https://github.com/foniod/redbpf/blob/main/cargo-bpf/src/load.rs#L35

As you said, cargo bpf load does not load tc_action BPF program.
Most of BPF programs are loaded automatically here but tc_action BPF program is an exception.

redbpf/redbpf/src/load/loader.rs

Lines 39 to 43 in a9b98c2

	for program in module.programs.iter_mut() {
	program
	.load(module.version, module.license.clone())
	.map_err(|e| LoaderError::LoadError(program.name().to_string(), e))?;
	}

[Sherlock-Holo]

Are there any reasons the loader not support tc bpf? I also read the loader codes, and doesn't found any tc load codes

[rhdxmr]

@rhdxmr though strip these sections can let tc to load the bpf tc, but it seems it lost a lot of debug info, I am writing a TCP port block bpf tc, and it denied by bpf Verifier, without any analysis info

I wrote a BPF program that is doing wrong pointer arithmetic.

#![no_std]
#![no_main]
use redbpf_probes::tc::prelude::*;

program!(0xFFFFFFFE, "GPL");

#[tc_action]
unsafe fn test(skb: SkBuff) -> TcActionResult {
    let invalid_p = skb.skb.add(10000) as *const u8;
    let a = *invalid_p;
    if a == 0 {
        Ok(TcAction::Ok)
    } else {
        Ok(TcAction::Shot)
    }
}

All sections that are problematic with tc utility are removed.

$ llvm-strip --no-strip-all --remove-section .BTF --remove-section .BTF.ext --remove-section .debug_loc --remove-section .debug_info --remove-section .debug_ranges --remove-section .debug_pubnames --remove-section .debug_pubtypes --remove-section .debug_frame --remove-section .debug_line --remove-section .text target/bpf/programs/learn_bpf/learn_bpf.elf
$ llvm-objdump --headers target/bpf/programs/learn_bpf/learn_bpf.elf

target/bpf/programs/learn_bpf/learn_bpf.elf:    file format elf64-bpf

Sections:
Idx Name           Size     VMA              Type
  0                00000000 0000000000000000 
  1 tc_action/test 00000038 0000000000000000 TEXT
  2 license        00000004 0000000000000000 DATA
  3 version        00000004 0000000000000000 DATA
  4 .debug_abbrev  000001ff 0000000000000000 
  5 .debug_str     0000071f 0000000000000000 
  6 .symtab        000000d8 0000000000000000 
  7 .strtab        00000076 0000000000000000 

And I loaded the BPF program with tc

$ sudo tc filter add ingress dev lo bpf da obj target/bpf/programs/learn_bpf/learn_bpf.elf sec tc_action/test

Prog section 'tc_action/test' rejected: Permission denied (13)!
 - Type:         3
 - Instructions: 7 (0 over limit)
 - License:      GPL

Verifier analysis:

0: (07) r1 += 1840000
1: (71) r1 = *(u8 *)(r1 +0)
dereference of modified ctx ptr R1 off=1840000 disallowed
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

Error fetching program/map!
Unable to load program

The output shows BPF verifier log that dumps BPF byte code.
How about adding verbose to tc command? Does it work?

$ sudo tc filter add ingress dev lo bpf da obj target/bpf/programs/learn_bpf/learn_bpf.elf sec tc_action/test verbose

[rhdxmr]

Are there any reasons the loader not support tc bpf? I also read the loader codes, and doesn't found any tc load codes

tc_action BPF program is intended to be loaded by tc utility. And.. I am not sure RedBPF is a suitable tool for loading tc_action BPF programs.

[Sherlock-Holo]

oh, add the verb argument it prints the verifier analysis info, and the error message is

R3 pointer arithmetic with <<= operator prohibited

thanks for your tips

[Sherlock-Holo]

oh, add the verb argument it prints the verifier analysis info, and the error message is

R3 pointer arithmetic with <<= operator prohibited

thanks for your tips

the problem is similar: I convert a *mut pointer to u32, it's not correct, and convert u32 to *mut c_void is also incorrect, it should be converted to usize or u64 at first

[rhdxmr]

oh, add the verb argument it prints the verifier analysis info, and the error message is

R3 pointer arithmetic with <<= operator prohibited

thanks for your tips

the problem is similar: I convert a *mut pointer to u32, it's not correct, and convert u32 to *mut c_void is also incorrect, it should be converted to usize or u64 at first

I guess I am not on the same page.. I don't know what you are saying. Can you elaborate?

[Sherlock-Holo]

sorry for the confusing content

I send it to the wrong web page, that's my fault.

[rhdxmr]

Ah ha that's OK.
Probably you are busy today.

[Sherlock-Holo]

llvm-strip --no-strip-all --remove-section .debug_loc learn_bpf.elf success llvm-strip --no-strip-all --remove-section .text learn_bpf.elf failed, report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_info+0x5a) has relocation against symbol ''

so try llvm-strip --no-strip-all --remove-section .debug_info learn_bpf.elf success but remove .text still failed,

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_ranges+0x0) has relocation against symbol ''

so keep trying llvm-strip --no-strip-all --remove-section .debug_ranges learn_bpf.elf, remove .text now report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_frame+0x18) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_frame learn_bpf.elf, then remove .text report

llvm-strip: error: 'learn_bpf.elf': section '.text' cannot be removed: (.debug_line+0x30) has relocation against symbol ''

llvm-strip --no-strip-all --remove-section .debug_line learn_bpf.elf, now can remove .text
now can load by tc

@rhdxmr though strip these sections can let tc to load the bpf tc, but it seems it lost a lot of debug info, I am writing a TCP port block bpf tc, and it denied by bpf Verifier, without any analysis info

I create a shell script to remove those sections, but I am not familiar with the elf section. Are there any sections is necessary for other BPF program? if all of them are optional, maybe we can remove them at

redbpf/examples/example-userspace/examples/tc-map-share.rs

Line 55 in a9b98c2

for sec in &[".BTF.ext", ".eh_frame", ".text", ".BTF"] {

[rhdxmr]

It's ok to remove .debug_xxxx sections at tc-map-share.rs. It's just an example program so it does not affect other BPF programs.

[d0u9]

Is the rust code for bpf are directly compiled by llvm? Something like:

llvm [options] -arch bpf -o bpf.rust

Or multiple steps are utilized to translate .rust to .xxx to .bpf?

[rhdxmr]

@d0u9
Good question.
You can find an elaborate answer here: https://blog.redsift.com/labs/oxidised-ebpf-ii-taming-llvm/

[d0u9]

I just found that article, and that is much complex than I thought before.

[rhdxmr]

@Sherlock-Holo

@rhdxmr well... that's so hard to try to remove the .text section :(

I have try similar bpf tc codes, written by c, and compile by clang -I ./headers -Wall -O3 -target bpf -c main.c -o main.o (please ignore the -I ./headers, actually there's nothing there)

and load by tc success directly, no need to remove any section

is a plan to let redbpf can load by tc directly without removing some section?

I agreed with your opinion so I tried to remove .text section from ELF relocatable.
The problem has been solved by this PR #224
The fix will be released soon with a RedBPF v2.2.0 release.
Hence, with the v2.2.0 perhaps don't need to struggle with removing the .text section manually.

Thanks,