Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug Report] Failed to start dae on kernel 6.9rc1 #482

Closed
3 tasks done
sakarie9 opened this issue Mar 26, 2024 · 15 comments · Fixed by #483
Closed
3 tasks done

[Bug Report] Failed to start dae on kernel 6.9rc1 #482

sakarie9 opened this issue Mar 26, 2024 · 15 comments · Fixed by #483
Assignees
@jschwinger233

Comments

@sakarie9
Copy link

Checks

  • I have searched the existing issues
  • I have read the documentation
  • Is it your first time sumbitting an issue

Current Behavior

3月 26 09:49:46 systemd[1]: Starting dae Service...
3月 26 09:49:46 dae[21143]: level=info msg="Include config files: [/etc/dae/config.dae]"
3月 26 09:49:46 dae[21143]: level=warning msg="No node found."
3月 26 09:49:46 dae[21143]: level=warning msg="No interface to bind."
3月 26 09:49:46 dae[21143]: level=info msg="Loading eBPF programs and maps into the kernel..."
3月 26 09:49:46 dae[21143]: level=info msg="The loading process takes about 120MB free memory, which will be released after loading. Insufficient memory will cause loading failure."
3月 26 09:49:46 dae[21143]: time="2024-03-26T09:49:46+08:00" level=fatal msg="load eBPF objects: field TproxyLanIngress: program tproxy_lan_ingress: load program: invalid argument: invalid func unknown#195896080 (376 line(s) omitted)"
3月 26 09:49:46 systemd[1]: dae.service: Main process exited, code=exited, status=1/FAILURE
3月 26 09:49:46 systemd[1]: dae.service: Failed with result 'exit-code'.
3月 26 09:49:46 systemd[1]: Failed to start dae Service.

Expected Behavior

Succeeded

Steps to Reproduce

  1. Use kernel 6.9.0-rc1
  2. Run dae

Environment

  • Dae version (use dae --version): 0.5.1
  • OS (e.g cat /etc/os-release): Arch Linux
  • Kernel (e.g. uname -a): 6.9.0-rc1-1-cachyos-rc-lto
  • Others:

Anything else?

minimal bootable config and working config pre 6.9 both crashed
@dae-prow
Copy link
Contributor

dae-prow bot commented Mar 26, 2024

Thanks for opening this issue!

@mzz2017
Copy link
Contributor

mzz2017 commented Mar 26, 2024

@sakarie9 按照文档检查一下内核选项

@sakarie9
Copy link
Author

sakarie9 commented Mar 26, 2024 •
edited
Loading

@sakarie9 按照文档检查一下内核选项

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_CGROUPS=y
CONFIG_KPROBES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_ACT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_KPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y

@mzz2017
Copy link
Contributor

mzz2017 commented Mar 26, 2024

@sakarie9 我（怀疑）是 bpf_printk 的问题，这个需要一个内核选项，你查一下看看有没有

@sakarie9
Copy link
Author

@sakarie9 我（怀疑）是 bpf_printk 的问题，这个需要一个内核选项，你查一下看看有没有

这些吗

CONFIG_PRINTK_INDEX=y
CONFIG_PRINTK=y
CONFIG_SND_VERBOSE_PRINTK=y
# CONFIG_USB_G_DBGP_PRINTK is not set
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_EARLY_PRINTK_USB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_EARLY_PRINTK_USB_XDBC=y

CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
# CONFIG_BPF_PRELOAD is not set
CONFIG_BPF_LSM=y
# end of BPF subsystem
CONFIG_CGROUP_BPF=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_BPF_LINK=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_BPF_LIRC_MODE2=y
# HID-BPF support
CONFIG_HID_BPF=y
# end of HID-BPF support
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
# CONFIG_TEST_BPF is not set

@jschwinger233
Copy link
Member

能把 log_level 换成 fatal 看看 verifier 日志吗？

@sakarie9
Copy link
Author 

3月 26 10:53:23 systemd[1]: Starting dae Service...
3月 26 10:53:23 dae[621]: level=fatal msg="0: R1=ctx() R10=fp0
3月 26 10:53:23 dae[621]: ; int tproxy_lan_ingress(struct __sk_buff *skb) { @ tproxy.c:854
3月 26 10:53:23 dae[621]: 0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
3月 26 10:53:23 dae[621]: ; if (get_link_h_len(skb->ifindex, &link_h_len)) { @ tproxy.c:864
3月 26 10:53:23 dae[621]: 1: (61) r1 = *(u32 *)(r6 +40)         ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6_w=ctx()
3月 26 10:53:23 dae[621]: 2: (63) *(u32 *)(r10 -176) = r1       ; R1_w=scalar(id=1,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-176=????scalar(id=1,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 3: (bf) r2 = r10                      ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 4: (07) r2 += -176                    ; R2_w=fp-176
3月 26 10:53:23 dae[621]: ; __u32 *plink_h_len = bpf_map_lookup_elem(&linklen_map, &ifindex); @ tproxy.c:806
3月 26 10:53:23 dae[621]: 5: (18) r1 = 0xffff8ec815422400       ; R1_w=map_ptr(map=linklen_map,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 7: (85) call bpf_map_lookup_elem#1    ; R0_w=map_value_or_null(id=2,map=linklen_map,ks=4,vs=4)
3月 26 10:53:23 dae[621]: ; if (!plink_h_len) { @ tproxy.c:807
3月 26 10:53:23 dae[621]: 8: (55) if r0 != 0x0 goto pc+2 11: R0_w=map_value(map=linklen_map,ks=4,vs=4) R6_w=ctx() R10=fp0 fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; *link_h_len = *plink_h_len; @ tproxy.c:810
3月 26 10:53:23 dae[621]: 11: (61) r1 = *(u32 *)(r0 +0)         ; R0_w=map_value(map=linklen_map,ks=4,vs=4) R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 12: (63) *(u32 *)(r10 -124) = r1      ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-128=mmmm????
3月 26 10:53:23 dae[621]: ; if (link_h_len == ETH_HLEN) { @ tproxy.c:481
3月 26 10:53:23 dae[621]: 13: (55) if r1 != 0xe goto pc+13      ; R1_w=14
3月 26 10:53:23 dae[621]: 14: (b7) r7 = 0                       ; R7_w=0
3月 26 10:53:23 dae[621]: 15: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; ret = bpf_skb_load_bytes(skb, offset, ethh, sizeof(struct ethhdr)); @ tproxy.c:482
3月 26 10:53:23 dae[621]: 16: (07) r3 += -16                    ; R3_w=fp-16
3月 26 10:53:23 dae[621]: 17: (b7) r9 = 14                      ; R9_w=14
3月 26 10:53:23 dae[621]: 18: (bf) r1 = r6                      ; R1_w=ctx() R6_w=ctx()
3月 26 10:53:23 dae[621]: 19: (b7) r2 = 0                       ; R2_w=0
3月 26 10:53:23 dae[621]: 20: (b7) r4 = 14                      ; R4_w=14
3月 26 10:53:23 dae[621]: 21: (85) call bpf_skb_load_bytes#26   ; R0=scalar() fp-8=??mmmmmm fp-16=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if (ret) { @ tproxy.c:483
3月 26 10:53:23 dae[621]: 22: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 23: (77) r0 >>= 32                    ; R0_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 24: (15) if r0 == 0x0 goto pc+10      ; R0_w=0 R6=ctx() R7=0 R9=14 R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-128=mmmm???? fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; ethh->h_proto = skb->protocol; @ tproxy.c:491
3月 26 10:53:23 dae[621]: 35: (b7) r7 = 0                       ; R7_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(iph, 0, sizeof(struct iphdr)); @ tproxy.c:496
3月 26 10:53:23 dae[621]: 36: (63) *(u32 *)(r10 -24) = r7       ; R7_w=0 R10=fp0 fp-24=????0
3月 26 10:53:23 dae[621]: 37: (7b) *(u64 *)(r10 -32) = r7       ; R7_w=0 R10=fp0 fp-32_w=0
3月 26 10:53:23 dae[621]: 38: (7b) *(u64 *)(r10 -40) = r7       ; R7_w=0 R10=fp0 fp-40_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(ipv6h, 0, sizeof(struct ipv6hdr)); @ tproxy.c:497
3月 26 10:53:23 dae[621]: 39: (7b) *(u64 *)(r10 -80) = r7       ; R7_w=0 R10=fp0 fp-80_w=0
3月 26 10:53:23 dae[621]: 40: (7b) *(u64 *)(r10 -72) = r7       ; R7_w=0 R10=fp0 fp-72_w=0
3月 26 10:53:23 dae[621]: 41: (7b) *(u64 *)(r10 -64) = r7       ; R7_w=0 R10=fp0 fp-64_w=0
3月 26 10:53:23 dae[621]: 42: (7b) *(u64 *)(r10 -56) = r7       ; R7_w=0 R10=fp0 fp-56_w=0
3月 26 10:53:23 dae[621]: 43: (7b) *(u64 *)(r10 -48) = r7       ; R7_w=0 R10=fp0 fp-48_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(icmp6h, 0, sizeof(struct icmp6hdr)); @ tproxy.c:498
3月 26 10:53:23 dae[621]: 44: (7b) *(u64 *)(r10 -88) = r7       ; R7_w=0 R10=fp0 fp-88_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(tcph, 0, sizeof(struct tcphdr)); @ tproxy.c:499
3月 26 10:53:23 dae[621]: 45: (7b) *(u64 *)(r10 -112) = r7      ; R7_w=0 R10=fp0 fp-112_w=0
3月 26 10:53:23 dae[621]: 46: (7b) *(u64 *)(r10 -104) = r7      ; R7_w=0 R10=fp0 fp-104_w=0
3月 26 10:53:23 dae[621]: 47: (63) *(u32 *)(r10 -96) = r7       ; R7_w=0 R10=fp0 fp-96=????0
3月 26 10:53:23 dae[621]: ; __builtin_memset(udph, 0, sizeof(struct udphdr)); @ tproxy.c:500
3月 26 10:53:23 dae[621]: 48: (7b) *(u64 *)(r10 -120) = r7      ; R7_w=0 R10=fp0 fp-120_w=0
3月 26 10:53:23 dae[621]: 49: (bf) r1 = r10                     ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 50: (07) r1 += -16                    ; R1_w=fp-16
3月 26 10:53:23 dae[621]: ; if (ethh->h_proto == bpf_htons(ETH_P_IP)) { @ tproxy.c:504
3月 26 10:53:23 dae[621]: 51: (69) r2 = *(u16 *)(r1 +12)        ; R1_w=fp-16 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 52: (55) if r2 != 0x8 goto pc+28      ; R2_w=8
3月 26 10:53:23 dae[621]: 53: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; if ((ret = bpf_skb_load_bytes(skb, offset, iph, sizeof(struct iphdr)))) { @ tproxy.c:506
3月 26 10:53:23 dae[621]: 54: (07) r3 += -40                    ; R3_w=fp-40
3月 26 10:53:23 dae[621]: 55: (bf) r1 = r6                      ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 56: (bf) r2 = r9                      ; R2_w=14 R9=14
3月 26 10:53:23 dae[621]: 57: (b7) r4 = 20                      ; R4_w=20
3月 26 10:53:23 dae[621]: 58: (85) call bpf_skb_load_bytes#26   ; R0=scalar() fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 59: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 60: (77) r0 >>= 32                    ; R0_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 61: (55) if r0 != 0x0 goto pc-37      ; R0_w=0
3月 26 10:53:23 dae[621]: 62: (bf) r1 = r10                     ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; offset += iph->ihl * 4; @ tproxy.c:510
3月 26 10:53:23 dae[621]: 63: (07) r1 += -40                    ; R1_w=fp-40
3月 26 10:53:23 dae[621]: 64: (71) r2 = *(u8 *)(r1 +0)          ; R1_w=fp-40 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 65: (67) r2 <<= 2                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=1020,var_off=(0x0; 0x3fc))
3月 26 10:53:23 dae[621]: 66: (57) r2 &= 60                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=60,var_off=(0x0; 0x3c))
3月 26 10:53:23 dae[621]: 67: (0f) r9 += r2                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=60,var_off=(0x0; 0x3c)) R9_w=scalar(smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c))
3月 26 10:53:23 dae[621]: ; *l4proto = iph->protocol; @ tproxy.c:513
3月 26 10:53:23 dae[621]: 68: (71) r8 = *(u8 *)(r1 +9)          ; R1_w=fp-40 R8_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: ; switch (iph->protocol) { @ tproxy.c:514
3月 26 10:53:23 dae[621]: 69: (15) if r8 == 0x11 goto pc+56     ; R8_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
3月 26 10:53:23 dae[621]: 70: (55) if r8 != 0x6 goto pc-46      ; R8=6
3月 26 10:53:23 dae[621]: 71: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_skb_load_bytes(skb, offset, tcph, sizeof(struct tcphdr)))) { @ tproxy.c:517
3月 26 10:53:23 dae[621]: 72: (07) r3 += -112                   ; R3_w=fp-112
3月 26 10:53:23 dae[621]: 73: (bf) r1 = r6                      ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 74: (bf) r2 = r9                      ; R2_w=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c)) R9=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c))
3月 26 10:53:23 dae[621]: 75: (b7) r4 = 20                      ; R4_w=20
3月 26 10:53:23 dae[621]: 76: (85) call bpf_skb_load_bytes#26   ; R0_w=scalar() fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if ((ret = @ tproxy.c:516
3月 26 10:53:23 dae[621]: 77: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 78: (77) r0 >>= 32                    ; R0=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 79: (15) if r0 == 0x0 goto pc+55 135: R0=0 R6=ctx() R7=0 R8=6 R9=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c)) R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0 fp-88=0 fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=0 fp-128=mmmm???? fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; if ((ret = @ tproxy.c:523
3月 26 10:53:23 dae[621]: 135: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: 136: (05) goto pc+67
3月 26 10:53:23 dae[621]: ; if (ret) { @ tproxy.c:869
3月 26 10:53:23 dae[621]: 204: (67) r1 <<= 32                   ; R1_w=0
3月 26 10:53:23 dae[621]: 205: (77) r1 >>= 32                   ; R1_w=0
3月 26 10:53:23 dae[621]: 206: (55) if r1 != 0x0 goto pc-182    ; R1_w=0
3月 26 10:53:23 dae[621]: 207: (15) if r8 == 0x3a goto pc-183   ; R8=6
3月 26 10:53:23 dae[621]: 208: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(tuples, 0, sizeof(*tuples)); @ tproxy.c:354
3月 26 10:53:23 dae[621]: 209: (7b) *(u64 *)(r10 -144) = r1     ; R1_w=0 R10=fp0 fp-144_w=0
3月 26 10:53:23 dae[621]: 210: (7b) *(u64 *)(r10 -136) = r1     ; R1_w=0 R10=fp0 fp-136_w=0
3月 26 10:53:23 dae[621]: 211: (7b) *(u64 *)(r10 -152) = r1     ; R1_w=0 R10=fp0 fp-152_w=0
3月 26 10:53:23 dae[621]: 212: (7b) *(u64 *)(r10 -160) = r1     ; R1_w=0 R10=fp0 fp-160_w=0
3月 26 10:53:23 dae[621]: 213: (7b) *(u64 *)(r10 -168) = r1     ; R1_w=0 R10=fp0 fp-168_w=0
3月 26 10:53:23 dae[621]: 214: (7b) *(u64 *)(r10 -176) = r1     ; R1_w=0 R10=fp0 fp-176_w=0
3月 26 10:53:23 dae[621]: ; tuples->five.l4proto = l4proto; @ tproxy.c:355
3月 26 10:53:23 dae[621]: 215: (73) *(u8 *)(r10 -140) = r8      ; R8=6 R10=fp0 fp-144_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:357
3月 26 10:53:23 dae[621]: 216: (61) r1 = *(u32 *)(r6 +16)       ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 217: (55) if r1 != 0x8 goto pc+13     ; R1_w=8
3月 26 10:53:23 dae[621]: 218: (18) r1 = 0xffff0000             ; R1_w=0xffff0000
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:358
3月 26 10:53:23 dae[621]: 220: (63) *(u32 *)(r10 -168) = r1     ; R1_w=0xffff0000 R10=fp0 fp-168_w=mmmm0xffff0000
3月 26 10:53:23 dae[621]: ; tuples->five.dip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:361
3月 26 10:53:23 dae[621]: 221: (63) *(u32 *)(r10 -152) = r1     ; R1_w=0xffff0000 R10=fp0 fp-152_w=mmmm0xffff0000
3月 26 10:53:23 dae[621]: 222: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:358
3月 26 10:53:23 dae[621]: 223: (07) r1 += -40                   ; R1_w=fp-40
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[3] = iph->saddr; @ tproxy.c:359
3月 26 10:53:23 dae[621]: 224: (61) r2 = *(u32 *)(r1 +12)       ; R1_w=fp-40 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: 225: (63) *(u32 *)(r10 -164) = r2     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-168_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuples->five.dip.u6_addr32[3] = iph->daddr; @ tproxy.c:362
3月 26 10:53:23 dae[621]: 226: (61) r2 = *(u32 *)(r1 +16)       ; R1_w=fp-40 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-24=????mmmm
3月 26 10:53:23 dae[621]: 227: (63) *(u32 *)(r10 -148) = r2     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-152_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; return (iph->tos & 0xfc) >> 2; @ tproxy.c:345
3月 26 10:53:23 dae[621]: 228: (71) r2 = *(u8 *)(r1 +1)         ; R1_w=fp-40 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 229: (77) r2 >>= 2                    ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: 230: (05) goto pc+34
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 265: (73) *(u8 *)(r10 -136) = r2      ; R2=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: ; if (l4proto == IPPROTO_TCP) { @ tproxy.c:372
3月 26 10:53:23 dae[621]: 266: (55) if r8 != 0x6 goto pc+7      ; R8=6
3月 26 10:53:23 dae[621]: 267: (b7) r3 = 2                      ; R3_w=2
3月 26 10:53:23 dae[621]: 268: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; } else { @ tproxy.c:375
3月 26 10:53:23 dae[621]: 269: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 270: (bf) r2 = r1                     ; R1_w=fp-112 R2_w=fp-112
3月 26 10:53:23 dae[621]: 271: (0f) r2 += r3                    ; R2_w=fp-110 R3_w=2
3月 26 10:53:23 dae[621]: 272: (b7) r3 = 0                      ; R3_w=0
3月 26 10:53:23 dae[621]: 273: (05) goto pc+6
3月 26 10:53:23 dae[621]: 280: (0f) r1 += r3                    ; R1_w=fp-112 R3=0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 281: (69) r1 = *(u16 *)(r1 +0)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: 282: (6b) *(u16 *)(r10 -144) = r1     ; R1_w=scalar(id=5,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmscalar(id=5,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
3月 26 10:53:23 dae[621]: 283: (69) r1 = *(u16 *)(r2 +0)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R2=fp-110 fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: 284: (6b) *(u16 *)(r10 -142) = r1     ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 285: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: ; struct bpf_sock_tuple tuple = {0}; @ tproxy.c:893
3月 26 10:53:23 dae[621]: 286: (63) *(u32 *)(r10 -184) = r1     ; R1_w=0 R10=fp0 fp-184=????0
3月 26 10:53:23 dae[621]: 287: (7b) *(u64 *)(r10 -192) = r1     ; R1_w=0 R10=fp0 fp-192_w=0
3月 26 10:53:23 dae[621]: 288: (7b) *(u64 *)(r10 -200) = r1     ; R1_w=0 R10=fp0 fp-200_w=0
3月 26 10:53:23 dae[621]: 289: (7b) *(u64 *)(r10 -208) = r1     ; R1_w=0 R10=fp0 fp-208_w=0
3月 26 10:53:23 dae[621]: 290: (7b) *(u64 *)(r10 -216) = r1     ; R1_w=0 R10=fp0 fp-216_w=0
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:900
3月 26 10:53:23 dae[621]: 291: (61) r1 = *(u32 *)(r6 +16)       ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 292: (7b) *(u64 *)(r10 -320) = r8     ; R8=6 R10=fp0 fp-320_w=6
3月 26 10:53:23 dae[621]: 293: (55) if r1 != 0x8 goto pc+24     ; R1_w=8
3月 26 10:53:23 dae[621]: 294: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; tuple.ipv4.daddr = tuples.five.dip.u6_addr32[3]; @ tproxy.c:901
3月 26 10:53:23 dae[621]: 295: (07) r1 += -216                  ; R1_w=fp-216
3月 26 10:53:23 dae[621]: 296: (61) r2 = *(u32 *)(r10 -148)     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-152=mmmmmmmm
3月 26 10:53:23 dae[621]: 297: (63) *(u32 *)(r1 +4) = r2        ; R1_w=fp-216 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-216_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuple.ipv4.saddr = tuples.five.sip.u6_addr32[3]; @ tproxy.c:902
3月 26 10:53:23 dae[621]: 298: (61) r2 = *(u32 *)(r10 -164)     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-168=mmmmmmmm
3月 26 10:53:23 dae[621]: 299: (63) *(u32 *)(r1 +0) = r2        ; R1_w=fp-216 R2_w=scalar(id=6,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-216_w=mmmmscalar(id=6,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: ; tuple.ipv4.dport = tuples.five.dport; @ tproxy.c:903
3月 26 10:53:23 dae[621]: 300: (69) r2 = *(u16 *)(r10 -142)     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 301: (6b) *(u16 *)(r1 +10) = r2       ; R1_w=fp-216 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-208_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuple.ipv4.sport = tuples.five.sport; @ tproxy.c:904
3月 26 10:53:23 dae[621]: 302: (69) r2 = *(u16 *)(r10 -144)     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 303: (6b) *(u16 *)(r1 +8) = r2        ; R1_w=fp-216 R2_w=scalar(id=7,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-208_w=mmmmmmscalar(id=7,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
3月 26 10:53:23 dae[621]: 304: (b7) r3 = 12                     ; R3_w=12
3月 26 10:53:23 dae[621]: ; if (l4proto == IPPROTO_TCP) { @ tproxy.c:914
3月 26 10:53:23 dae[621]: 305: (55) if r8 != 0x6 goto pc+1      ; R8=6
3月 26 10:53:23 dae[621]: 306: (05) goto pc+43
3月 26 10:53:23 dae[621]: 350: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; if (tcph.syn && !tcph.ack) { @ tproxy.c:916
3月 26 10:53:23 dae[621]: 351: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 352: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 353: (57) r1 &= 512                   ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 354: (15) if r1 == 0x0 goto pc+5      ; R1_w=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 355: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 356: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 357: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 358: (57) r1 &= 4096                  ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 359: (15) if r1 == 0x0 goto pc+13     ; R1_w=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 360: (bf) r2 = r10                    ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; sk = bpf_skc_lookup_tcp(skb, &tuple, tuple_size, BPF_F_CURRENT_NETNS, 0); @ tproxy.c:920
3月 26 10:53:23 dae[621]: 361: (07) r2 += -216                  ; R2_w=fp-216
3月 26 10:53:23 dae[621]: 362: (bf) r1 = r6                     ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 363: (b7) r4 = -1                     ; R4_w=-1
3月 26 10:53:23 dae[621]: 364: (b7) r5 = 0                      ; R5_w=0
3月 26 10:53:23 dae[621]: 365: (85) call bpf_skc_lookup_tcp#99          ; R0_w=sock_common_or_null(id=9,ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 366: (bf) r8 = r0                     ; R0_w=sock_common_or_null(id=9,ref_obj_id=9) R8_w=sock_common_or_null(id=9,ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: ; if (sk) { @ tproxy.c:921
3月 26 10:53:23 dae[621]: 367: (15) if r8 == 0x0 goto pc+5      ; R8_w=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 368: (b7) r9 = 1                      ; R9_w=1 refs=9
3月 26 10:53:23 dae[621]: ; if (sk->state != BPF_TCP_LISTEN) { @ tproxy.c:922
3月 26 10:53:23 dae[621]: 369: (61) r1 = *(u32 *)(r8 +72)       ; R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R8=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 370: (55) if r1 != 0xa goto pc+136    ; R1=10 refs=9
3月 26 10:53:23 dae[621]: ; bpf_sk_release(sk); @ tproxy.c:926
3月 26 10:53:23 dae[621]: 371: (bf) r1 = r8                     ; R1_w=sock_common(ref_obj_id=9) R8=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 systemd[1]: dae.service: Main process exited, code=exited, status=1/FAILURE
3月 26 10:53:23 dae[621]: 372: (85) call bpf_sk_release#86      ; R0_w=scalar()
3月 26 10:53:23 dae[621]: 373: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 374: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: ; __builtin_memset(flag, 0, sizeof(flag)); @ tproxy.c:932
3月 26 10:53:23 dae[621]: 375: (63) *(u32 *)(r10 -220) = r7     ; R7=0 R10=fp0 fp-224=0000????
3月 26 10:53:23 dae[621]: 376: (63) *(u32 *)(r10 -224) = r7     ; R7=0 R10=fp0 fp-224=00000
3月 26 10:53:23 dae[621]: 377: (63) *(u32 *)(r10 -228) = r7     ; R7=0 R10=fp0 fp-232=0000????
3月 26 10:53:23 dae[621]: 378: (63) *(u32 *)(r10 -232) = r7     ; R7=0 R10=fp0 fp-232=00000
3月 26 10:53:23 dae[621]: 379: (63) *(u32 *)(r10 -236) = r7     ; R7=0 R10=fp0 fp-240=0000????
3月 26 10:53:23 dae[621]: 380: (63) *(u32 *)(r10 -240) = r7     ; R7=0 R10=fp0 fp-240=00000
3月 26 10:53:23 dae[621]: ; if (!(tcph.syn && !tcph.ack)) { @ tproxy.c:934
3月 26 10:53:23 dae[621]: 381: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 382: (57) r1 &= 512                   ; R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 383: (15) if r1 == 0x0 goto pc-359    ; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 384: (bf) r2 = r10                    ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 385: (07) r2 += -112                  ; R2_w=fp-112
3月 26 10:53:23 dae[621]: 386: (b7) r1 = 1                      ; R1_w=1
3月 26 10:53:23 dae[621]: 387: (69) r3 = *(u16 *)(r2 +12)       ; R2_w=fp-112 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 388: (57) r3 &= 4096                  ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 389: (55) if r3 != 0x0 goto pc-365    ; R3_w=0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 390: (63) *(u32 *)(r10 -248) = r1     ; R1_w=1 R10=fp0 fp-248=????1
3月 26 10:53:23 dae[621]: 391: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 392: (07) r1 += -16                   ; R1_w=fp-16
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[0] << 8) + (ethh.h_source[1])), @ tproxy.c:954
3月 26 10:53:23 dae[621]: 393: (71) r4 = *(u8 *)(r1 +7)         ; R1_w=fp-16 R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-16=mmmmmmmm
3月 26 10:53:23 systemd[1]: dae.service: Failed with result 'exit-code'.
3月 26 10:53:23 dae[621]: 394: (71) r3 = *(u8 *)(r1 +6)         ; R1_w=fp-16 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-16=mmmmmmmm
3月 26 10:53:23 dae[621]: 395: (67) r3 <<= 8                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00))
3月 26 10:53:23 dae[621]: 396: (4f) r3 |= r4                    ; R3_w=scalar() R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:945
3月 26 10:53:23 dae[621]: 397: (61) r4 = *(u32 *)(r6 +16)       ; R4_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 398: (b7) r7 = 2                      ; R7_w=2
3月 26 10:53:23 dae[621]: 399: (b7) r5 = 1                      ; R5=1
3月 26 10:53:23 dae[621]: 400: (15) if r4 == 0x8 goto pc+1      ; R4=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 401: (b7) r5 = 2                      ; R5_w=2
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 402: (71) r0 = *(u8 *)(r1 +8)         ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R1=fp-16 fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 403: (67) r0 <<= 24                   ; R0_w=scalar(smin=0,smax=umax=umax32=0xff000000,smax32=0x7f000000,var_off=(0x0; 0xff000000))
3月 26 10:53:23 dae[621]: 404: (71) r4 = *(u8 *)(r1 +9)         ; R1=fp-16 R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 405: (67) r4 <<= 16                   ; R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff0000,var_off=(0x0; 0xff0000))
3月 26 10:53:23 dae[621]: 406: (4f) r4 |= r0                    ; R0_w=scalar(smin=0,smax=umax=umax32=0xff000000,smax32=0x7f000000,var_off=(0x0; 0xff000000)) R4_w=scalar()
3月 26 10:53:23 dae[621]: ; flag[6] = tuples.dscp; @ tproxy.c:950
3月 26 10:53:23 dae[621]: 407: (71) r0 = *(u8 *)(r10 -136)      ; R0_w=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: 408: (63) *(u32 *)(r10 -224) = r0     ; R0_w=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-224=0000scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 systemd[1]: Failed to start dae Service.
3月 26 10:53:23 dae[621]: 409: (63) *(u32 *)(r10 -244) = r5     ; R5_w=2 R10=fp0 fp-248=mmmmmmmm
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[0] << 8) + (ethh.h_source[1])), @ tproxy.c:954
3月 26 10:53:23 dae[621]: 410: (dc) r3 = be32 r3                ; R3_w=scalar()
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 411: (63) *(u32 *)(r10 -256) = r3     ; R3_w=scalar() R10=fp0 fp-256=????scalar()
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 412: (71) r3 = *(u8 *)(r1 +10)        ; R1=fp-16 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 413: (67) r3 <<= 8                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00))
3月 26 10:53:23 dae[621]: 414: (4f) r4 |= r3                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00)) R4_w=scalar()
3月 26 10:53:23 dae[621]: 415: (71) r1 = *(u8 *)(r1 +11)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 416: (4f) r4 |= r1                    ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R4_w=scalar()
3月 26 10:53:23 dae[621]: 417: (b7) r8 = 0                      ; R8_w=0
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 418: (63) *(u32 *)(r10 -260) = r8     ; R8_w=0 R10=fp0 fp-264=0000????
3月 26 10:53:23 dae[621]: 419: (63) *(u32 *)(r10 -264) = r8     ; R8_w=0 R10=fp0 fp-264=00000
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 420: (dc) r4 = be32 r4                ; R4_w=scalar()
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 421: (63) *(u32 *)(r10 -252) = r4     ; R4_w=scalar() R10=fp0 fp-256=mmmmmmmm
3月 26 10:53:23 dae[621]: 422: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 423: (07) r1 += -248                  ; R1_w=fp-248
3月 26 10:53:23 dae[621]: 424: (bf) r3 = r10                    ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 425: (07) r3 += -176                  ; R3_w=fp-176
3月 26 10:53:23 dae[621]: ; tuples.five.dip.u6_addr32, mac)) < 0) { @ tproxy.c:960
3月 26 10:53:23 dae[621]: 426: (bf) r4 = r10                    ; R4_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 427: (07) r4 += -160                  ; R4_w=fp-160
3月 26 10:53:23 dae[621]: 428: (bf) r5 = r10                    ; R5_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 429: (07) r5 += -264                  ; R5_w=fp-264
3月 26 10:53:23 dae[621]: ; if ((s64_ret = route(flag, l4hdr, tuples.five.sip.u6_addr32, @ tproxy.c:959
3月 26 10:53:23 dae[621]: 430: (85) call pc+252
3月 26 10:53:23 dae[621]: caller:
3月 26 10:53:23 dae[621]:  R6=ctx() R7=2 R8_w=0 R9=1 R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0 fp-88=0 fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=0 fp-128=mmmm???? fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) fp-144=mmmmmmmm fp-152=mmmmmmmm fp-160=0 fp-168=mmmmmmmm fp-176=0 fp-184=????0 fp-192=0 fp-200=0 fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=0000scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) fp-232=00000 fp-240=00000 fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=00000 fp-320=6
3月 26 10:53:23 dae[621]: callee:
3月 26 10:53:23 dae[621]:  frame1: R1_w=fp[0]-248 R2=fp[0]-112 R3_w=fp[0]-176 R4_w=fp[0]-160 R5_w=fp[0]-264 R10=fp0
3月 26 10:53:23 dae[621]: 683: frame1: R1=fp[0]-248 R2=fp[0]-112 R3=fp[0]-176 R4=fp[0]-160 R5=fp[0]-264 R10=fp0
3月 26 10:53:23 dae[621]: ; route(const __u32 flag[8], const void *l4hdr, const __be32 saddr[4], @ tproxy.c:558
3月 26 10:53:23 dae[621]: 683: (7b) *(u64 *)(r10 -56) = r5      ; frame1: R5=fp[0]-264 R10=fp0 fp-56_w=fp[0]-264
3月 26 10:53:23 dae[621]: 684: (7b) *(u64 *)(r10 -48) = r4      ; frame1: R4=fp[0]-160 R10=fp0 fp-48_w=fp[0]-160
3月 26 10:53:23 dae[621]: 685: (bf) r7 = r3                     ; frame1: R3=fp[0]-176 R7_w=fp[0]-176
3月 26 10:53:23 dae[621]: 686: (bf) r6 = r2                     ; frame1: R2=fp[0]-112 R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: 687: (bf) r8 = r1                     ; frame1: R1=fp[0]-248 R8_w=fp[0]-248
3月 26 10:53:23 dae[621]: 688: (b7) r1 = 5                      ; frame1: R1_w=5
3月 26 10:53:23 dae[621]: ; __u32 key = MatchType_L4Proto; @ tproxy.c:568
3月 26 10:53:23 dae[621]: 689: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=5 R10=fp0 fp-32=mmmm????
3月 26 10:53:23 dae[621]: 690: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 691: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:573
3月 26 10:53:23 dae[621]: 692: (18) r1 = 0xffff8ec8103e8c00     ; frame1: R1_w=map_ptr(map=l4proto_ipversi,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 694: (bf) r3 = r8                     ; frame1: R3_w=fp[0]-248 R8_w=fp[0]-248
3月 26 10:53:23 dae[621]: 695: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 696: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 697: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=10) R9_w=scalar(id=10)
3月 26 10:53:23 dae[621]: 698: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 699: (15) if r9 == 0x0 goto pc+2 702: frame1: R0=scalar(id=10) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmm???? fp-48=fp[0]-160 fp-56=fp[0]-264
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 702: (b7) r1 = 6                      ; frame1: R1_w=6
3月 26 10:53:23 dae[621]: ; key = MatchType_IpVersion; @ tproxy.c:577
3月 26 10:53:23 dae[621]: 703: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=6 R10=fp0 fp-32=mmmm????
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:578
3月 26 10:53:23 dae[621]: 704: (bf) r3 = r8                     ; frame1: R3_w=fp[0]-248 R8=fp[0]-248
3月 26 10:53:23 dae[621]: 705: (07) r3 += 4                     ; frame1: R3_w=fp[0]-244
3月 26 10:53:23 dae[621]: 706: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 707: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:578
3月 26 10:53:23 dae[621]: 708: (18) r1 = 0xffff8ec8103e8c00     ; frame1: R1_w=map_ptr(map=l4proto_ipversi,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 710: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 711: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 712: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=22) R9_w=scalar(id=22)
3月 26 10:53:23 dae[621]: 713: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 714: (15) if r9 == 0x0 goto pc+1 716: frame1: R0=scalar(id=22) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmm???? fp-48=fp[0]-160 fp-56=fp[0]-264
3月 26 10:53:23 dae[621]: ; if (_l4proto_type == L4ProtoType_TCP) { @ tproxy.c:584
3月 26 10:53:23 dae[621]: 716: (61) r1 = *(u32 *)(r8 +0)        ; frame1: R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R8=fp[0]-248
3月 26 10:53:23 dae[621]: 717: (7b) *(u64 *)(r10 -64) = r8      ; frame1: R8=fp[0]-248 R10=fp0 fp-64_w=fp[0]-248
3月 26 10:53:23 dae[621]: 718: (55) if r1 != 0x1 goto pc+5      ; frame1: R1_w=1
3月 26 10:53:23 dae[621]: ; h_dport = bpf_ntohs(((struct tcphdr *)l4hdr)->dest); @ tproxy.c:585
3月 26 10:53:23 dae[621]: 719: (69) r1 = *(u16 *)(r6 +2)        ; frame1: R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R6=fp[0]-112
3月 26 10:53:23 dae[621]: 720: (dc) r1 = be16 r1                ; frame1: R1_w=scalar()
3月 26 10:53:23 dae[621]: 721: (6b) *(u16 *)(r10 -30) = r1      ; frame1: R1_w=scalar() R10=fp0 fp-32=mmmmmm??
3月 26 10:53:23 dae[621]: 722: (b7) r1 = 0                      ; frame1: R1_w=0
3月 26 10:53:23 dae[621]: 723: (05) goto pc+4
3月 26 10:53:23 dae[621]: ; h_dport = bpf_ntohs(((struct udphdr *)l4hdr)->dest); @ tproxy.c:588
3月 26 10:53:23 dae[621]: 728: (0f) r6 += r1                    ; frame1: R1=0 R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 729: (69) r1 = *(u16 *)(r6 +0)        ; frame1: R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: 730: (b7) r2 = 4                      ; frame1: R2_w=4
3月 26 10:53:23 dae[621]: ; key = MatchType_SourcePort; @ tproxy.c:592
3月 26 10:53:23 dae[621]: 731: (63) *(u32 *)(r10 -28) = r2      ; frame1: R2_w=4 R10=fp0 fp-32=mmmmmm??
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 732: (dc) r1 = be16 r1                ; frame1: R1_w=scalar()
3月 26 10:53:23 dae[621]: 733: (6b) *(u16 *)(r10 -32) = r1      ; frame1: R1_w=scalar() R10=fp0 fp-32=mmmmmmscalar()
3月 26 10:53:23 dae[621]: 734: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 735: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: 736: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 737: (07) r3 += -32                   ; frame1: R3_w=fp-32
3月 26 10:53:23 dae[621]: ; if (unlikely( @ tproxy.c:593
3月 26 10:53:23 dae[621]: 738: (18) r1 = 0xffff8ec8103ec400     ; frame1: R1_w=map_ptr(map=h_port_map,ks=4,vs=2)
3月 26 10:53:23 dae[621]: 740: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 741: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 742: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=23) R9_w=scalar(id=23)
3月 26 10:53:23 dae[621]: 743: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 744: (15) if r9 == 0x0 goto pc+1 746: frame1: R0=scalar(id=23) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmmmmmm fp-48=fp[0]-160 fp-56=fp[0]-264 fp-64=fp[0]-248
3月 26 10:53:23 dae[621]: 746: (b7) r1 = 3                      ; frame1: R1_w=3
3月 26 10:53:23 dae[621]: ; key = MatchType_Port; @ tproxy.c:597
3月 26 10:53:23 dae[621]: 747: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=3 R10=fp0 fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: 748: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 749: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: 750: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 751: (07) r3 += -30                   ; frame1: R3_w=fp-30
3月 26 10:53:23 dae[621]: ; if (unlikely( @ tproxy.c:598
3月 26 10:53:23 dae[621]: 752: (18) r1 = 0xffff8ec8103ec400     ; frame1: R1_w=map_ptr(map=h_port_map,ks=4,vs=2)
3月 26 10:53:23 dae[621]: 754: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 755: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 756: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=24) R9_w=scalar(id=24)
3月 26 10:53:23 dae[621]: 757: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 758: (15) if r9 == 0x0 goto pc+1 760: frame1: R0=scalar(id=24) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmmmmmm fp-48=fp[0]-160 fp-56=fp[0]-264 fp-64=fp[0]-248
3月 26 10:53:23 dae[621]: 760: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; lpm_key_instance.trie_key.prefixlen = IPV6_BYTE_LENGTH * 8; @ tproxy.c:603
3月 26 10:53:23 dae[621]: 761: (07) r3 += -24                   ; frame1: R3_w=fp-24
3月 26 10:53:23 dae[621]: 762: (b7) r1 = 128                    ; frame1: R1_w=128
3月 26 10:53:23 dae[621]: 763: (85) call unknown#195896080
3月 26 10:53:23 dae[621]: invalid func unknown#195896080
3月 26 10:53:23 dae[621]: processed 423 insns (limit 1000000) max_states_per_insn 1 total_states 29 peak_states 29 mark_read 11"

@jschwinger233
Copy link
Member

可能是 CO-RE 的问题，你能导出一份 vmlinux 吗？

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

或者安装 pahole 运行 pahole -C bpf_lpm_trie_key 会更简单

@sakarie9
Copy link
Author

vmlinux.h.tar.gz

但是 pahole: type 'bpf_lpm_trie_key' not found

@jschwinger233
Copy link
Member

~~ 那就对了，~~ 没有 bpf_lpm_trie_key 这个类型，无法做 CO-RE，所以挂了。

@jschwinger233
Copy link
Member

看到你的 vmlinux 里定义的是 bpf_lpm_trie_key_u8 而不是 bpf_lpm_trie_key

@sakarie9
Copy link
Author

https://lore.kernel.org/bpf/202402221046.020C94D@keescook/T/

大概是这里改动的

@jschwinger233
Copy link
Member

在 vmlinux 里做兼容吧，大概是这样的：

if (LINUX_KERNEL_VERSION > KERNEL_VERSION(6, 9, 0)) {
    /* bpf_lpm_trie_key_u8 */
} else {
    /* bpf_lpm_trie_key */ 
}

我看看有没有更好的办法

@jschwinger233 jschwinger233 mentioned this issue Mar 26, 2024
Merged
3 tasks
@jschwinger233
Copy link
Member

bpf 和内核社区还完全不知道这个问题，恭喜成为全球第一个报告此 bug 的人类。

请试试 #483 这个 PR，可以直接从 PR build 里下载二进制。

@sakarie9
Copy link
Author

正常了！

dae version unstable-20240326.pr-483.r1.9aafbb

3月 26 18:19:42 systemd[1]: Starting dae Service...
3月 26 18:19:42 dae[17759]: level=info msg="Include config files: [/etc/dae/config.dae]"
3月 26 18:19:42 dae[17759]: level=info msg="Loading eBPF programs and maps into the kernel..."
3月 26 18:19:42 dae[17759]: level=info msg="The loading process takes about 120MB free memory, which will be released after loading. Insufficient memory will cause loading failure."
3月 26 18:19:43 dae[17759]: level=info msg="Loaded eBPF programs and maps"
3月 26 18:19:43 dae[17759]: level=info msg="Bind to WAN: enp34s0"
3月 26 18:19:43 dae[17759]: level=info msg="Group "clash" node list:"
3月 26 18:19:43 dae[17759]: level=info msg="        clash"
3月 26 18:19:43 dae[17759]: level=info msg="Routing match set len: 14/64"
3月 26 18:19:43 systemd[1]: Started dae Service.

@jschwinger233 jschwinger233 self-assigned this Mar 26, 2024
@jschwinger233 jschwinger233 mentioned this issue Nov 8, 2024
Closed
Integral-Tech added a commit to Integral-Tech/dae that referenced this issue Nov 9, 2024
@Integral-Tech
ci(workflows/kernel-test): add latest mainline kernel
3756458 
Sometimes upstream API changes can trigger CO-RE failure in mainline
kernel, such as issue daeuniverse#482. Therefore, it is recommended to add latest
mainline kernel to tests.
@Integral-Tech Integral-Tech mentioned this issue Nov 9, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jschwinger233 jschwinger233

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Fix bpf CO-RE issue on 6.9 daeuniverse/dae
3 participants
@jschwinger233 @sakarie9 @mzz2017