"The agent could not be able to start eBPF programs" on OCP 4.10 on top of IBM Rocks

[andresmareca-ibm]

We (me and @eranra) have managed to start all the necessary pods to be able to monitor the network following the following procedure.

1. Clone newtork-observability-operator
2. run make deploy ocp-deploy
3. run oc apply -f config/samples/flows_v1alpha1_flowcollector.yaml
4. Set ebpf-agent pods as privileged as is described here: Pod crash loop back-off when deploying on OCP 4.10 on top of IBM Rocks #34

Now onnetobserv-ebpf-agent-* pods I get the following error message:

time="2022-06-23T11:04:51Z" level=info msg="starting NetObserv eBPF Agent"
time="2022-06-23T11:04:51Z" level=info msg="initializing Flows agent" component=agent.Flows
time="2022-06-23T11:04:51Z" level=info msg="push CTRL+C or send SIGTERM to interrupt execution"
time="2022-06-23T11:04:51Z" level=info msg="starting Flows agent" component=agent.Flows
time="2022-06-23T11:04:51Z" level=warning msg="can't remove mem lock. The agent could not be able to start eBPF programs" component=systemSetup error="unexpected error detecting memory cgroup accounting: permission denied"
time="2022-06-23T11:04:51Z" level=info msg="Flows agent successfully started" component=agent.Flows
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=eth0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=eth0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=eth1
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=eth1
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=vethlocal
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=vethlocal
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=tunl0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=tunl0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib83c121d54f
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib83c121d54f
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali4ffc7d0c2cd
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali4ffc7d0c2cd
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia361f7a2ef0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia361f7a2ef0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali2d403d4ded4
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali2d403d4ded4
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali08aea38f7a5
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali08aea38f7a5
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali5132e419805
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali5132e419805
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib5e3a5f56f6
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib5e3a5f56f6
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif6df68b6a97
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif6df68b6a97
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali24dc76da138
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali24dc76da138
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali7ede1555ef2
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali7ede1555ef2
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib26e009b516
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib26e009b516
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali9eea980de3d
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali9eea980de3d
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali4458a90e5f5
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali4458a90e5f5
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali209bf37fc41
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali209bf37fc41
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali988f8eb4584
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali988f8eb4584
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali62d9ccb9b76
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali62d9ccb9b76
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali724c631a936
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali724c631a936
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=califfa03e79339
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=califfa03e79339
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali9c18e337f00
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali9c18e337f00
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calic38c45648fd
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calic38c45648fd
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali435d7b6f56c
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali435d7b6f56c
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif2d31cc4040
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif2d31cc4040
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif0d73f1499c
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif0d73f1499c
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia32c3fcf987
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia32c3fcf987
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali0a8626f2f08
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali0a8626f2f08
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali090d631fef7
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali090d631fef7
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali49f1199c7c6
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali49f1199c7c6
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calice4e6974a40
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calice4e6974a40
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali7c05cd50d45
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali7c05cd50d45
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali57e83446543
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali57e83446543
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calibc77efe6d8a
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calibc77efe6d8a
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calicf3cf0bb3bb
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calicf3cf0bb3bb
time="2022-06-23T11:04:55Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia8a8ea72ccb
time="2022-06-23T11:04:55Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia8a8ea72ccb

If I check the kernel version it shows me the following:

sh-4.4# chroot /host
sh-4.2# uname -a
Linux kube-caop3qnf0asm47p248i0-testocp410-default-00000121.iks.ibm 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed Apr 27 20:34:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Has it happened to any of you?

[eranra]

@mariomac @praveingk please your help ^^^

[mariomac]

@andresmareca-ibm This kernel version is 3.10, which seems way too old to run eBPF according to: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

We are supporting version 4.18+, which is the kernel version supported by RHEL 8, although some earlier versions could also work, but 3.10 seems too old even for basic eBPF support, according to the documentation.

[eranra]

@elevran FYI kernel version is 3.10 so whatever we talked about with regards to alternatives to eBPF using he kernel is not relevant for such an old kernel :-(