Change map convention (prefix to name instead of suffix to section) +…

… Update to latest Cilium (#103)

* example/capable: Fix Readme. print->counter

* example/tcpconnlat: Fix Readme. print->counter

* loader: Use prefix instead of map section name

This removes the issue with having non standard section names for
the maps

* pkg/cli: Generate maps with prefixes in its name instead of a suffix to its sectionname

* docs: Update regarding map naming scheme instead of secion suffix

* Update go to 1.18

* pkg/loader,decoder: Update to latest cilium/ebpf

The old way of saving the (non public) BTF struct is not needed anymore

Co-authored-by: Alban Crequy <albancrequy@linux.microsoft.com>

---------

Co-authored-by: Alban Crequy <albancrequy@linux.microsoft.com>

.github/workflows/pull_request.yaml

@@ -16,10 +16,10 @@ jobs:
       - uses: actions/checkout@v2
       - run: |
           git fetch --prune --unshallow
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         uses: actions/setup-go@v1
         with:
-          go-version: 1.17.2
+          go-version: 1.18.10
       - uses: actions/cache@v1
         with:
           path: ~/go/pkg/mod
@@ -45,10 +45,10 @@ jobs:
       - uses: actions/checkout@v2
       - run: |
           git fetch --prune --unshallow
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         uses: actions/setup-go@v1
         with:
-          go-version: 1.17.2
+          go-version: 1.18.10
       - uses: actions/cache@v1
         with:
           path: ~/go/pkg/mod

.github/workflows/release.yaml

@@ -20,10 +20,10 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v2
       - run: git fetch --prune --unshallow
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         uses: actions/setup-go@v1
         with:
-          go-version: 1.17.6
+          go-version: 1.18.10
       - name: Setup Cache
         uses: actions/cache@v1
         with:
@@ -74,10 +74,10 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@v2
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         uses: actions/setup-go@v1
         with:
-          go-version: 1.17.6
+          go-version: 1.18.10
       - name: Setup Cache
         uses: actions/cache@v1
         with:
@@ -105,10 +105,10 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v2
 
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         uses: actions/setup-go@v1
         with:
-          go-version: 1.17.6
+          go-version: 1.18.10
       - name: Setup Cache
         uses: actions/cache@v1
         with:

docs/concepts.md

@@ -49,18 +49,18 @@ struct {
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
 	__uint(max_entries, 1 << 24);
 	__type(value, struct event_t);
-} events SEC(".maps.print");
+} print_events SEC(".maps");
 ```
 
-The other aspect of the above program worth noting is the section name: `SEC(".maps.print")`. Specifically the `.print` suffix. For those familiar with `BPF` programs this will look new. Please see the [output formats](#Output-Formats) section below for more info. The `RingBuffer` map type supports the `.print` and `.counter` keywords.
+The other aspect of the above program worth noting is its name: `print_events SEC(".maps")`. Specifically the `print_` prefix. Please see the [output formats](#Output-Formats) section below for more info. The `RingBuffer` map type supports the `print_` and `counter_` prefix.
 
 The final thing worth noting about the `RingBuffer` is it's event based nature. Each object is handled only once, and then never read from again. This differs from the `HashMap`, which will be discussed in greater detail below.
 
 #### HashMap
 
 Like `RingBuffer` above, `HashMap` is a generic map type to store data, with some key differences. The `HashMap` does not function as a queue, but rather as a traditional map, with both keys and values, which retains it's data until manually removed.
 
-In addition, `HashMap` supports section keywords to enable special [output formats](#Output-Formats). The valid keywords for this type of map are: `.print`, `.counter`, and `.gauge`.
+In addition, `HashMap` supports section keywords to enable special [output formats](#Output-Formats). The valid prefixes for this type of map are: `print_`, `counter_`, and `gauge_`.
 
 
 ### Programs
@@ -74,7 +74,7 @@ Part of what makes `bee` so special, as mentioned above, is that it allows us to
 
 These special conventions and keywords come in the form of additional kernel code additions, some in section names, and some to the code itself. Let's begin with the section names.
 
-Maps in `BPF` programs are defined using the `SEC(".maps")` keyword. When running using the `bee` runner, extra suffixes can be added to describe how this data should be handled. These can be roughly broken down into 2 behaviors, metrics and logging. Metrics turns the data into prometheus metrics which can be consumed by any standard prometheus deployments. And logging which emits structured json logs with the provided data, and can be consumed by any structured logging applications.
+Maps in `BPF` programs are defined using the `SEC(".maps")` keyword. When running using the `bee` runner, extra prefixes to its name can be added to describe how this data should be handled. These can be roughly broken down into 2 behaviors, metrics and logging. Metrics turns the data into prometheus metrics which can be consumed by any standard prometheus deployments. And logging which emits structured json logs with the provided data, and can be consumed by any structured logging applications.
 
 The second convention we have added is a set of `typedef`s which describe to our runner how the underlying type is meant to be processed after it leaves the kernel. These are stored in a file called `solo_types.h` and are made available automatically when building with `bee`. Some examples include:
 ```C
@@ -118,10 +118,10 @@ struct {
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
 	__uint(max_entries, 1 << 24);
 	__type(value, struct dimensions_t);
-} events_ring SEC(".maps.counter");
+} counter_events_ring SEC(".maps");
 ``` 
 We can see that the `struct dimensions_t` type is being passed into the `RingBuffer`.
-So as TCP connections are established, the source and destination IP of these connections will be sent as events into this `RingBuffer`. Also note the `.maps.counter` section name.
+So as TCP connections are established, the source and destination IP of these connections will be sent as events into this `RingBuffer`. Also note the `counter_` prefix to its name.
 This tells `bee` to "watch" this map and log the data (in addition to emitting counter metrics, which will explore more in a later section).
 
 When running the program, as new TCP connections happen (by e.g. making a `curl 1.1.1.1` request in a separate terminal) we can see the TUI log the data:
@@ -141,7 +141,7 @@ struct {
 	__uint(max_entries, 8192);
 	__type(key, struct dimensions_t);
 	__type(value, u64);
-} events_hash SEC(".maps.counter");
+} counter_events_hash SEC(".maps");
 ```
 
 As above, we can see that the `struct dimensions_t` type is being used, this time as the `key` type for our `HashMap`.
@@ -167,25 +167,25 @@ An example of the both the `RingBuffer` counter and `HashMap` counter exist in t
 
 After starting the program, and curling httpbin a few times we can, we can get the metrics from `curl localhost:9091/metrics | grep events`
 ```
-# HELP ebpf_solo_io_events_hash 
-# TYPE ebpf_solo_io_events_hash counter
-ebpf_solo_io_events_hash{daddr="18.232.227.86",saddr="10.128.0.79"} 9
-ebpf_solo_io_events_hash{daddr="3.216.167.140",saddr="10.128.0.79"} 5
-# HELP ebpf_solo_io_events_ring 
-# TYPE ebpf_solo_io_events_ring counter
-ebpf_solo_io_events_ring{daddr="18.232.227.86",saddr="10.128.0.79"} 9
-ebpf_solo_io_events_ring{daddr="3.216.167.140",saddr="10.128.0.79"} 5
+# HELP ebpf_solo_io_counter_events_hash 
+# TYPE ebpf_solo_io_counter_events_hash counter
+ebpf_solo_io_counter_events_hash{daddr="18.232.227.86",saddr="10.128.0.79"} 9
+ebpf_solo_io_counter_events_hash{daddr="3.216.167.140",saddr="10.128.0.79"} 5
+# HELP ebpf_solo_io_counter_events_ring 
+# TYPE ebpf_solo_io_counter_events_ring counter
+ebpf_solo_io_counter_events_ring{daddr="18.232.227.86",saddr="10.128.0.79"} 9
+ebpf_solo_io_counter_events_ring{daddr="3.216.167.140",saddr="10.128.0.79"} 5
 ```
 
 As we can see the number of connections are being tracked both from our `HashMap` and `RingBuffer` implementation.
 
 #### Gauge 
 
 Gauges are used to track numeric values that can change over time.
-BumbleBee supports automatically exporting gauge style metrics for both `RingBuffer` and `HashMap` type maps as long as your map is correctly defined with a section name with a `.gauge` suffix.
+BumbleBee supports automatically exporting gauge style metrics for both `RingBuffer` and `HashMap` type maps as long as your map is correctly defined with a name with a `gauge_` prefix.
 
 An example of a gauge is the number of active connections to a given host.
-The [/examples/activeconn/activeconn.c](/examples/activeconn/activeconn.c) file contains an implementation of active connection tracking by using a `HashMap` type map with a `.gauge` output type.
+The [/examples/activeconn/activeconn.c](/examples/activeconn/activeconn.c) file contains an implementation of active connection tracking by using a `HashMap` type map with a `gauge_` output type.
 
 Let's take a closer look at the `struct` which defines the map that will contain the connection counts.
 ```c
@@ -194,12 +194,12 @@ struct {
 	__uint(max_entries, 8192);
 	__type(key, struct dimensions_t);
 	__type(value, u64);
-} sockets_ext SEC(".maps.gauge");
+} gauge_sockets_ext SEC(".maps");
 ```
 
 This defines a `HashMap` containing integer values for connection counts which are keyed by `struct dimensions_t` (which we explored in the [HashMap](#hashmap-1) section).
 In other words, this means that each source and destination address pair will point to an integer representing the current number of active connections.
 
-The exporting of metrics is automatically handled thanks to the section name of `.maps.gauge`.
+The exporting of metrics is automatically handled thanks to the name prefix of `gauge_`.
 This tells the `bee` runner to export gauge metrics of the current value for each entry in the `HashMap` map each time the value of the map is polled.
 Alternatively, if we were using a `RingBuffer` with gauge output, when each entry is processed by the `bee` runner, the gauge value will be updated accordingly.

docs/getting_started.md

@@ -79,7 +79,7 @@ struct {
 	__uint(max_entries, 1 << 24);
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
 	__type(value, struct event_t);
-} events SEC(".maps.print");
+} print_events SEC(".maps");
 
 
 SEC("kprobe/tcp_v4_connect")
@@ -89,7 +89,7 @@ int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
 	struct event_t *event;
 
 	// Reserve a spot in the ringbuffer for our event
-	event = bpf_ringbuf_reserve(&events, sizeof(struct event_t), 0);
+	event = bpf_ringbuf_reserve(&print_events, sizeof(struct event_t), 0);
 	if (!event) {
 		return 0;
 	}
@@ -121,13 +121,13 @@ struct {
 	__uint(max_entries, 1 << 24);
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
 	__type(value, struct event_t);
-} events SEC(".maps.print");
+} print_events SEC(".maps");
 ```
 
 This defines a BPF map of type ring-buffer. A ring-buffer map is commonly used to stream events from 
 kernel space to user space. The kernel eBPF probe writes to the ring buffer, and a user-mode program can asynchronously read events from the buffer.
 
-Note the section the map is in: `.maps.print` - this has special meaning in `bee` - it instructs it to display this map as a stream of events (think logs and not metrics).
+Note the map name has the prefix `print_` - this has special meaning in `bee` - it instructs it to display this map as a stream of events (think logs and not metrics).
 
 Note also that unlike libbpf ring buffer map, this one has a `__type` defined. This allows `bee` to automatically output the events written to the map. 
 
@@ -171,7 +171,7 @@ struct {
 	__uint(max_entries, 8192);
 	__type(key, struct dimensions_t);
 	__type(value, u64);
-} connection_count SEC(".maps.counter");
+} counter_connection_count SEC(".maps");
 ```
 
 Note the `ipv4_addr` type. This type is defined in `solo_types.h`. While it is simply defined to be a `u32`, this type definition is a hint to `bee` to format this field as an IPv4 address.
@@ -192,7 +192,7 @@ int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
 	daddr = BPF_CORE_READ(usin, sin_addr.s_addr);
 
 	// Reserve a spot in the ringbuffer for our event
-	event = bpf_ringbuf_reserve(&events, sizeof(struct event_t), 0);
+	event = bpf_ringbuf_reserve(&print_events, sizeof(struct event_t), 0);
 	if (!event) {
 		return 0;
 	}
@@ -204,15 +204,15 @@ int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
 
 	// increment the counter for this address
 	hash_key.daddr = daddr;
-	counterp = bpf_map_lookup_elem(&connection_count, &hash_key);
+	counterp = bpf_map_lookup_elem(&counter_connection_count, &hash_key);
 	if (counterp) {
 		__sync_fetch_and_add(counterp, 1);
 	} else {
 		// we may miss N events, where N is number of CPUs. We may want to 
 		// fix this for prod, by adding another lookup/update calls here.
 		// we skipped these for brevity
 		counter = 1;
-		bpf_map_update_elem(&connection_count, &hash_key, &counter, BPF_NOEXIST);
+		bpf_map_update_elem(&counter_connection_count, &hash_key, &counter, BPF_NOEXIST);
 	}
 
 	return 0;
@@ -248,7 +248,7 @@ struct {
 	__uint(max_entries, 8192);
 	__type(key, struct dimensions_t);
 	__type(value, u64);
-} connection_count SEC(".maps.counter");
+} counter_connection_count SEC(".maps");
 
 // This is the definition for the global map which both our
 // bpf program and user space program can access.
@@ -257,7 +257,7 @@ struct {
 	__uint(max_entries, 1 << 24);
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
 	__type(value, struct event_t);
-} events SEC(".maps.print");
+} print_events SEC(".maps");
 
 SEC("kprobe/tcp_v4_connect")
 int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
@@ -273,7 +273,7 @@ int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
 	daddr = BPF_CORE_READ(usin, sin_addr.s_addr);
 
 	// Reserve a spot in the ringbuffer for our event
-	event = bpf_ringbuf_reserve(&events, sizeof(struct event_t), 0);
+	event = bpf_ringbuf_reserve(&print_events, sizeof(struct event_t), 0);
 	if (!event) {
 		return 0;
 	}
@@ -285,15 +285,15 @@ int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
 
 	// increment the counter for this address
 	hash_key.daddr = daddr;
-	counterp = bpf_map_lookup_elem(&connection_count, &hash_key);
+	counterp = bpf_map_lookup_elem(&counter_connection_count, &hash_key);
 	if (counterp) {
 		__sync_fetch_and_add(counterp, 1);
 	} else {
 		// we may miss N events, where N is number of CPUs. We may want to 
 		// fix this for prod, by adding another lookup/update calls here.
 		// we skipped these for brevity
 		counter = 1;
-		bpf_map_update_elem(&connection_count, &hash_key, &counter, BPF_NOEXIST);
+		bpf_map_update_elem(&counter_connection_count, &hash_key, &counter, BPF_NOEXIST);
 	}
 
 	return 0;

examples/activeconn/activeconn.c

@@ -23,7 +23,7 @@ struct {
 	__uint(max_entries, 8192);
 	__type(key, struct dimensions_t);
 	__type(value, u64);
-} sockets_ext SEC(".maps.gauge");
+} gauge_sockets_ext SEC(".maps");
 
 static __always_inline int
 enter_tcp_connect(struct pt_regs *ctx, struct sock *sk)
@@ -65,7 +65,7 @@ record(struct pt_regs *ctx, int ret, int op)
 	key.saddr = saddr;
 	key.daddr = daddr;
 
-	valp = bpf_map_lookup_elem(&sockets_ext, &key);
+	valp = bpf_map_lookup_elem(&gauge_sockets_ext, &key);
 	if (!valp) {
 		bpf_printk("no entry for {saddr: %u, daddr: %u}", key.saddr, key.daddr);
 		val = 1;
@@ -74,7 +74,7 @@ record(struct pt_regs *ctx, int ret, int op)
 		bpf_printk("found existing value '%llu' for {saddr: %u, daddr: %u}", *valp, key.saddr, key.daddr);
 		val = *valp + op;
 	}
-	bpf_map_update_elem(&sockets_ext, &key, &val, 0);
+	bpf_map_update_elem(&gauge_sockets_ext, &key, &val, 0);
 	bpf_map_delete_elem(&sockets, &tid);
 	return 0;
 }

examples/capable/README.md

@@ -28,14 +28,14 @@ struct cap_event {
 For example, to filter for all the capability checks where the `task` is `ping`, you can use: 
 
 ```console
-bee run -f="events,task,ping" ghcr.io/solo-io/bumblebee/capable:$(bee version)
+bee run -f="print_events,task,ping" ghcr.io/solo-io/bumblebee/capable:$(bee version)
 ```
 
 # Prometheus integration
 
 Let's say, you want to visualize the rate of such syscalls in your Prometheus stack, or want to alert on certain syscalls.
 
-You can modify your `events` map to generate a `counter` from your cap_capable() calls:
+You can modify your `print_events` map to generate a `counter` from your cap_capable() calls:
 
 > Note: you can rename `events` to `cap_events` to illustrate the goal of the exposed events better.
 
@@ -44,7 +44,7 @@ struct {
         __uint(type, BPF_MAP_TYPE_RINGBUF);
         __uint(max_entries, 1 << 24);
         __type(value, struct cap_event);
-} events SEC(".maps.print");
+} counter_events SEC(".maps");
 ```
 
 You should consider removing high cardinality fields from your map to avoid overloading your Prometheus instance, e.g. `mntnsid`.

examples/capable/capable.c

@@ -30,7 +30,7 @@ struct {
         __uint(type, BPF_MAP_TYPE_RINGBUF);
         __uint(max_entries, 1 << 24);
         __type(value, struct cap_event);
-} events SEC(".maps.print");
+} print_events SEC(".maps");
 
 struct {
         __uint(type, BPF_MAP_TYPE_HASH);
@@ -63,7 +63,7 @@ int BPF_KPROBE(kprobe__cap_capable, const struct cred *cred, struct user_namespa
 
         struct cap_event *event;
 
-        event = bpf_ringbuf_reserve(&events, sizeof(struct cap_event), 0);
+        event = bpf_ringbuf_reserve(&print_events, sizeof(struct cap_event), 0);
         if (!event) {
                 return 0;
         }

examples/exitsnoop/README.md

@@ -30,31 +30,31 @@ struct event {
 For example, to filter for all the syscalls where the `comm` is `bash`, you can use: 
 
 ```console
-bee run -f="exits,comm,bash" ghcr.io/solo-io/bumblebee/exitsnoop:$(bee version)
+bee run -f="print_exits,comm,bash" ghcr.io/solo-io/bumblebee/exitsnoop:$(bee version)
 ```
 
 # Prometheus integration
 
 Let's say, you want to visualize the rate of such syscalls in your Prometheus stack, or want to alert on certain syscalls.
 
-You can modify your `exit` map to generate a `counter` from your exit() calls:
+You can modify your `print_exits` map to generate a `counter` from your exit() calls:
 
 ```c
 struct {
         __uint(type, BPF_MAP_TYPE_RINGBUF);
         __uint(max_entries, 1 << 24);
         __type(value, struct event);
-} exits SEC(".maps.counter");
+} counter_exits SEC(".maps");
 ```
 
 This will generate Prometheus metrics like this:
 
 ```console
-# HELP ebpf_solo_io_exits 
-# TYPE ebpf_solo_io_exits counter
-ebpf_solo_io_exits{comm="infocmp",exit_code="0",exit_time="2396600383928",pid="16234",ppid="16221",sig="0",start_time="2396599269958",tid="16234"} 1
-ebpf_solo_io_exits{comm="ip6tables",exit_code="0",exit_time="2402597588547",pid="16237",ppid="2115",sig="0",start_time="2402596148575",tid="16237"} 1
-ebpf_solo_io_exits{comm="iptables",exit_code="0",exit_time="2397474999464",pid="16235",ppid="2115",sig="0",start_time="2397473028592",tid="16235"} 1
+# HELP ebpf_solo_io_counter_exits
+# TYPE ebpf_solo_io_counter_exits counter
+ebpf_solo_io_counter_exits{comm="infocmp",exit_code="0",exit_time="2396600383928",pid="16234",ppid="16221",sig="0",start_time="2396599269958",tid="16234"} 1
+ebpf_solo_io_counter_exits{comm="ip6tables",exit_code="0",exit_time="2402597588547",pid="16237",ppid="2115",sig="0",start_time="2402596148575",tid="16237"} 1
+ebpf_solo_io_counter_exits{comm="iptables",exit_code="0",exit_time="2397474999464",pid="16235",ppid="2115",sig="0",start_time="2397473028592",tid="16235"} 1
 ```
 
 Note that some of the fields are not important in this usecase, and these can also overload your Prometheus instace, so if your use-case is only about Prometheus, you should consider removing these high cardinality fields from your map.