Custom Fields that aren't Prometheus Label compliant cause HTTP panic

[marc-sensenich]

Describe the bug

When specifying a custom field with a name that doesn't adhere to the regex for Prometheus metric labels as outlined at https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels, it causes a panic and the sidekick is unable to process the event from Falco; e.g.

2022/06/13 17:49:06 [INFO]  : Falco Sidekick version: 2.25.0
2022/06/13 17:49:06 [INFO]  : Enabled Outputs : [WebUI]
2022/06/13 17:49:06 [INFO]  : Falco Sidekick is up and listening on :2801
2022/06/13 17:49:16 http: panic serving 10.244.0.22:55974: inconsistent label cardinality: expected 4 label values but got 5 in prometheus.Labels{"example-field":"value", "k8s_ns_name":"falco", "k8s_pod_name":"falco-event-generator-597f7cdcb-5s7f5", "priority":"Warning", "rule":"Read sensitive file trusted after startup"}
goroutine 11 [running]:
net/http.(*conn).serve.func1()
        go/src/net/http/server.go:1825 +0xbf
panic({0x18238a0, 0xc0006de190})
        go/src/runtime/panic.go:844 +0x258
github.com/prometheus/client_golang/prometheus.(*CounterVec).With(0xc0000e3768?, 0xc0006c2930?)
        pkg/mod/github.com/prometheus/client_golang@v1.9.0/prometheus/counter.go:259 +0x7b
main.newFalcoPayload({0x7f2da9855998?, 0xc00039ee80})
        /home/circleci/project/handlers.go:116 +0x907
main.mainHandler({0x1dece40, 0xc000476d20}, 0xc00031de00)
        /home/circleci/project/handlers.go:41 +0x265
net/http.HandlerFunc.ServeHTTP(0x17d0560?, {0x1dece40?, 0xc000476d20?}, 0xc00007ff00?)
        go/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0x0?, {0x1dece40, 0xc000476d20}, 0xc00031de00)
        go/src/net/http/server.go:2462 +0x149
net/http.serverHandler.ServeHTTP({0x1de0ce0?}, {0x1dece40, 0xc000476d20}, 0xc00031de00)
        go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00053f040, {0x1dee120, 0xc0006c2690})
        go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
        go/src/net/http/server.go:3071 +0x4db

How to reproduce it

# Run Falco sidekick with `CUSTOMFIELDS="example-field:value"`
$ docker run -d -p 2801:2801 -e CUSTOMFIELDS="example-field:value" --name falcosidekick falcosecurity/falcosidekick:2.25.0
0489e7871eaeebaf33fef64b59f5a33088d9a15afe067fd1c9dd5b7bced85249
# Send a test event
$ curl -XPOST http://localhost:2802/test
curl: (52) Empty reply from server
# See error in logs
$ docker logs falcosidekick
2022/06/13 19:07:13 [INFO]  : Falco Sidekick version: 2.25.0
2022/06/13 19:07:13 [INFO]  : Enabled Outputs : []
2022/06/13 19:07:13 [INFO]  : Falco Sidekick is up and listening on :2801
2022/06/13 19:07:18 http: panic serving 172.17.0.1:56704: inconsistent label cardinality: expected 4 label values but got 5 in prometheus.Labels{"example-field":"value", "k8s_ns_name":"", "k8s_pod_name":"", "priority":"Debug", "rule":"Test rule"}
goroutine 27 [running]:
net/http.(*conn).serve.func1()
        go/src/net/http/server.go:1825 +0xbf
panic({0x18238a0, 0xc0006a1fc0})
        go/src/runtime/panic.go:844 +0x258
github.com/prometheus/client_golang/prometheus.(*CounterVec).With(0xc0001136e0?, 0xc0006b62d0?)
        pkg/mod/github.com/prometheus/client_golang@v1.9.0/prometheus/counter.go:259 +0x7b
main.newFalcoPayload({0x7f07d6015148?, 0xc0006a1da0})
        /home/circleci/project/handlers.go:116 +0x907
main.mainHandler({0x1dece40, 0xc000538b60}, 0xc0006c2000)
        /home/circleci/project/handlers.go:41 +0x265
main.testHandler({0x1dece40, 0xc000538b60}, 0xc0006c2000)
        /home/circleci/project/handlers.go:73 +0x19d
net/http.HandlerFunc.ServeHTTP(0x7f07d6176f28?, {0x1dece40?, 0xc000538b60?}, 0x40d325?)
        go/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0x0?, {0x1dece40, 0xc000538b60}, 0xc0006c2000)
        go/src/net/http/server.go:2462 +0x149
net/http.serverHandler.ServeHTTP({0xc0006b60c0?}, {0x1dece40, 0xc000538b60}, 0xc0006c2000)
        go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00067ef00, {0x1dee120, 0xc00066dfb0})
        go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
        go/src/net/http/server.go:3071 +0x4db

Expected behaviour

Falco sidekick shouldn't panic when a custom field key doesn't adhere to the regex for Prometheus metric labels.

Screenshots

Environment

• Falco version:

falco --version
Falco version: 0.32.0
Driver version: 39ae7d40496793cf3d3e7890c9bbdc202263836b

• System info:

falco --support | jq .system_info
Mon Jun 13 18:05:39 2022: Falco version 0.32.0 (driver version 39ae7d40496793cf3d3e7890c9bbdc202263836b)
Mon Jun 13 18:05:39 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Jun 13 18:05:39 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Jun 13 18:05:39 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
{
  "machine": "x86_64",
  "nodename": "falco-vm",
  "release": "5.15.11-200.fc35.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Wed Dec 22 15:41:11 UTC 2021"
}

• Cloud provider or hardware configuration: Fedora VM running a kind cluster
• OS:

NAME="Fedora Linux"
VERSION="35 (Thirty Five)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Thirty Five)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

• Kernel:

uname -a
Linux falco-vm 5.15.11-200.fc35.x86_64 #1 SMP Wed Dec 22 15:41:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:

Kubernetes

Additional context

Custom fields to Prometheus labels PR: #314

[marc-sensenich]

@Issif for this issue, how do you feel about replacing all non-conforming characters in custom fields and output fields with underscores when they are being sent over to Prometheus and Loki as labels?

[Issif]

Hello,
I'm working on this bug right now, I noticed other issues. I will propose a PR soon.