Add the ability to utilize Go templates to add dynamic custom fields

[marc-sensenich]

Motivation

Currently we're using custom fields to add a static field to all of the Falco events being handled by sidekick. However, we'd like to utilize the Kubernetes labels on the Falco event to determine value for the field we're adding to Falco and fall back to a default if need be.

Feature

When declaring custom fields to add to Falco events flowing through sidekick, allow for the ability to add a custom field based on a Go template, similar to the message format Go templates; this may have to be in the configuration file only.

customfields: # custom fields are added to falco events
  Akey: '{{ or (index . k8s.ns.labels.foo) "bar" }}'

Alternatives

• I've used -p on Falco to add the field, but it doesn't support the logic provided by Go templates. Can achieve something similar by using the program_output
• I've considered using a Falco custom plugin for doing enrichment of the events, but due to limitations as of today with the plugin architecture it can't pick up syscall events.
• I've considered using another tool as an intermediate process to forward events from Falco to sidekick. However, would not like to add an additional system if not needed.

Additional context

[Issif]

Hello,

I don't understand exactly what you're trying to do. In your example, is k8s.ns.labels.foo the label of the namespace of Falco or of the namespace of the Pod within the event occurred?
If it's the label of the namespace of Falco, it's a static intel, you can add as custom_fields easily --set falcosidekick.config.customfields="Akey:<ns_falco>".
It it's the label of the Pod within the event occurred, the labels for the namespace are available because Falco scrapes the metadata from K8s API for each event with -k option (by default in the Helm chart), you have to edit the output of your rules to add %k8s.ns.labels.foo and it will available as field.

[marc-sensenich]

@Issif I'm trying to use the label from the Pod within the event occurred and if that is not defined use a default value. Then rename the field to something that's not k8s.pod.label.foo; e.g.fizz. So for example, if I have Falco running with -k and -p fizz=%k8s.pod.label.foo, I'd get an event from Falco to Falco Sidekick that looks something like

{
    "output": "14:12:32.215952832: <message> (<falco_fields>...) fizz=example",
    "priority": "Warning",
    "rule": "Read sensitive file trusted after startup",
    "source": "syscall",
    "tags": [
        "filesystem",
        "mitre_credential_access"
    ],
    "time": "2022-06-14T14:12:32.215952832Z",
    "output_fields": {
        "container.id": "cd15541bbb03",
        "container.image.repository": "docker.io/falcosecurity/event-generator",
        "evt.time": 1655215952215952832,
        "fd.name": "/etc/shadow",
        "k8s.pod.label.foo": "example",
        "proc.aname[2]": "<NA>",
        "proc.cmdline": "httpd --loglevel info run ^syscall.ReadSensitiveFileUntrusted$ --sleep 6s",
        "proc.pname": "httpd",
        "user.loginuid": -1,
        "user.name": "<NA>"
    }
}

Notice how the output_fields contains k8s.pod.label.foo and not fizz as the key. What I'd desire to do is using either custom_fields or maybe a new attribute called templated_fields is to define a Go template which allows for me to add a field to the events forwarded by Falco Sidekick based on the event that Falco Sidekick has received. For example

# Static custom fields
customfields:
  custom_field_one: "static"

templatedfields:
  templated_field_one:  '{{ or (index . k8s.pod.label.foo) "bar" }}'

So that the final fields forwarded along to the configured outputs of Falco Sidekick would look like the following

{
    "output": "14:12:32.215952832: <message> (<falco_fields>...) fizz=example",
    "priority": "Warning",
    "rule": "Read sensitive file trusted after startup",
    "source": "syscall",
    "tags": [
        "filesystem",
        "mitre_credential_access"
    ],
    "time": "2022-06-14T14:12:32.215952832Z",
    "output_fields": {
        "container.id": "cd15541bbb03",
        "container.image.repository": "docker.io/falcosecurity/event-generator",
        "evt.time": 1655215952215952832,
        "fd.name": "/etc/shadow",
        "k8s.pod.label.foo": "example",
        "proc.aname[2]": "<NA>",
        "proc.cmdline": "httpd --loglevel info run ^syscall.ReadSensitiveFileUntrusted$ --sleep 6s",
        "proc.pname": "httpd",
        "user.loginuid": -1,
        "user.name": "<NA>",
        "custom_field_one": "static",
        "templated_field_one": "example"
    }
}

Currently customfields supports the static use case, but there isn't the ability to set a field with Falco Sidekick based on the input of the Falco event. Something similar can be achieved with program output from Falco, but it feels like it could be a feature of Falco sidekick as Falco Sidekick already supports templating for certain outputs.

[Issif]

If -p fizz=%k8s.pod.label.foo adds an item in the output but fizz: example isn't present in the output_fields, it means the issue is at Falco level. Falcosidekick has only what Falco sends to it, adding fields that are not in the source JSON can't be done, except by adding a way for Falcosidekick to collect them from K8s control plane, what I don't want. Falcosidekick is a proxy, I want to keep it unaware is running inside a cluster. Does it make sense to you?

[marc-sensenich]

Without digging too deep into the Falco code, I'm assuming that in the example I've shared that k8s.pod.label.foo appears in place of foo due to Falco itself only projecting its supported fields as defined at https://falco.org/docs/rules/supported-fields. I'd say that's expected behavior and not an issue.

It makes sense that you don't want Falcosidekick to be aware of where it's running. A I don't believe that this feature request is a request to make Falcosidekick as part of this feature request is that Falcosidekick has the ability to ability to template new custom output fields with the existing JSON that Falco sends to it, similar to how there are output sources that can format a message given the existing JSON from Falco.

[Issif]

Ok, now I see what you're talking about. I'll try to propose you something asap.

[marc-sensenich]

Thank you @Issif

[Issif]

Here's the current status of my tests.

With these settings:

templatedfields: # templated fields are added to falco events and metrics
  Dkey: '{{ or (index . "k8s.ns.labels.foo") "bar" }}'

A test curl with the value k8s.ns.labels.foo present in the output_fields:

curl -XPOST "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","k8s.ns.labels.foo": "default","proc.cmdline":"touch /bin/hack","user.name":"root"}}'

Gives this result:

2022/07/28 18:39:03 [DEBUG] : Falco's payload : {"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z","output_fields":{"akey":"AValue","bkey":"BValue","ckey":"CValue","dkey":"default","evt.time":1507591916746609046,"fd.name":"/bin/hack","k8s.ns.labels.foo":"default","proc.cmdline":"touch /bin/hack","user.name":"root"},"source":"syscalls"}

Without the value in output_fields, we have:

2022/07/28 18:45:38 [DEBUG] : Falco's payload : {"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z","output_fields":{"akey":"AValue","bkey":"BValue","ckey":"CValue","dkey":"bar","evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"},"source":"syscalls"}

The templated fields are executed in last, it means you can use also use the customfields in your template:

customfields:
  Akey: "AValue"
  Bkey: "BValue"
  Ckey: "CValue"

templatedfields: # templated fields are added to falco events and metrics and used all output_fields as data
  Dkey: '{{ or (index . "ckey") "bar" }}'

Do you want to test an image?

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten