Log pushes for Kubernetes Audit Events to Loki are failing

[fjogeleit]

Describe the bug

I'm using falcosidekick to send host os events and Kubernetes audit events to Grafana Loki. If a host os rule is triggered, the event is send to Loki as expected. If I trigger a Kubernetes Audit Event the push to Loki failed with 400 - Header Missing.

Working example for Host Rule:

The Example described in the Documentation https://falco.org/docs/event-sources/kubernetes-audit/#example failed:

2021/02/06 19:26:16 [DEBUG] : Falco's payload : {"output":"20:25:53.706802944: Notice K8s ConfigMap Deleted (user=system:serviceaccount:cattle-system:kontainer-engine configmap=my-config ns=stage resp=200 decision=allow reason=RBAC: allowed by ClusterRoleBinding \"globaladmin-user-r62pf\" of ClusterRole \"cluster-admin\" to User \"user-r62pf\")","priority":"Notice","rule":"K8s ConfigMap Deleted","time":"2021-02-06T19:25:53.706802944Z","output_fields":{"jevt.time":"20:25:53.706802944","ka.auth.decision":"allow","ka.auth.reason":"RBAC: allowed by ClusterRoleBinding \"globaladmin-user-r62pf\" of ClusterRole \"cluster-admin\" to User \"user-r62pf\"","ka.response.code":"200","ka.target.name":"my-config","ka.target.namespace":"stage","ka.user.name":"system:serviceaccount:cattle-system:kontainer-engine","source":"falco"}}
2021/02/06 19:26:16 [DEBUG] : Loki payload : {"streams":[{"labels":"{katargetname=\"my-config\",katargetnamespace=\"stage\",kausername=\"system:serviceaccount:cattle-system:kontainer-engine\",source=\"falco\",jevttime=\"20:25:53.706802944\",kaauthdecision=\"allow\",kaauthreason=\"RBAC: allowed by ClusterRoleBinding \"globaladmin-user-r62pf\" of ClusterRole \"cluster-admin\" to User \"user-r62pf\"\",karesponsecode=\"200\",rule=\"K8s ConfigMap Deleted\",priority=\"Notice\"}","entries":[{"ts":"2021-02-06T19:25:53Z","line":"20:25:53.706802944: Notice K8s ConfigMap Deleted (user=system:serviceaccount:cattle-system:kontainer-engine configmap=my-config ns=stage resp=200 decision=allow reason=RBAC: allowed by ClusterRoleBinding \"globaladmin-user-r62pf\" of ClusterRole \"cluster-admin\" to User \"user-r62pf\")"}]}]}

2021/02/06 19:26:16 [ERROR] : Loki - Header missing (400)
2021/02/06 19:26:16 [ERROR] : Loki - Header missing

How to reproduce it

1. Installing falco
2. Configure Kube API Server to send Kubernetes Audit Events to falco
3. Install falcosidekick and configure falco http_output send logs
4. Configure falcosidekick to send logs to Loki
5. Try the ConfigMap Example from the Documentation

Expected behaviour

Sends Kubernetes Audit Events successfully to Loki

Screenshots

See the issue description

Environment

• Falco version:

Falco 0.27.0
Driver version: 5c0b863ddade7a45568c0ac97d037422c9efb750

• System info:

{
"machine": "x86_64",
"nodename": "dev-node-3",
"release": "5.11.0-051100rc6-generic",
"sysname": "Linux",
"version": "#202101312230 SMP Sun Jan 31 22:33:58 UTC 2021"
}

• OS: Ubuntu 20.04
• Kernel: Linux 5.11.0-051100rc6-generic
• Installation method: DEB for falco, Helm for falcosidekick

[fjogeleit]

@Issif just as verification, its working now with the latest image

[CXYALEX]

@fjogeleit I have the same problem. how did you solve it.

[epcim]

I see it either, 50% events to loki.

Very simple log on sidekick side.

Could \u003cNA\u003e (without ") be the cause. .I received <NA> without any problem on fd.name for example in other cases.. but in "

2022/09/30 08:48:58 [ERROR] : Loki - header missing (400)
2022/09/30 08:48:58 [ERROR] : Loki - header missing
2022/09/30 08:49:32 [DEBUG] : Falco's payload : {"output":"08:49:32.780838175: Warning Privilege escalation activity (user=root auser=\u003cNA\u003e command=kubectl get po -n monitoring ppid=31150 apid=23500proc.apid pid=23500 gparent=su ggparent=sudo gggparent=bash user_loginuid=2201 parent=bash pcmdline=bash ) k8s.ns=\u003cNA\u003e k8s.pod=\u003cNA\u003e container=host","priority":"Warning","rule":"Detect su or sudo","time":"2022-09-30T08:49:32.780838175Z","output_fields":{"":23500,"cluster_name":"sv10-sjc-dev-int-xxx.xx","container.id":"host","evt.time":1664527772780838175,"group":"Ves-Internal","identifier":"falco.sv10-sjc-dev.int.xxx-xx","k8s.ns.name":null,"k8s.pod.name":null,"proc.aname[2]":"su","proc.aname[3]":"sudo","proc.aname[4]":"bash","proc.cmdline":"kubectl get po -n monitoring","proc.pcmdline":"bash","proc.pid":23500,"proc.pname":"bash","proc.ppid":31150,"tenant":"ves-sre","user.loginname":"\u003cNA\u003e","user.loginuid":2201,"user.name":"root"},"source":"syscall","tags":["process","su","sudo"],"origin_host":""}
2022/09/30 08:49:32 [DEBUG] : Loki payload : {"streams":[{"labels":"{=\"23500\",procaname3=\"sudo\",procpcmdline=\"bash\",userloginuid=\"2201\",identifier=\"falco.sv10-sjc-dev.int.xxx.xx\",userloginname=\"\u003cNA\u003e\",cluster_name=\"sv10-sjc-dev-int-xxx-xx\",tenant=\"ves-sre\",containerid=\"host\",procaname2=\"su\",proccmdline=\"kubectl get po -n monitoring\",group=\"Ves-Internal\",evttime=\"1664527772780838175\",procaname4=\"bash\",procpid=\"23500\",procpname=\"bash\",procppid=\"31150\",username=\"root\",tags=\"process,su,sudo\",rule=\"Detect su or sudo\",source=\"syscall\",priority=\"Warning\",app=\"falco\",type=\"event\",severity=\"minor\",origin_host=\"10.62.53.10\"}","entries":[{"ts":"2022-09-30T08:49:32Z","line":"08:49:32.780838175: Warning Privilege escalation activity (user=root auser=\u003cNA\u003e command=kubectl get po -n monitoring ppid=31150 apid=23500proc.apid pid=23500 gparent=su ggparent=sudo gggparent=bash user_loginuid=2201 parent=bash pcmdline=bash ) k8s.ns=\u003cNA\u003e k8s.pod=\u003cNA\u003e container=host"}]}]}

2022/09/30 08:49:32 [ERROR] : Loki - header missing (400)
2022/09/30 08:49:32 [ERROR] : Loki - header missing

mind

"labels":"{=\"23500\"

and

apid=23500proc.apid 

I do have some local modifications (but aware of these .. ) going to try to skip value with missing key.. now

[epcim]

possibly related https://githubhelp.com/falcosecurity/falcosidekick/issues/77

[Issif]

The Loki output for next release 2.27 will be upgraded, hope it will help you #356

[Issif]

The 2.27.0 is out, and the helm charts are updated falco.org/blog/falcosidekick-2-27-0-ui-2-1-0

[sreejithsoman-mc]

@Issif I get this issue in falcosidekick 2.27v version as well ? Do we have any fix for it?

2023/06/12 09:00:48 [ERROR] : Loki - header missing (400): entry with timestamp 2023-06-11 20:50:23.331614 +0000 UTC ignored, reason: 'entry too far behind' for stream: {cluster="devstage", from="falcosidekick", hostname="falco-z2rwf", priority="Informational", rule="K8s Serviceaccount Created", source="k8s_audit", tags="k8s"}, total ignored: 1 out of 1

[Issif]

First time I see this issue.

Are you sure your hosts' times are correct? I see a big diff between the log line of falcosidekick (2023/06/12 09:00:48) and the timestamp of the event 2023-06-11 20:50:23.331614 +0000 UTC).