Amazon Linux 2 Support for BPF-LSM

[vennemp]

I have AL2 EKS worker nodes running on Kernel version 5.15. Do you have any guidance or documentation on enabling BPF-LSM in kubearmor for AL2 nodes?

The active LSM is blank and all the Postures are listed as "audit" mode.

[nyrahul]

@vennemp , this seems strange... Can you provide us with the AMI ID so that we can setup a local env?

[nyrahul]

@vennemp , can you explain the steps on how to get this setup done? Thanks

[vennemp]

@nyrahul Sorry just saw this. I used the latest AL2 K8s 1.21 ami - ami-0ebf68cf21e6c55fd (there may be anewer one). This uses kernel 5.4 by default so I manually updated the kernel to 5.15 using the steps outlined here.

https://aws.amazon.com/premiumsupport/knowledge-center/amazon-linux-2-kernel-upgrade/

Are there additionally packages that need to be installed? I reached out to our TAM at AWS and they did not have any guidance on this - they only had guidance for Bottlerocket. However, even that was not working after following the instructions in this article. Maybe I'm missing something. Are there custom packages that need to be installed for BPF-LSM to be enabled on AL2 and Bottlerocket?

https://aws.amazon.com/blogs/containers/secure-bottlerocket-deployments-on-amazon-eks-with-kubearmor/

[nyrahul]

Hey @vennemp , thank you for the response. We will setup this env ourselves and check this support. As mentioned in the kubearmor support matrix, with the default amazon linux (kernel 5.4) only audit will work. But with higher kernel versions, the BPF-LSM typically is available and should work. We validated this before by installing amazon linux 2 (kernel 5.8) on an EC2 instance and then using k3s on top it.

Btw, the EKS Bottlerocket simply works with these 3 steps:

1. curl -O https://raw.githubusercontent.com/bottlerocket-os/bottlerocket/develop/sample-eksctl.yaml
2. eksctl create cluster -f sample-eksctl.yaml
3. karmor install

[vennemp]

I have run this and it works it still does not show blocking mode - just audit mode. But it does show BPF-LSM mode enabled

[vennemp]

Disregard the last piece - it appears to just be a bug in the karmor probe command. I created a sample policy to block sleep and it successfully blocked.

Appreciate your help. Let me know what you find with the AL2 support

[nyrahul]

I have run this and it works it still does not show blocking mode - just audit mode. But it does show BPF-LSM mode enabled

The Host/Container Security is set to true, which indicates that the blocking mode is possible.

The "Default Posture" indicates what should happen to actions that are not in the allowed list. Should they be audited or blocked? For e.g., if you have the following policy,

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: auto-generarated-policy
  namespace: dvwa
spec: 
  action: Allow
  file:
    matchDirectories:
    - dir: /
      recursive: true
  process:
    matchPaths:
    - path: /usr/sbin/apache2
    - path: /bin/ping
  selector:
    matchLabels:
      app: dvwa-web
      tier: frontend
  severity: 1
---

The policy states that allow execution of only /usr/sbin/apache2 and /bin/ping processes. If any other process is executed, it would result in an Audit event. Note that if there is an explicit Block policy that denies something specific, it would result in blocking action.
The Default Posture indicates what happens to unknown events that are not part of the allowed list.

Further regarding Default Posture, it is possible to set posture at individual namespace level (ref). For e.g, you can have two namespaces, one with default audit and another with default block. Our users indicated that they have workloads with different maturity in different namespaces and hence they would like different default postures.

Hope this makes sense. Would be great to hear your thoughts on this.

[vennemp]

That makes sense! Thank you for the explanation.

How is it coming with the analysis of AL2 / EBF-LPM support?

[nyrahul]

Hey @vennemp, Good News! We know the reason why BPF-LSM didn't work for AL2 5.15 kernel. Al2 5.15 kernel already has BPF-LSM enabled but the bpf filesystem is not mounted.

If you mount the bpf filesystem from the AL2 node, it will work. Steps:

1. Mount bpf fs: mount -t bpf bpffs /sys/fs/bpf
2. Restart kubearmor pod: kubectl delete pod -n kube-system <KUBEARMOR-POD>
3. karmor probe will now start showing BPFLSM as the enforcer and Container Security will be set to true.
4. Blocking policies should start working

We will check how to automate the fs mounting. This will be automated in v0.9 for sure.

karmor probe output

❯ karmor probe

Found KubeArmor running in Kubernetes

Daemonset :
        kubearmor       Desired: 1      Ready: 1        Available: 1
Deployments : 
        kubearmor-annotation-manager    Desired: 1      Ready: 1        Available: 1
        kubearmor-host-policy-manager   Desired: 1      Ready: 1        Available: 1
        kubearmor-policy-manager        Desired: 1      Ready: 1        Available: 1
        kubearmor-relay                 Desired: 1      Ready: 1        Available: 1
Containers : 
        kubearmor-annotation-manager-85857fc8d7-k5lgg   Running: 2      Image Version: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
        kubearmor-bzcqn                                 Running: 1      Image Version: kubearmor/kubearmor:stable               
        kubearmor-host-policy-manager-7989bc8f6c-bnn47  Running: 2      Image Version: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
        kubearmor-policy-manager-8876798d6-t8682        Running: 2      Image Version: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
        kubearmor-relay-64c6fff875-9wj4t                Running: 1      Image Version: kubearmor/kubearmor-relay-server:latest  
Node 1 : 
        OS Image:                       Amazon Linux 2             
        Kernel Version:                 5.15.86-53.137.amzn2.x86_64
        Kubelet Version:                v1.24.7-eks-fb459a0        
        Container Runtime:              containerd://1.6.6         
        Active LSM:                     BPFLSM                     
        Host Security:                  true                       
        Container Security:             true                       
        Container Default Posture:      audit(File)                     audit(Capabilities)     audit(Network)
        Host Default Posture:           audit(File)                     audit(Capabilities)     audit(Network)
Armored Up pods : 
+-----------------+----------------------------+-----------------------------+
|    NAMESPACE    |            NAME            |           POLICY            |
+-----------------+----------------------------+-----------------------------+
| wordpress-mysql | mysql-6c6fcdccf-twqvg      |                             |
+                 +----------------------------+-----------------------------+
|                 | wordpress-84dbf54bb8-mrwf4 | ksp-wordpress-block-process |
+-----------------+----------------------------+-----------------------------+