Collect audit logs from kube-apiserver

[jaehnri]

This is how the audit logs are shown in OpenSearch:

{
    "_id": "5f2b9baf-cbd4-44bf-b81f-b5ad28e0aff1",
    "_index": "logs-v0.5.4-000003",
    "_score": null,
    "_source": {
        "anomaly_level": "",
        "Body.value": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"234fa224-b400-404e-931e-46effd64a321\",\"stage\":\"ResponseStarted\",\"requestURI\":\"/api/v1/pods?allowWatchBookmarks=true\\u0026resourceVersion=3963946\\u0026timeout=5m12s\\u0026timeoutSeconds=312\\u0026watch=true\",\"verb\":\"watch\",\"user\":{\"username\":\"system:kube-controller-manager\",\"groups\":[\"system:authenticated\"]},\"sourceIPs\":[\"127.0.0.1\"],\"userAgent\":\"k3s/v1.26.7+k3s1 (linux/amd64) kubernetes/e47cfc0/shared-informers\",\"objectRef\":{\"resource\":\"pods\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2023-08-30T19:46:38.513630Z\",\"stageTimestamp\":\"2023-08-30T19:46:38.514778Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:kube-controller-manager\\\" of ClusterRole \\\"system:kube-controller-manager\\\" to User \\\"system:kube-controller-manager\\\"\"}}",
        "cluster_id": "e1967963-f4ec-4e4a-afa1-a766ff0ddb18",
        "deployment": "",
        "k8s.auditlog.audit_id": "234fa224-b400-404e-931e-46effd64a321",
        "k8s.auditlog.level": "Metadata",
        "k8s.auditlog.resource": "pods",
        "k8s.auditlog.stage": "ResponseStarted",
        "k8s.auditlog.stage_timestamp": "2023-08-30T19:46:38.514778Z",
        "kubernetes_component": "",
        "log": "",
        "log_type": "controlplane",
        "namespace_name": "",
        "pod_name": "",
        "service": "",
        "SeverityNumber": 0,
        "template_matched": "",
        "time": 1693424798514,
        "TraceFlags": 0
    },
    "_version": 1,
    "fields": {
        "time": [
            "2023-08-30T19:46:38.514Z"
        ]
    },
    "sort": [
        1693424798514
    ]
}

As the audit log is a bit complex, I chose a few properties that make more sense to be parsed from the JSON log:

• "k8s.auditlog.audit_id": "234fa224-b400-404e-931e-46effd64a321",
• "k8s.auditlog.level": "Metadata";
• "k8s.auditlog.resource": "pods";
• "k8s.auditlog.stage": "ResponseStarted";
• "k8s.auditlog.stage_timestamp": "2023-08-30T19:46:38.514778Z"

The whole log can still be checked in the body.

Testing config

For testing, I used the following Kubernetes auditLog configuration:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods"]

And this logging.opni.io configuration in a K3s cluster:

...
spec:
  kubeAuditLogs:
    auditFilename: audit.log
    enabled: true
    pathPrefix: /var/lib/rancher/k3s/server/logs
  provider: k3s
  selector: {}

Has been addressed.