From 6d81861f9b45cc07f1b52f095c121be258ced925 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Thu, 23 Nov 2017 19:45:11 +0100 Subject: [PATCH 1/5] Change auditing docs page for 1.9 release Signed-off-by: Mik Vyatskov --- docs/tasks/debug-application-cluster/audit.md | 372 ++++++------------ 1 file changed, 116 insertions(+), 256 deletions(-) diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index 3105b7cb15375..a465f8ab54f1c 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -9,7 +9,7 @@ title: Auditing * TOC {:toc} -Kubernetes Audit provides a security-relevant chronological set of records documenting +Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. It allows cluster administrator to answer the following questions: @@ -22,105 +22,54 @@ answer the following questions: - from where was it initiated? - to where was it going? -## Legacy Audit - -Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests -processed by the server. Each audit log entry contains two lines: - -1. The request line containing a unique ID to match the response and request metadata, such as the source IP, requesting user, impersonation information, resource being requested, etc. -2. The response line containing a unique ID matching the request line and the response code. - -Example output for `admin` user listing pods in the `default` namespace: - -``` -2017-03-21T03:57:09.106841886-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" ip="127.0.0.1" method="GET" user="admin" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/api/v1/namespaces/default/pods" -2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200" -``` - -Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default. -To fallback to this legacy audit, disable the advanced auditing feature -using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]: - -``` ---feature-gates=AdvancedAuditing=false -``` - -### Configuration - -[Kube-apiserver][kube-apiserver] provides the following options which are responsible -for configuring where and how audit logs are handled: - -- `audit-log-path` - enables the audit log pointing to a file where the requests are being logged to, '-' means standard out. -- `audit-log-maxage` - specifies maximum number of days to retain old audit log files based on the timestamp encoded in their filename. -- `audit-log-maxbackup` - specifies maximum number of old audit log files to retain. -- `audit-log-maxsize` - specifies maximum size in megabytes of the audit log file before it gets rotated. Defaults to 100MB. - -If an audit log file already exists, Kubernetes appends new audit logs to that file. -Otherwise, Kubernetes creates an audit log file at the location you specified in -`audit-log-path`. If the audit log file exceeds the size you specify in `audit-log-maxsize`, -Kubernetes will rename the current log file by appending the current timestamp on -the file name (before the file extension) and create a new audit log file. -Kubernetes may delete old log files when creating a new log file; you can configure -how many files are retained and how old they can be by specifying the `audit-log-maxbackup` -and `audit-log-maxage` options. - -## Advanced audit - -Kubernetes 1.7 expands auditing with experimental functionality such as event -filtering and a webhook for integration with external systems. Kubernetes 1.8 -upgrades the advanced audit feature to beta, and some backward incompatible changes -have been committed. +[Kube-apiserver][kube-apiserver] performs auditing. Each request on each stage +of its execution generates an event, which is then pre-processed according to +a certain policy and written to a backend. You can find more details about the +pipeline in the [design proposal][auditing-proposal]. +## Audit Policy -`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded, -and backends, which persist records. Backend implementations include logs files and -webhooks. +Audit policy defines rules about what events should be recorded and what data +they should include. When an event is processed, it's compared against the list +of rules in order. The first matching rule sets the [audit level][auditing-level] +of the event. The audit policy object structure is defined in the +[`audit.k8s.io` API group][audit-api]. -The structure of audit events changes when enabling the `AdvancedAuditing` feature -flag. This includes some cleanups, such as the `method` reflecting the verb evaluated -by the [authorization layer](/docs/admin/authorization/) instead of the [HTTP verb](/docs/admin/authorization/#determine-the-request-verb). -Also, instead of always generating two events per request, events are recorded with an associated "stage". -The known stages are: - -- `RequestReceived` - The stage for events generated as soon as the audit handler receives the request. -- `ResponseStarted` - Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch). -- `ResponseComplete` - Once the response body has been completed. -- `Panic` - Events generated when a panic occurred. - -### Audit Policy - -Audit policy is a document defining rules about what events should be recorded. -The policy is passed to the [kube-apiserver][kube-apiserver] using the -`--audit-policy-file` flag. - -``` ---audit-policy-file=/etc/kubernetes/audit-policy.yaml -``` - -If `AdvancedAuditing` is enabled and this flag is omitted, no events are logged. - -The policy file holds rules that determine the level of an event. Known audit levels are: - -- `None` - don't log events that match this rule. -- `Metadata` - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body. -- `Request` - log event metadata and request body but not response body. -- `RequestResponse` - log event metadata, request and response bodies. - -When an event is processed, it's compared against the list of rules in order. -The first matching rule sets the audit level of the event. The audit policy is -defined by the [`audit.k8s.io` API group][audit-api]. -Some new fields are supported in beta version, like `resourceNames` and `omitStages`. - -In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in -the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide -a valid `apiVersion` and `kind` value will be treated as illgal. +You can pass a file with the policy to [kube-apiserver][kube-apiserver] +using the `--audit-policy-file` flag. If the flag is omitted, no events are logged. +__Note:__ `kind` and `apiVersion` fields along with `rules` __must__ be provided +in the audit policy file. A policy with 0 rules, or a policy that doesn't +provide valid `apiVersion` and `kind` values is treated as illgal. Some example audit policy files: ```yaml -apiVersion: audit.k8s.io/v1beta1 #this is required in Kubernetes 1.8 +apiVersion: audit.k8s.io/v1beta1 # This is required. kind: Policy +# Don't generate audit events for all requests in RequestReceived stage. +omitStages: + - "RequestReceived" rules: + # Log pod changes at RequestResponse level + - level: RequestResponse + resources: + - group: "" + # Resource "pods" doesn't match requests to any subresource of pods, + # which is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] + + # Don't log requests to a configmap called "controller-leader" + - level: None + resources: + - group: "" + resources: ["configmaps"] + resourceNames: ["controller-leader"] + # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] @@ -159,38 +108,8 @@ rules: # A catch-all rule to log all other requests at the Metadata level. - level: Metadata -``` - -The next audit policy file shows new features introduced in Kubernetes 1.8: - -```yaml -apiVersion: audit.k8s.io/v1beta1 -kind: Policy -rules: - # Log pod changes at Request level - - level: Request - resources: - - group: "" - # Resource "pods" no longer matches requests to any subresource of pods, - # This behavior is consistent with the RBAC policy. - resources: ["pods"] - # Log "pods/log", "pods/status" at Metadata level - - level: Metadata - resources: - - group: "" - resources: ["pods/log", "pods/status"] - - # Don't log requests to a configmap called "controller-leader" - - level: None - resources: - - group: "" - resources: ["configmaps"] - resourceNames: ["controller-leader"] - - # A catch-all rule to log all other requests at the Metadata level. - # For this rule we use "omitStages" to omit events at "ReqeustReceived" stage. - # Events in this stage will not be sent to backend. - - level: Metadata + # Long-running requests like watches that fall under this rule will not + # generate an audit event in RequestReceived. omitStages: - "RequestReceived" ``` @@ -208,153 +127,50 @@ rules: The [audit profile used by GCE][gce-audit-profile] should be used as reference by admins constructing their own audit profiles. -### Audit backends - -Audit backends implement strategies for emitting events. The [kube-apiserver][kube-apiserver] -provides a logging and webhook backend. - -Each request to the API server can generate multiple events, one when the request is received, -another when the response is sent, and additional events for long running requests (such as -watches). The ID of events will be the same if they were generated from the same request. - -The event format is defined by the `audit.k8s.io` API group. The `v1alpha1` format of this -API can be found [here][audit-api] with more details about the exact fields captured. - -#### Log backend +## Audit backends -The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing` -feature flag. All generated events defined by `--audit-policy-file` are recorded in structured -json format: +Audit backends implement exporting audit events to an external storage. +[Kube-apiserver][kube-apiserver] out of the box provides two backends: -``` -{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}} -{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}} -``` - -In alpha version, objectRef.apiVersion holds both the api group and version. -In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion. +- Log backend, which writes events to a disk. +- Webhook backend, which sends events to an external API. -Starting from Kubernetes 1.8, structured json format is used for log backend by default. -Use the following option to switch log to legacy format: - -``` ---audit-log-format=legacy -``` +In both cases, audit events structure is defined by the API in the +`audit.k8s.io` API group. The current version of the API is +[`v1beta1`][auditing-beta-api]. -With legacy format, events are formatted as follows: +### Log backend -``` -2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="" -2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200" -``` +Log backend writes audit events to a file in JSON format. You can configure +log audit backend using the following [kube-apiserver][kube-apiserver] flags: -Logged events omit the request and response bodies. The `Request` and -`RequestResponse` levels are equivalent to `Metadata` for legacy format. This legacy format -of advanced audit is different from the [Legacy Audit](# Legacy Audit) discussed above, such -as changes to the method values and the introduction of a "stage" for each event. +- `--audit-log-path` specifies the log file path, that log backend uses to write + audit events. Not specifying this flag disables log backend. +- `--audit-log-maxage` defined the maximum number of days to retain old audit log files. +- `--audit-log-maxbackup` defines the maximum number of audit log files to retain. +- `--audit-log-maxsize` defines the maximum size of the audit log file before it gets rotated. -#### Webhook backend +### Webhook backend -The audit webhook backend can be used to have [kube-apiserver][kube-apiserver] -send audit events to a remote service. The webhook requires the `AdvancedAuditing` -feature flag and is configured using the following command line flags: +Webhook backend sends audit events to a remote API, which is assumed to be the +same API as [kube-apiserver][kube-apiserver] exposes. You can configure webhook +audit backend using the following kube-apiserver flags: -``` ---audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig ---audit-webhook-mode=batch -``` - -`audit-webhook-mode` controls buffering strategies used by the webhook. Known modes are: - -- `batch` - buffer events and asynchronously send the set of events to the external service. -- `blocking` - block API server responses on sending each event to the external service. +- `--audit-webhook-config-file` specifies the path to a file with a webhook + configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig]. +- `--audit-webhook-mode` define the buffering strategy, one of the following: + - `batch` - buffer events and asynchronously send the set of events to the external service. + - `blocking` - block API server responses on sending each event to the external service. The webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it. -``` -# clusters refers to the remote service. -clusters: - - name: name-of-remote-audit-service - cluster: - certificate-authority: /path/to/ca.pem # CA for verifying the remote service. - server: https://audit.example.com/audit # URL of remote service to query. Must use 'https'. - -# users refers to the API server's webhook configuration. -users: - - name: name-of-api-server - user: - client-certificate: /path/to/cert.pem # cert for the webhook plugin to use - client-key: /path/to/key.pem # key matching the cert - -# kubeconfig files require a context. Provide one for the API server. -current-context: webhook -contexts: -- context: - cluster: name-of-remote-audit-service - user: name-of-api-sever - name: webhook -``` - -Events are POSTed as a JSON serialized `EventList`. An example payload: - -```json -{ - "apiVersion": "audit.k8s.io/v1beta1", - "items": [ - { - "auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886", - "level": "Metadata", - "metadata": { - "creationTimestamp": null - }, - "objectRef": { - "apiGroup": "rbac.authorization.k8s.io", - "apiVersion": "v1", - "name": "jane", - "namespace": "default", - "resource": "roles" - }, - "requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane", - "responseStatus": { - "code": 200, - "metadata": {} - }, - "sourceIPs": [ - "172.16.116.128" - ], - "stage": "ResponseComplete", - "timestamp": "2017-09-05T10:20:24Z", - "user": { - "groups": [ - "system:masters", - "system:authenticated" - ], - "username": "kubecfg" - }, - "verb": "get" - } - ], - "kind": "EventList", - "metadata": {} -} -``` - -### Audit-Id - -Audit-Id is a unique ID for each http request to kube-apiserver. The ID of events will be the -same if they were generated from the same request. Starting from Kubernetes 1.8, if an audit -event is generated for the request, kube-apiserver will respond with an Audit-Id in the HTTP header. -Note that for some special requests like `kubectl exec`, `kubectl attach`, kube-apiserver works -like a proxy, no Audit-Id will be returned even if audit events are recorded. - -### Log Collector Examples +## Log Collector Examples -#### Use fluentd to collect and distribute audit events from log file +### Use fluentd to collect and distribute audit events from log file [Fluentd][fluentd] is an open source data collector for unified logging layer. In this example, we will use fluentd to split audit events by different namespaces. -Note that this example requries json format output support in Kubernetes 1.8. 1. install [fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter][fluentd_install_doc] in the kube-apiserver node 1. create a config file for fluentd @@ -412,7 +228,7 @@ Note that this example requries json format output support in Kubernetes 1.8. --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json 1. check audits for different namespaces in /var/log/audit-*.log -#### Use logstash to collect and distribute audit events from webhook backend +### Use logstash to collect and distribute audit events from webhook backend [Logstash][logstash] is an open source, server-side data processing tool. In this example, we will use logstash to collect audit events from webhook backend, and save events of @@ -477,9 +293,53 @@ Note that in addition to file output plugin, logstash has a variety of outputs t let users route data where they want. For example, users can emit audit events to elasticsearch plugin which supports full-text search and analytics. -[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go +## Legacy Audit + +__Note:__ Legacy Audit is deprecated and is disabled by defaule since Kubernetes 1.8. +To fallback to this legacy audit, disable the advanced auditing feature +using the `AdvancedAuditing` feature gate in [kube-apiserver][kube-apiserver]: + +``` +--feature-gates=AdvancedAuditing=false +``` + +In legacy format, each audit log entry contains two lines: + +1. The request line containing a unique ID to match the response and request metadata, such as the source IP, requesting user, impersonation information, resource being requested, etc. +2. The response line containing a unique ID matching the request line and the response code. + +Example output for `admin` user listing pods in the `default` namespace: + +``` +2017-03-21T03:57:09.106841886-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" ip="127.0.0.1" method="GET" user="admin" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/api/v1/namespaces/default/pods" +2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200" +``` + +### Configuration + +[Kube-apiserver][kube-apiserver] provides the following options which are responsible +for configuring where and how audit logs are handled: + +- `audit-log-path` - enables the audit log pointing to a file where the requests are being logged to, '-' means standard out. +- `audit-log-maxage` - specifies maximum number of days to retain old audit log files based on the timestamp encoded in their filename. +- `audit-log-maxbackup` - specifies maximum number of old audit log files to retain. +- `audit-log-maxsize` - specifies maximum size in megabytes of the audit log file before it gets rotated. Defaults to 100MB. + +If an audit log file already exists, Kubernetes appends new audit logs to that file. +Otherwise, Kubernetes creates an audit log file at the location you specified in +`audit-log-path`. If the audit log file exceeds the size you specify in `audit-log-maxsize`, +Kubernetes will rename the current log file by appending the current timestamp on +the file name (before the file extension) and create a new audit log file. +Kubernetes may delete old log files when creating a new log file; you can configure +how many files are retained and how old they can be by specifying the `audit-log-maxbackup` +and `audit-log-maxage` options. + [kube-apiserver]: /docs/admin/kube-apiserver -[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532 +[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md +[auditing-level]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md#levels +[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go +[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L532 +[kubeconfig]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ [fluentd]: http://www.fluentd.org/ [fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd [logstash]: https://www.elastic.co/products/logstash From c8e6f71fd479ba52fce866b519458233bb2e9210 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Mon, 27 Nov 2017 16:13:19 +0100 Subject: [PATCH 2/5] Address review comments Signed-off-by: Mik Vyatskov --- docs/tasks/debug-application-cluster/audit.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index a465f8ab54f1c..c015b967f93ea 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -33,13 +33,13 @@ Audit policy defines rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the [audit level][auditing-level] of the event. The audit policy object structure is defined in the -[`audit.k8s.io` API group][audit-api]. +[`audit.k8s.io` API group][auditing-api]. You can pass a file with the policy to [kube-apiserver][kube-apiserver] using the `--audit-policy-file` flag. If the flag is omitted, no events are logged. __Note:__ `kind` and `apiVersion` fields along with `rules` __must__ be provided in the audit policy file. A policy with 0 rules, or a policy that doesn't -provide valid `apiVersion` and `kind` values is treated as illgal. +provide valid `apiVersion` and `kind` values is treated as illegal. Some example audit policy files: @@ -137,18 +137,18 @@ Audit backends implement exporting audit events to an external storage. In both cases, audit events structure is defined by the API in the `audit.k8s.io` API group. The current version of the API is -[`v1beta1`][auditing-beta-api]. +[`v1beta1`][auditing-api]. ### Log backend Log backend writes audit events to a file in JSON format. You can configure log audit backend using the following [kube-apiserver][kube-apiserver] flags: -- `--audit-log-path` specifies the log file path, that log backend uses to write - audit events. Not specifying this flag disables log backend. +- `--audit-log-path` specifies the log file path that log backend uses to write + audit events. Not specifying this flag disables log backend. `-` means standard out - `--audit-log-maxage` defined the maximum number of days to retain old audit log files. - `--audit-log-maxbackup` defines the maximum number of audit log files to retain. -- `--audit-log-maxsize` defines the maximum size of the audit log file before it gets rotated. +- `--audit-log-maxsize` defines the maximum size in megabytes of the audit log file before it gets rotated. ### Webhook backend @@ -160,6 +160,7 @@ audit backend using the following kube-apiserver flags: configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig]. - `--audit-webhook-mode` define the buffering strategy, one of the following: - `batch` - buffer events and asynchronously send the set of events to the external service. + This is the default. - `blocking` - block API server responses on sending each event to the external service. The webhook config file uses the kubeconfig format to specify the remote address of @@ -295,7 +296,7 @@ plugin which supports full-text search and analytics. ## Legacy Audit -__Note:__ Legacy Audit is deprecated and is disabled by defaule since Kubernetes 1.8. +__Note:__ Legacy Audit is deprecated and is disabled by default since Kubernetes 1.8. To fallback to this legacy audit, disable the advanced auditing feature using the `AdvancedAuditing` feature gate in [kube-apiserver][kube-apiserver]: From 4ed86a0bb5da67ea6ee6cfa4ef5c0ca0a11df732 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Mon, 4 Dec 2017 14:58:46 +0100 Subject: [PATCH 3/5] Address review comments Signed-off-by: Mik Vyatskov --- docs/tasks/debug-application-cluster/audit.md | 89 +++---------------- 1 file changed, 11 insertions(+), 78 deletions(-) diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index c015b967f93ea..e07d8ca690c6b 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -9,6 +9,8 @@ title: Auditing * TOC {:toc} +{% include feature-state-beta.md %} + Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. It allows cluster administrator to @@ -38,81 +40,12 @@ of the event. The audit policy object structure is defined in the You can pass a file with the policy to [kube-apiserver][kube-apiserver] using the `--audit-policy-file` flag. If the flag is omitted, no events are logged. __Note:__ `kind` and `apiVersion` fields along with `rules` __must__ be provided -in the audit policy file. A policy with 0 rules, or a policy that doesn't +in the audit policy file. A policy with no (0) rules, or a policy that doesn't provide valid `apiVersion` and `kind` values is treated as illegal. Some example audit policy files: -```yaml -apiVersion: audit.k8s.io/v1beta1 # This is required. -kind: Policy -# Don't generate audit events for all requests in RequestReceived stage. -omitStages: - - "RequestReceived" -rules: - # Log pod changes at RequestResponse level - - level: RequestResponse - resources: - - group: "" - # Resource "pods" doesn't match requests to any subresource of pods, - # which is consistent with the RBAC policy. - resources: ["pods"] - # Log "pods/log", "pods/status" at Metadata level - - level: Metadata - resources: - - group: "" - resources: ["pods/log", "pods/status"] - - # Don't log requests to a configmap called "controller-leader" - - level: None - resources: - - group: "" - resources: ["configmaps"] - resourceNames: ["controller-leader"] - - # Don't log watch requests by the "system:kube-proxy" on endpoints or services - - level: None - users: ["system:kube-proxy"] - verbs: ["watch"] - resources: - - group: "" # core API group - resources: ["endpoints", "services"] - - # Don't log authenticated requests to certain non-resource URL paths. - - level: None - userGroups: ["system:authenticated"] - nonResourceURLs: - - "/api*" # Wildcard matching. - - "/version" - - # Log the request body of configmap changes in kube-system. - - level: Request - resources: - - group: "" # core API group - resources: ["configmaps"] - # This rule only applies to resources in the "kube-system" namespace. - # The empty string "" can be used to select non-namespaced resources. - namespaces: ["kube-system"] - - # Log configmap and secret changes in all other namespaces at the Metadata level. - - level: Metadata - resources: - - group: "" # core API group - resources: ["secrets", "configmaps"] - - # Log all other resources in core and extensions at the Request level. - - level: Request - resources: - - group: "" # core API group - - group: "extensions" # Version of group should NOT be included. - - # A catch-all rule to log all other requests at the Metadata level. - - level: Metadata - # Long-running requests like watches that fall under this rule will not - # generate an audit event in RequestReceived. - omitStages: - - "RequestReceived" -``` +{% include code.html language="yaml" file="policy.yaml" ghlink="/docs/tasks/debug-application-cluster/audit-policy.yaml" %} You can use a minimal audit policy file to log all requests at the `Metadata` level: @@ -132,8 +65,8 @@ admins constructing their own audit profiles. Audit backends implement exporting audit events to an external storage. [Kube-apiserver][kube-apiserver] out of the box provides two backends: -- Log backend, which writes events to a disk. -- Webhook backend, which sends events to an external API. +- Log backend, which writes events to a disk +- Webhook backend, which sends events to an external API In both cases, audit events structure is defined by the API in the `audit.k8s.io` API group. The current version of the API is @@ -146,9 +79,9 @@ log audit backend using the following [kube-apiserver][kube-apiserver] flags: - `--audit-log-path` specifies the log file path that log backend uses to write audit events. Not specifying this flag disables log backend. `-` means standard out -- `--audit-log-maxage` defined the maximum number of days to retain old audit log files. -- `--audit-log-maxbackup` defines the maximum number of audit log files to retain. -- `--audit-log-maxsize` defines the maximum size in megabytes of the audit log file before it gets rotated. +- `--audit-log-maxage` defined the maximum number of days to retain old audit log files +- `--audit-log-maxbackup` defines the maximum number of audit log files to retain +- `--audit-log-maxsize` defines the maximum size in megabytes of the audit log file before it gets rotated ### Webhook backend @@ -160,8 +93,8 @@ audit backend using the following kube-apiserver flags: configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig]. - `--audit-webhook-mode` define the buffering strategy, one of the following: - `batch` - buffer events and asynchronously send the set of events to the external service. - This is the default. - - `blocking` - block API server responses on sending each event to the external service. + This is the default + - `blocking` - block API server responses on sending each event to the external service The webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it. From 0a82e2d53d9cf7144a51422b0db6338729122611 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Tue, 5 Dec 2017 19:41:01 +0100 Subject: [PATCH 4/5] Address review comments Signed-off-by: Mik Vyatskov --- .../audit-policy.yaml | 68 +++++++++++++++++++ docs/tasks/debug-application-cluster/audit.md | 2 +- 2 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 docs/tasks/debug-application-cluster/audit-policy.yaml diff --git a/docs/tasks/debug-application-cluster/audit-policy.yaml b/docs/tasks/debug-application-cluster/audit-policy.yaml new file mode 100644 index 0000000000000..b5d66b3a18f28 --- /dev/null +++ b/docs/tasks/debug-application-cluster/audit-policy.yaml @@ -0,0 +1,68 @@ +apiVersion: audit.k8s.io/v1beta1 # This is required. +kind: Policy +# Don't generate audit events for all requests in RequestReceived stage. +omitStages: + - "RequestReceived" +rules: + # Log pod changes at RequestResponse level + - level: RequestResponse + resources: + - group: "" + # Resource "pods" doesn't match requests to any subresource of pods, + # which is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] + + # Don't log requests to a configmap called "controller-leader" + - level: None + resources: + - group: "" + resources: ["configmaps"] + resourceNames: ["controller-leader"] + + # Don't log watch requests by the "system:kube-proxy" on endpoints or services + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core API group + resources: ["endpoints", "services"] + + # Don't log authenticated requests to certain non-resource URL paths. + - level: None + userGroups: ["system:authenticated"] + nonResourceURLs: + - "/api*" # Wildcard matching. + - "/version" + + # Log the request body of configmap changes in kube-system. + - level: Request + resources: + - group: "" # core API group + resources: ["configmaps"] + # This rule only applies to resources in the "kube-system" namespace. + # The empty string "" can be used to select non-namespaced resources. + namespaces: ["kube-system"] + + # Log configmap and secret changes in all other namespaces at the Metadata level. + - level: Metadata + resources: + - group: "" # core API group + resources: ["secrets", "configmaps"] + + # Log all other resources in core and extensions at the Request level. + - level: Request + resources: + - group: "" # core API group + - group: "extensions" # Version of group should NOT be included. + + # A catch-all rule to log all other requests at the Metadata level. + - level: Metadata + # Long-running requests like watches that fall under this rule will not + # generate an audit event in RequestReceived. + omitStages: + - "RequestReceived" \ No newline at end of file diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index e07d8ca690c6b..b05dcbe130467 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -92,7 +92,7 @@ audit backend using the following kube-apiserver flags: - `--audit-webhook-config-file` specifies the path to a file with a webhook configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig]. - `--audit-webhook-mode` define the buffering strategy, one of the following: - - `batch` - buffer events and asynchronously send the set of events to the external service. + - `batch` - buffer events and asynchronously send the set of events to the external service This is the default - `blocking` - block API server responses on sending each event to the external service From ff1f784d3b8a0a10de47461142fb32b294bc69ae Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Thu, 7 Dec 2017 11:52:02 +0100 Subject: [PATCH 5/5] Fix broken link Signed-off-by: Mik Vyatskov --- docs/tasks/debug-application-cluster/audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index b05dcbe130467..b133b5928c69b 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -45,7 +45,7 @@ provide valid `apiVersion` and `kind` values is treated as illegal. Some example audit policy files: -{% include code.html language="yaml" file="policy.yaml" ghlink="/docs/tasks/debug-application-cluster/audit-policy.yaml" %} +{% include code.html language="yaml" file="audit-policy.yaml" ghlink="/docs/tasks/debug-application-cluster/audit-policy.yaml" %} You can use a minimal audit policy file to log all requests at the `Metadata` level: