From 2b3d68942a2696f7c07ec87f0b984168481faa36 Mon Sep 17 00:00:00 2001 From: Torin Sandall Date: Thu, 18 Apr 2019 08:31:13 -0700 Subject: [PATCH] Add Kubernetes admission control debugging tips Fixes #1039 Signed-off-by: Torin Sandall --- .../guides-kubernetes-admission-control.md | 140 +++++++++++++++++- 1 file changed, 139 insertions(+), 1 deletion(-) diff --git a/docs/content/docs/guides-kubernetes-admission-control.md b/docs/content/docs/guides-kubernetes-admission-control.md index ff8de3e4a7..f1055d7898 100644 --- a/docs/content/docs/guides-kubernetes-admission-control.md +++ b/docs/content/docs/guides-kubernetes-admission-control.md @@ -513,4 +513,142 @@ response: For more detail on how Kubernetes Admission Control works, see [this blog post](https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) -on kubernetes.io. \ No newline at end of file +on kubernetes.io. + +## Debugging Tips + +If you run into problems getting OPA to enforce admission control policies in +Kubernetes there are a few things you can check to make sure everything is +configured correctly. If none of these tips work, feel free to join +[slack.openpolicyagent.org](https://slack.openpolicyagent.org) and ask for help. + +### Check for the `openpolicyagent.org/policy-status` annotation on ConfigMaps containing policies + +If you are loading policies into OPA via +[kube-mgmt](https://github.com/open-policy-agent/kube-mgmt) you can check the +`openpolicyagent.org/policy-status` annotation on ConfigMaps that contain your +policies. The annotation should be set to `"ok"` if the policy was loaded +successfully. If errors occured during loading (e.g., because the policy +contained a syntax error) the cause will be reported here. + +If the annotation is +missing entirely, check the `kube-mgmt` container logs for connection errors +between the container and the Kubernetes API server. + +### Check the `kube-mgmt` container logs for error messages + +When `kube-mgmt` is healthy, the container logs will be quiet/empty. If you are +trying to enforce policies based on Kubernetes context (e.g., to check for +ingress conflicts) then you need to make sure that `kube-mgmt` can replicate +Kubernetes objects into OPA. If `kube-mgmt` is unable to list/watch resources in +the Kubernetes API server, they will not be replicated into OPA and the policy +will not get enforced. + +### Check the `opa` container logs for TLS errors + +Communication between the Kubernetes API server and OPA is secured with TLS. If +the CA bundle specified in the webhook configuration is out-of-sync with the +server certificate that OPA is configured with, OPA will log errors indicating a +TLS issue. Verify that the CA bundle specified in the validating or mutating +webhook configurations matches the server certificate you configured OPA to use. + +### Check for POST requests in the `opa` container logs + +When the Kubernetes API server queries OPA for admission control decisions, it +sends HTTP `POST` requests. If there are no `POST` requests contained in the +`opa` container logs, it indicates that the webhook configuration is wrong or +there is a network connectivity problem between the Kubernetes API server and +OPA. + +* If you have access too the Kubernetes API server logs, review them to see if + they indicate the cause. +* If you are running on AWS EKS make sure your security group settings allow + traffic from Kubernetes "master" nodes to the node(s) where OPA is running. + +### Ensure mutating policies construct JSON Patches correctly + +If you are using OPA to enforce mutating admission policies you must ensure the +JSON Patch objects you generate escape "/" characters in the JSON Pointer. For +example, if you are generating a JSON Patch that sets annotations like +`acmecorp.com/myannotation` you need to escape the "/" character in the +annotation name using `~1` (per [RFC +6901](https://tools.ietf.org/html/rfc6901#section-3)). + +**Correct**: + +```json +{ + "op": "add", + "path": "/metadata/annotations/acmecorp.com~1myannotation", + "value": "somevalue" +} +``` + +**Incorrect**: + + +```json +{ + "op": "add", + "path": "/metadata/annotations/acmecorp.com/myannotation", + "value": "somevalue" +} +``` + +In addition, when your policy generates the response for the Kubernetes API +server, you must use the `base64.encode` built-in function to encode the JSON +Patch objects. DO NOT use the `base64url.encode` function because the Kubernetes +API server will not process it: + +**Correct**: + +```ruby +main = { + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": response, +} + +response = { + "allowed": true, + "patchType": "JSONPatch", + "patch": base64.encode(json.marshal(patches)) # <-- GOOD: uses base64.encode +} + +patches = [ + { + "op": "add", + "path": "/metadata/annotations/acmecorp.com~1myannotation", + "value": "somevalue" + } +] +``` + +**Incorrect**: + +```ruby +main = { + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": response, +} + +response = { + "allowed": true, + "patchType": "JSONPatch", + "patch": base64url.encode(json.marshal(patches)) # <-- BAD: uses base64url.encode +} + +patches = [ + { + "op": "add", + "path": "/metadata/annotations/acmecorp.com~1myannotation", + "value": "somevalue" + } +] +``` + +Also, for more examples of how to construct mutating policies and integrating +them with validating policies, see [these +examples](https://github.com/open-policy-agent/library/tree/master/kubernetes/mutating-admission) +in https://github.com/open-policy-agent/library. \ No newline at end of file