Add Kubernetes admission control debugging tips

Fixes #1039

Signed-off-by: Torin Sandall <torinsandall@gmail.com>

docs/content/docs/guides-kubernetes-admission-control.md

@@ -513,4 +513,142 @@ response:
 
 For more detail on how Kubernetes Admission Control works, see [this blog
 post](https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/)
-on kubernetes.io.
+on kubernetes.io.
+
+## Debugging Tips
+
+If you run into problems getting OPA to enforce admission control policies in
+Kubernetes there are a few things you can check to make sure everything is
+configured correctly. If none of these tips work, feel free to join
+[slack.openpolicyagent.org](https://slack.openpolicyagent.org) and ask for help.
+
+### Check for the `openpolicyagent.org/policy-status` annotation on ConfigMaps containing policies
+
+If you are loading policies into OPA via
+[kube-mgmt](https://github.com/open-policy-agent/kube-mgmt) you can check the
+`openpolicyagent.org/policy-status` annotation on ConfigMaps that contain your
+policies. The annotation should be set to `"ok"` if the policy was loaded
+successfully. If errors occured during loading (e.g., because the policy
+contained a syntax error) the cause will be reported here.
+
+If the annotation is
+missing entirely, check the `kube-mgmt` container logs for connection errors
+between the container and the Kubernetes API server.
+
+### Check the `kube-mgmt` container logs for error messages
+
+When `kube-mgmt` is healthy, the container logs will be quiet/empty. If you are
+trying to enforce policies based on Kubernetes context (e.g., to check for
+ingress conflicts) then you need to make sure that `kube-mgmt` can replicate
+Kubernetes objects into OPA. If `kube-mgmt` is unable to list/watch resources in
+the Kubernetes API server, they will not be replicated into OPA and the policy
+will not get enforced.
+
+### Check the `opa` container logs for TLS errors
+
+Communication between the Kubernetes API server and OPA is secured with TLS. If
+the CA bundle specified in the webhook configuration is out-of-sync with the
+server certificate that OPA is configured with, OPA will log errors indicating a
+TLS issue. Verify that the CA bundle specified in the validating or mutating
+webhook configurations matches the server certificate you configured OPA to use.
+
+### Check for POST requests in the `opa` container logs
+
+When the Kubernetes API server queries OPA for admission control decisions, it
+sends HTTP `POST` requests. If there are no `POST` requests contained in the
+`opa` container logs, it indicates that the webhook configuration is wrong or
+there is a network connectivity problem between the Kubernetes API server and
+OPA.
+
+* If you have access too the Kubernetes API server logs, review them to see if
+  they indicate the cause.
+* If you are running on AWS EKS make sure your security group settings allow
+  traffic from Kubernetes "master" nodes to the node(s) where OPA is running.
+
+### Ensure mutating policies construct JSON Patches correctly
+
+If you are using OPA to enforce mutating admission policies you must ensure the
+JSON Patch objects you generate escape "/" characters in the JSON Pointer. For
+example, if you are generating a JSON Patch that sets annotations like
+`acmecorp.com/myannotation` you need to escape the "/" character in the
+annotation name using `~1` (per [RFC
+6901](https://tools.ietf.org/html/rfc6901#section-3)).
+
+**Correct**:
+
+```json
+{
+   "op": "add",
+   "path": "/metadata/annotations/acmecorp.com~1myannotation",
+   "value": "somevalue"
+}
+```
+
+**Incorrect**:
+
+
+```json
+{
+   "op": "add",
+   "path": "/metadata/annotations/acmecorp.com/myannotation",
+   "value": "somevalue"
+}
+```
+
+In addition, when your policy generates the response for the Kubernetes API
+server, you must use the `base64.encode` built-in function to encode the JSON
+Patch objects. DO NOT use the `base64url.encode` function because the Kubernetes
+API server will not process it:
+
+**Correct**:
+
+```ruby
+main = {
+	"apiVersion": "admission.k8s.io/v1beta1",
+	"kind": "AdmissionReview",
+	"response": response,
+}
+
+response = {
+  "allowed": true,
+  "patchType": "JSONPatch",
+  "patch": base64.encode(json.marshal(patches))   # <-- GOOD: uses base64.encode
+}
+
+patches = [
+  {
+    "op": "add",
+    "path": "/metadata/annotations/acmecorp.com~1myannotation",
+    "value": "somevalue"
+  }
+]
+```
+
+**Incorrect**:
+
+```ruby
+main = {
+	"apiVersion": "admission.k8s.io/v1beta1",
+	"kind": "AdmissionReview",
+	"response": response,
+}
+
+response = {
+  "allowed": true,
+  "patchType": "JSONPatch",
+  "patch": base64url.encode(json.marshal(patches))   # <-- BAD: uses base64url.encode
+}
+
+patches = [
+  {
+    "op": "add",
+    "path": "/metadata/annotations/acmecorp.com~1myannotation",
+    "value": "somevalue"
+  }
+]
+```
+
+Also, for more examples of how to construct mutating policies and integrating
+them with validating policies, see [these
+examples](https://github.com/open-policy-agent/library/tree/master/kubernetes/mutating-admission)
+in https://github.com/open-policy-agent/library.