Skip to content

Commit

Permalink
Deploy custom policy as part of cluster deployment (#345)
Browse files Browse the repository at this point in the history
* Add custom policy example.

* Fix indentation issues, align with other conventions.

* Typo fix.

* Typo / language fixes and improvements, updated URLs.

Co-authored-by: Chad Kittel <chad.kittel@gmail.com>

* Remove policyName as parameter; fix typo in output names.

Co-authored-by: Chad Kittel <chad.kittel@gmail.com>

* Remove obsolete parameter declaration.

* Other typo fix.

* Fix: policy name rather then policy id as param to guid(.).

* Hint on Base64-based policy implementation.

Co-authored-by: Chad Kittel <chad.kittel@gmail.com>
  • Loading branch information
ulkeba and ckittel authored Sep 19, 2022
1 parent 73d9b1f commit 5d97ee0
Show file tree
Hide file tree
Showing 6 changed files with 240 additions and 8 deletions.
1 change: 1 addition & 0 deletions 01-prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This is the starting point for the instructions on deploying the [AKS Baseline r
>
> * [Contributor role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#contributor) is _required_ at the subscription level to have the ability to create resource groups and perform deployments.
> * [User Access Administrator role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) is _required_ at the subscription level since you'll be performing role assignments to managed identities across various resource groups.
> * [Resource Policy Contributor role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#resource-policy-contributor) is _required_ at the subscription level since you'll be creating custom Azure policy definitions to govern resources in your AKS cluster.
1. An Azure AD tenant to associate your Kubernetes RBAC Cluster API authentication to.

Expand Down
17 changes: 10 additions & 7 deletions 08-workload-prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping

## Check Azure Policies are in place

> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliance with the organization's security best practices. Moreover [data is generated by Azure Policy](https://learn.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://learn.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects.
> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliant with the organization's security best practices. Moreover [data is generated by Azure Policy](https://learn.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://learn.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects.
>
> Beyond that, internal governance requires the team to ensure that any public endpoint is exposed through a full-qualified domain name ends with a company-owned domain suffix. To enforce this for all endpoints exposed by the cluster's ingress controller, they define a custom policy using [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/) and leverage the capability to [deploy it via Azure Policy](https://learn.microsoft.com/azure/aks/use-azure-policy#create-and-assign-a-custom-policy-definition) to their cluster.
1. Confirm policies are applied to the AKS cluster

Expand All @@ -58,12 +60,13 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping
A similar output as the one showed below should be returned

```output
NAME AGE
k8sazureallowedcapabilities 21m
k8sazureallowedseccomp 21m
… more …
k8sazurereadonlyrootfilesystem 21m
k8sazurevolumetypes 21m
NAME AGE
k8sazureallowedcapabilities 21m
k8sazureallowedseccomp 21m
… more …
k8sazurereadonlyrootfilesystem 21m
k8sazurevolumetypes 21m
k8scustomingresstlshostshavedefineddomainsuffix 21m
```

### Save your work in-progress
Expand Down
43 changes: 43 additions & 0 deletions 11-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,49 @@ If instead Kubernetes RBAC is backed directly by Azure AD, then you'll need to e

No matter which backing store you use, the user assigned to the group will then be able to `az aks get-credentials` to the cluster and you can validate that user is limited to a _read only_ view of the a0008 namespace.

## Validate Azure Policy

Built-in as well as custom policies are applied to the cluster as part of the [cluster deployment step](./06-aks-cluster.md) to ensure that workloads deployed to the cluster comply with the team's governance rules. Policy assignments with effect [`audit`](https://learn.microsoft.com/azure/governance/policy/concepts/effects#audit) will create a warning in the activity log and show violations in the Azure Policy blade in the portal, providing an aggregated view of the compliance state and the option to identify violating resources. Policy assignments with effect [`deny`](https://learn.microsoft.com/azure/governance/policy/concepts/effects#deny) will be enforced with the help of [Gatekeeper's admission controller webhook](https://open-policy-agent.github.io/gatekeeper/website/docs/) by denying API requests that would violate a policy otherwise.

:bulb: Gatekeeper policies are implemented in the [policy language 'Rego'](https://learn.microsoft.com/azure/governance/policy/concepts/policy-for-kubernetes#policy-language). To deploy the policy of this reference architecture with the Azure platform, the Rego specification is Base64-encoded and stored in a field of the Azure Policy resource defined in `nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep`. It might be insightful to decode the string with an Base64 decoder of your choice and investigate the declarative implementation.

### Steps

1. Try to add a second `Ingress` resource to your workload namespace with the following command.

Notice that the host value specified in the `rules` and the `tls` sections defines a domain name with suffix `invalid-domain.com` rather than the domain suffix you defined for your setup when you [created your certificates](./02-ca-certificates.md)).

```bash
cat <<EOF | kubectl create -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: aspnetapp-ingress-violating
namespace: a0008
spec:
tls:
- hosts:
- bu0001a0008-00.aks-ingress.invalid-domain.com
rules:
- host: bu0001a0008-00.aks-ingress.invalid-domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: aspnetapp-service
port:
number: 80
EOF
```
2. Inspect the error message and remark that Gatekeeper's admission webhook rejects `bu0001a0008-00.aks-ingress.invalid-domain.com` as incompliant host.
```output
Error from server (Forbidden): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8scustomingresstlshostshavede-e64871e795ce3239cd99] TLS host must have one of defined domain suffixes. Valid domain names are ["contoso.com"]; defined TLS hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}; incompliant hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}.
```
## Validate Web Application Firewall functionality
Your workload is placed behind a Web Application Firewall (WAF), which has rules designed to stop intentionally malicious activity. You can test this by triggering one of the built-in rules with a request that looks malicious.
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,6 @@ Kubernetes and, by extension, AKS are fast-evolving products. The [AKS roadmap](
This implementation will not include every preview feature, but instead only those that add significant value to a general-purpose cluster. There are some additional preview features you may wish to evaluate in pre-production clusters that augment your posture around security, manageability, etc. As these features come out of preview, this reference implementation may be updated to incorporate them. Consider trying out and providing feedback on the following:

- [BYO Kubelet Identity](https://learn.microsoft.com/azure/aks/use-managed-identity#bring-your-own-kubelet-mi)
- [Custom Azure Policy for Kubernetes support](https://techcommunity.microsoft.com/t5/azure-governance-and-management/azure-policy-for-kubernetes-releases-support-for-custom-policy/ba-p/2699466)
- [Planned maintenance window](https://learn.microsoft.com/azure/aks/planned-maintenance)
- [BYO CNI (`--network-plugin none`)](https://learn.microsoft.com/azure/aks/use-byo-cni)
- [Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on](https://learn.microsoft.com/azure/aks/keda)
Expand Down
33 changes: 33 additions & 0 deletions cluster-stamp.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -1411,6 +1411,38 @@ resource paManagedIdentitiesEnabled 'Microsoft.Authorization/policyAssignments@2
}
}

// Deploying and applying the custom policy 'Kubernetes cluster ingress TLS hosts must have defined domain suffix' as defined in nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep
// Note: Policy definition must be deployed as module since policy definitions require a targetScope of 'subscription'.

module modK8sIngressTlsHostsHaveDefinedDomainSuffix 'nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep' = {
name: 'modK8sIngressTlsHostsHaveDefinedDomainSuffix'
scope: subscription()
}

resource paK8sIngressTlsHostsHaveSpecificDomainSuffix 'Microsoft.Authorization/policyAssignments@2021-06-01' = {
name: guid('K8sCustomIngressTlsHostsHaveDefinedDomainSuffix', resourceGroup().id, clusterName)
location: 'global'
scope: resourceGroup()
properties: {
displayName: take('[${clusterName}] ${modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyName}', 120)
description: modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyDescription
policyDefinitionId: modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyId
parameters: {
excludedNamespaces: {
value: []
}
effect: {
value: 'deny'
}
allowedDomainSuffixes: {
value: [
domainName
]
}
}
}
}

// The control plane identity used by the cluster. Used for networking access (VNET joining and DNS updating)
resource miClusterControlPlane 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
name: 'mi-${clusterName}-controlplane'
Expand Down Expand Up @@ -1817,6 +1849,7 @@ resource mc 'Microsoft.ContainerService/managedClusters@2022-03-02-preview' = {
paEnforceInternalLoadBalancers
paEnforceResourceLimits
paRoRootFilesystem
paK8sIngressTlsHostsHaveSpecificDomainSuffix

// Azure Resource Provider policies that we'd like to see in place before the cluster is deployed
// They are not technically a dependency, but logically they would have existed on the resource group
Expand Down
153 changes: 153 additions & 0 deletions nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
targetScope = 'subscription'

/*** RESOURCES ***/

resource pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix 'Microsoft.Authorization/policyDefinitions@2021-06-01' = {
scope: subscription()
name: 'K8sCustomIngressTlsHostsHaveDefinedDomainSuffix'
properties: {
policyType: 'Custom'
mode: 'Microsoft.Kubernetes.Data'
displayName: 'Kubernetes cluster ingress TLS hosts must have defined domain suffix'
description: 'Kubernetes cluster ingress TLS hosts must have defined domain suffix'
policyRule: {
if: {
field: 'type'
in: [
'Microsoft.ContainerService/managedClusters'
]
}
then: {
effect: '[parameters(\'effect\')]'
details: {
templateInfo: {
sourceType: 'Base64Encoded'
content: 'YXBpVmVyc2lvbjogdGVtcGxhdGVzLmdhdGVrZWVwZXIuc2gvdjFiZXRhMQpraW5kOiBDb25zdHJhaW50VGVtcGxhdGUKbWV0YWRhdGE6CiAgbmFtZTogazhzY3VzdG9taW5ncmVzc3Rsc2hvc3RzaGF2ZWRlZmluZWRkb21haW5zdWZmaXgKc3BlYzoKICBjcmQ6CiAgICBzcGVjOgogICAgICBuYW1lczoKICAgICAgICBraW5kOiBrOHNjdXN0b21pbmdyZXNzdGxzaG9zdHNoYXZlZGVmaW5lZGRvbWFpbnN1ZmZpeAogICAgICB2YWxpZGF0aW9uOgogICAgICAgIG9wZW5BUElWM1NjaGVtYToKICAgICAgICAgIHR5cGU6IG9iamVjdAogICAgICAgICAgcHJvcGVydGllczoKICAgICAgICAgICAgYWxsb3dlZERvbWFpblN1ZmZpeGVzOgogICAgICAgICAgICAgIHR5cGU6IGFycmF5CiAgICAgICAgICAgICAgaXRlbXM6CiAgICAgICAgICAgICAgICB0eXBlOiBzdHJpbmcKICB0YXJnZXRzOgogICAgLSB0YXJnZXQ6IGFkbWlzc2lvbi5rOHMuZ2F0ZWtlZXBlci5zaAogICAgICByZWdvOiB8CiAgICAgICAgcGFja2FnZSBrOHNjdXN0b21pbmdyZXNzdGxzaG9zdHNoYXZlZGVmaW5lZGRvbWFpbnN1ZmZpeAoKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XSB7CiAgICAgICAgICBhbGxvd2VkRG9tYWluU3VmZml4ZXMgOj0gaW5wdXQucGFyYW1ldGVycy5hbGxvd2VkRG9tYWluU3VmZml4ZXMKICAgICAgICAgIGFsbERlZmluZWRIb3N0cyA6PSB7biB8IG4gOj0gaW5wdXQucmV2aWV3Lm9iamVjdC5zcGVjLnRsc1tfXS5ob3N0c1tfXX0KICAgICAgICAgIG1hdGNoZWRIb3N0cyA6PSB7IHsiaG9zdCI6IGgsICJtYXRjaGVkRG9tYWlucyI6IGR9IHwKICAgICAgICAgICAgaCA6PSBhbGxEZWZpbmVkSG9zdHNbX10gOwogICAgICAgICAgICBkIDo9IHsgbiB8IG4gOj0gYWxsb3dlZERvbWFpblN1ZmZpeGVzW19dIDsgZW5kc3dpdGgoaCwgbil9CiAgICAgICAgICB9CiAgICAgICAgICB1bm1hdGNoZWRIb3N0cyA6PSB7IGggfAogICAgICAgICAgICBoIDo9IG1hdGNoZWRIb3N0c1t4XS5ob3N0IDsgCiAgICAgICAgICAgIGQgOj0gbWF0Y2hlZEhvc3RzW3hdLm1hdGNoZWREb21haW5zIDsgCiAgICAgICAgICAgIGNvdW50KGQpID09IDAKICAgICAgICAgIH0KICAgICAgICAgIGNvdW50KHVubWF0Y2hlZEhvc3RzKSA+IDAKICAgICAgICAgIG1zZyA6PSBzcHJpbnRmKCJUTFMgaG9zdCBtdXN0IGhhdmUgb25lIG9mIGRlZmluZWQgZG9tYWluIHN1ZmZpeGVzLiBWYWxpZCBkb21haW4gbmFtZXMgYXJlICV2OyBkZWZpbmVkIFRMUyBob3N0cyBhcmUgJXY7IGluY29tcGxpYW50IGhvc3RzIGFyZSAldi4iLCBbYWxsb3dlZERvbWFpblN1ZmZpeGVzLCBhbGxEZWZpbmVkSG9zdHMsIHVubWF0Y2hlZEhvc3RzXSkKICAgICAgICB9'
}
apiGroups: [
'networking.k8s.io'
]
kinds: [
'Ingress'
]
namespaces: '[parameters(\'namespaces\')]'
excludedNamespaces: '[parameters(\'excludedNamespaces\')]'
labelSelector: '[parameters(\'labelSelector\')]'
values: {
allowedDomainSuffixes: '[parameters(\'allowedDomainSuffixes\')]'
}
}
}
}
parameters: {
effect: {
type: 'String'
metadata: {
displayName: 'Effect'
description: '\'audit\' allows a non-compliant resource to be created or updated, but flags it as non-compliant. \'deny\' blocks the non-compliant resource creation or update. \'disabled\' turns off the policy.'
}
allowedValues: [
'audit'
'Audit'
'deny'
'Deny'
'disabled'
'Disabled'
]
defaultValue: 'audit'
}
excludedNamespaces: {
type: 'Array'
metadata: {
displayName: 'Namespace exclusions'
description: 'List of Kubernetes namespaces to exclude from policy evaluation.'
}
defaultValue: [
'kube-system'
'gatekeeper-system'
'azure-arc'
]
}
namespaces: {
type: 'Array'
metadata: {
displayName: 'Namespace inclusions'
description: 'List of Kubernetes namespaces to only include in policy evaluation. An empty list means the policy is applied to all resources in all namespaces.'
}
defaultValue: []
}
labelSelector: {
type: 'Object'
metadata: {
displayName: 'Kubernetes label selector'
description: 'Label query to select Kubernetes resources for policy evaluation. An empty label selector matches all Kubernetes resources.'
}
defaultValue: {
}
schema: {
description: 'A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all resources.'
type: 'object'
properties: {
matchLabels: {
description: 'matchLabels is a map of {key,value} pairs.'
type: 'object'
additionalProperties: {
type: 'string'
}
minProperties: 1
}
matchExpressions: {
description: 'matchExpressions is a list of values, a key, and an operator.'
type: 'array'
items: {
type: 'object'
properties: {
key: {
description: 'key is the label key that the selector applies to.'
type: 'string'
}
operator: {
description: 'operator represents a key\'s relationship to a set of values.'
type: 'string'
enum: [
'In'
'NotIn'
'Exists'
'DoesNotExist'
]
}
values: {
description: 'values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty.'
type: 'array'
items: {
type: 'string'
}
}
}
required: [
'key'
'operator'
]
additionalProperties: false
}
minItems: 1
}
}
additionalProperties: false
}
}
allowedDomainSuffixes: {
type: 'Array'
metadata: {
displayName: 'List of compliant domain suffixes'
description: 'List of compliant domain suffixes'
}
}
}
}
}


output policyId string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.id
output policyName string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.properties.displayName
output policyDescription string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.properties.description

0 comments on commit 5d97ee0

Please sign in to comment.