Documentation for Windows GMSA feature

content/en/docs/getting-started-guides/windows/_index.md

@@ -297,6 +297,7 @@ This example assumes you are running on Windows Server 1709, so uses the image t
 ### Secrets and ConfigMaps
 Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be used as environment variables. See limitations section below for additional details.
 
+
 **Examples:**
 
 Windows pod with secrets mapped to environment variables
@@ -378,6 +379,254 @@ PS > Get-Service kubelet; Get-Service kube-proxy;
 CMD > sc.exe queryex kubelet && sc qc kubelet && sc.exe queryex kube-proxy && sc.exe qc kube-proxy
 ```
 
+### Group Managed Service Accounts (GMSA) support
+
+{{< feature-state for_k8s_version="v1.14" state="alpha" >}}
+
+Starting with Kubernetes v1.14, Windows container workloads can be configured to use [Group Managed Service Accounts](https://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview) (GMSA). Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.
+
+In Kubernetes, GMSA credential specs are configured at a Kubernetes cluster-wide scope as custom resources. Windows pods, as well as individual containers within a pod, can be configured to use a GMSA for domain based functions (e.g. Kerberos authentication) when interacting with other Windows services. As of v1.14, the only container runtime interface that supports GMSA for Windows workloads is Dockershim. Implementation of GMSA through CRI and other runtimes is planned for the future.
+
+#### Setup
+As an alpha feature, there are some additional setup steps that must be taken in order to use the GMSA feature:
+1. Enable the `WindowsGMSA` feature gate on kubelet on Windows nodes.
+2. Install the GMSACredentialSpec CRD
+3. Create GMSA credential spec resources
+4. Install webhooks to expand and validate references to GMSA credential spec resources from pod specs
+5. Create cluster roles to allow service accounts to use GMSA credential spec resources 
+6. Configure pods to use GMSA credential specs along with a service account authorized to use the GMSA credential specs
+
+##### Enable the WindowsGMSA feature gate
+In the alpha state, the `WindowsGMSA` feature gate needs to be enabled on kubelet on Windows nodes. This is required to pass down the GMSA credential specs from the cluster scoped configurations to the container runtime. See [Feature Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) for an explanation of enabling feature gates. 
+
+##### Install the GMSACredentialSpec CRD
+A [CustomResourceDefinition](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/) (CRD) for GMSA credential spec resources needs to be configured on the cluster to define the custom resource type `GMSACredentialSpec`. Download the GMSA CRD [YAML](https://github.com/kubernetes-sigs/windows-gmsa/blob/master/admission-webhook/deploy/gmsa-webhook.yml.tpl#L131-L148) and save it as gmsa-crd.yaml.
+Next, install the CRD with `kubectl apply -f gmsa-crd.yaml`
+
+##### Create GMSA credspec resources
+With the GMSACredentialSpec CRD installed, GMSA credspec custom resources can now be configured. The GMSA credential spec does not contain secret or sensitive data. It is information that a container runtime can use to describe the desired GMSA of a container to Windows. The GMSA credspec resources can be generated in JSON format with a utility [PowerShell][] script. Following are the steps for generating a GMSA credspec YAML based on the JSON:
+1. Import the CredentialSpec module: `ipmo CredentialSpec.psm1`
+2. Create a credential spec in JSON format using `New-CredentialSpec`. To create a GMSA credspec named WebApp1: `New-CredentialSpec -Name WebApp1 -AccountName WebApp1 -Domain $(Get-ADDomain -Current LocalComputer)`
+3. Use `Get-CredentialSpec` to show the path of the JSON file. 
+4. Convert the credspec file from JSON to YAML format and apply the necessary header fields `apiVersion`, `kind`, `metadata` and `credspec` to make it a GMSACredentialSpec custom resource. An example based on a GMSA credspec named WebApp1 is below:
+
+```
+apiVersion: windows.k8s.io/v1alpha1
+kind: GMSACredentialSpec
+metadata:
+  name: gmsa-WebApp1  #This is an arbitrary name but it will be used as a reference
+credspec:
+  ActiveDirectoryConfig:
+    GroupManagedServiceAccounts:
+    - Name: WebApp1   #Username of the GMSA account
+      Scope: CONTOSO  #NETBIOS Domain Name
+    - Name: WebApp1   #Username of the GMSA account
+      Scope: contoso.com #DNS Domain Name
+  CmsPlugins:
+  - ActiveDirectory
+  DomainJoinConfig:
+    DnsName: contoso.com  #DNS Domain Name
+    DnsTreeName: contoso.com #DNS Domain Name Root
+    Guid: 244818ae-87ac-4fcd-92ec-e79e5252348a  #GUID
+    MachineAccountName: WebApp1 #Username of the GMSA account
+    NetBiosName: CONTOSO  #NETBIOS Domain Name
+    Sid: S-1-5-21-2126449477-2524075714-3094792973 #SID of GMSA
+```
+
+5. Deploy the credential spec resource: `kubectl apply -f gmsa-Webapp1-credspec.yml`
+
+[PowerShell]: https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/windows-server-container-tools/ServiceAccounts/CredentialSpec.psm1
+
+#### Install Webhooks for GMSA
+Two webhooks need to be configured on the Kubernetes cluster to populate and validate GMSA credential spec references at the pod or container level. The mutating webhook expands references to GMSAs (by name from a pod specification) into the full credential spec in JSON form within the pod spec. The validating webhook ensures all references to GMSAs are authorized to be used by the pod service account. 
+
+Installing the webhooks require several steps: First, create a certificate key pair (that will be used to allow the webhook container to communicate to the cluster) and install a corresponding secret. Next, create a deployment for the core webhook logic. Finally, create the validating and mutating webhook configurations referring to the deployment. The YAML file below can be used to configure the webhooks and the associated deployment and secret. Replace the values in curly braces {} with desired values, save it as `gmsa-webhooks.yml` and apply using `kubectl apply -f gmsa-webhooks.yml`
+
+```
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${DEPLOYMENT_NAME}
+  namespace: ${NAMESPACE}
+data:
+  tls_private_key: ${TLS_PRIVATE_KEY}
+  tls_certificate: ${TLS_CERTIFICATE}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ${DEPLOYMENT_NAME}
+  namespace: ${NAMESPACE}
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ${DEPLOYMENT_NAME}
+  template:
+    metadata:
+      labels:
+        app: ${DEPLOYMENT_NAME}
+    spec:
+      containers:
+      - name: ${DEPLOYMENT_NAME}
+        image: k8ssigwindows/gmsa-admission-webhook
+        imagePullPolicy: IfNotPresent
+        ports:
+        - containerPort: 443
+        volumeMounts:
+          - name: tls
+            mountPath: "/tls"
+            readOnly: true
+        env:
+          - name: TLS_KEY
+            value: /tls/key
+          - name: TLS_CRT
+            value: /tls/crt
+      volumes:
+      - name: tls
+        secret:
+          secretName: ${DEPLOYMENT_NAME}
+          items:
+          - key: tls_private_key
+            path: key
+          - key: tls_certificate
+            path: crt
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ${DEPLOYMENT_NAME}
+  namespace: ${NAMESPACE}
+spec:
+  ports:
+  - port: 443
+    targetPort: 443
+  selector:
+    app: ${DEPLOYMENT_NAME}
+---
+# add a label to the deployment's namespace so that we can exclude it
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ${NAMESPACE}
+  labels:
+    gmsa-webhook: disabled
+---
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: ${DEPLOYMENT_NAME}
+webhooks:
+- name: k8s-gmsa-admission-webhook.sig-windows.k8s.io
+  clientConfig:
+    service:
+      name: ${DEPLOYMENT_NAME}
+      namespace: ${NAMESPACE}
+      path: "/validate"
+    caBundle: ${CA_BUNDLE}
+  rules:
+  - operations: ["CREATE", "UPDATE"]
+    apiGroups: [""]
+    apiVersions: ["*"]
+    resources: ["pods"]
+  failurePolicy: Fail
+  # don't run on ${NAMESPACE}
+  namespaceSelector:
+    matchExpressions:
+      - key: gmsa-webhook
+        operator: NotIn
+        values: [disabled]
+---
+apiVersion: admissionregistration.k8s.io/v1beta1
+kind: MutatingWebhookConfiguration
+metadata:
+  name: ${DEPLOYMENT_NAME}
+webhooks:
+- name: k8s-gmsa-admission-webhook.sig-windows.k8s.io
+  clientConfig:
+    service:
+      name: ${DEPLOYMENT_NAME}
+      namespace: ${NAMESPACE}
+      path: "/mutate"
+    caBundle: ${CA_BUNDLE}
+  rules:
+  - operations: ["CREATE"]
+    apiGroups: [""]
+    apiVersions: ["*"]
+    resources: ["pods"]
+  failurePolicy: Fail
+  # don't run on ${NAMESPACE}
+  namespaceSelector:
+    matchExpressions:
+    - key: gmsa-webhook
+      operator: NotIn
+      values: [disabled]
+```
+
+##### Configure cluster role and RBAC for service accounts to use GMSA credential specs
+A service account that a pod is configured with needs to be authorized for the "use" verb on the desired GMSA credential spec resources. The following is a template using the default service account demonstrating the cluster role and role binding to use a GMSA credential spec from above. Save the file as gmsa-roles.yml and apply using `kubectl apply -f gmsa-roles.yml`
+
+```
+#Create the Role to read the credspec
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: my-rbac-reader
+rules:
+- apiGroups: ["windows.k8s.io"]
+  resources: ["gmsacredentialspecs"]
+  verbs: ["use"]
+  resourceNames: ["gmsa-WebApp1"]
+---
+#Assign the role to the service account
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: allow-default-svc-account-read-on-gmsa-WebApp1
+  namespace: default
+subjects:
+- kind: ServiceAccount
+  name: default
+  namespace: default
+roleRef:
+  kind: ClusterRole
+  name: my-rbac-reader
+  apiGroup: rbac.authorization.k8s.io
+```
+
+##### Configure GMSA credential spec reference in pod spec
+In the alpha stage of the feature, the annotation `pod.alpha.windows.kubernetes.io/gmsa-credential-spec-name` is used to specify references to desired GMSA credential spec custom resources from pod specs. A sample pod spec with the annotation populated:
+
+```
+apiVersion: apps/v1beta1
+kind: Deployment
+metadata:
+  labels:
+    run: with-creds
+  name: with-creds
+  namespace: default
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      run: with-creds
+  template:
+    metadata:
+      creationTimestamp: null
+      labels:
+        run: with-creds
+      annotations:
+        pod.alpha.windows.kubernetes.io/gmsa-credential-spec-name: gmsa-WebApp1
+    spec:
+      containers:
+      - image: xxxxx/test-ad-loop
+        imagePullPolicy: Always
+        name: with-creds
+      nodeSelector:
+        beta.kubernetes.io/os: windows
+```
+
+
 ## Known Limitations for Windows Server Containers with v1.9
 
 Some of these limitations will be addressed by the community in future releases of Kubernetes:

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -147,6 +147,7 @@ different Kubernetes components.
 | `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | - |
 | `ScheduleDaemonSetPods` | `false` | Alpha | 1.11 | 1.11 |
 | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | |
+| `WindowsGMSA` | `false` | Alpha | 1.14 | |
 
 ## Using a Feature
 
@@ -311,5 +312,6 @@ Each feature gate is designed for enabling/disabling a specific feature:
   type when used together with the `PersistentLocalVolumes` feature gate.
 - `VolumeSnapshotDataSource`: Enable volume snapshot data source support.
 - `VolumeSubpathEnvExpansion`: Enable `subPathExpr` field for expanding environment variables into a `subPath`.
+- `WindowsGMSA`: Enables passing of GMSA credential specs from pods to container runtimes.
 
 {{% /capture %}}