fix: IRSA and Pod Identity cart pod should fail on startup (#1104)

manifests/base-application/carts/deployment.yaml

@@ -44,12 +44,18 @@ spec:
             readOnlyRootFilesystem: true
             runAsNonRoot: true
             runAsUser: 1000
-          image: "public.ecr.aws/aws-containers/retail-store-sample-cart:0.7.0"
+          image: "public.ecr.aws/aws-containers/retail-store-sample-cart:0.8.0"
           imagePullPolicy: IfNotPresent
           ports:
             - name: http
               containerPort: 8080
               protocol: TCP
+          readinessProbe:
+            httpGet:
+              path: /actuator/health/readiness
+              port: 8080
+            initialDelaySeconds: 15
+            periodSeconds: 3
           livenessProbe:
             httpGet:
               path: /actuator/health/liveness

website/docs/security/amazon-eks-pod-identity/index.md

@@ -25,6 +25,6 @@ You can view the Terraform that applies these changes [here](https://github.com/
 
 :::
 
-Applications in a Pod’s containers can use a supported AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. For example, applications may need to upload files to an S3 bucket or query a DynamoDB table, and in order to do so, they must sign their AWS API requests with AWS credentials. [EKS Pod Identities](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 Instance Profiles provide credentials to instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's Role, you can associate an IAM Role with a Kubernetes Service Account and configure your Pods with it. Check out EKS documentation [here](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html) for the exact list of versions supported.
+Applications in a Pod’s containers can use a supported AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. For example, applications may need to upload files to an S3 bucket or query a DynamoDB table, and in order to do so, they must sign their AWS API requests with AWS credentials. [EKS Pod Identities](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 Instance Profiles provide credentials to instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you can associate an IAM role with a Kubernetes Service Account and configure your Pods with it. Check out EKS documentation [here](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html) for the exact list of versions supported.
 
 In this chapter we'll re-configure one of the sample application components to leverage the AWS API and provide it with the appropriate privileges.

website/docs/security/amazon-eks-pod-identity/tests/hook-enable-dynamo.sh

@@ -7,19 +7,10 @@ before() {
 after() {
   sleep 10
 
-  kubectl rollout status deployment/carts -n carts --timeout 180s
+  if [[ $TEST_OUTPUT != *"timed out waiting"* ]]; then
+    echo "Failed to match expected output"
+    echo $TEST_OUTPUT
 
-  export endpoint=$(kubectl -n kube-system get svc -n ui ui-nlb -o json | jq -r '.status.loadBalancer.ingress[0].hostname')
-
-  EXIT_CODE=0
-
-  timeout -s TERM 400 bash -c \
-    'while [[ "$(curl -s -o /dev/null -L -w ''%{http_code}'' ${endpoint}/home)" != "500" ]];\
-    do echo "Waiting for ${endpoint}" && sleep 30;\
-    done' || EXIT_CODE=$?
-
-  if [ $EXIT_CODE -ne 0 ]; then
-    >&2 echo "Load balancer did not become available or return HTTP 500 for 180 seconds"
     exit 1
   fi
 }

website/docs/security/amazon-eks-pod-identity/tests/hook-pod-logs.sh

@@ -0,0 +1,18 @@
+set -Eeuo pipefail
+
+before() {
+  echo "noop"
+}
+
+after() {
+  sleep 10
+
+  if [[ $TEST_OUTPUT != *"An error occurred when accessing Amazon DynamoDB"* ]]; then
+    echo "Failed to match expected output"
+    echo $TEST_OUTPUT
+
+    exit 1
+  fi
+}
+
+"$@"

website/docs/security/amazon-eks-pod-identity/understanding.md

@@ -5,19 +5,25 @@ sidebar_position: 33
 
 The first place to look for the issue is the logs of the `carts` service:
 
-```bash
-$ kubectl -n carts logs deployment/carts
-... ommitted output
-```
+```bash hook=pod-logs
+$ LATEST_POD=$(kubectl get pods -n carts --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].metadata.name}')
+$ kubectl logs -n carts -p $LATEST_POD
+[...]
+***************************
+APPLICATION FAILED TO START
+***************************
+
+Description:
+
+An error occurred when accessing Amazon DynamoDB:
+
+User: arn:aws:sts::1234567890:assumed-role/eksctl-eks-workshop-nodegroup-defa-NodeInstanceRole-rjjGEigUX8KZ/i-01f378b057326852a is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-west-2:1234567890:table/eks-workshop-carts/index/idx_global_customerId because no identity-based policy allows the dynamodb:Query action (Service: DynamoDb, Status Code: 400, Request ID: PUIFHHTQ7SNQVERCRJ6VHT8MBBVV4KQNSO5AEMVJF66Q9ASUAAJG)
 
-This will return a lot of logs, so lets filter it to get a succinct view of the problem:
+Action:
 
-```bash
-$ kubectl -n carts logs deployment/carts \
-  | grep DynamoDb
-User: arn:aws:sts::1234567890:assumed-role/eksctl-eks-workshop-nodegroup-def-NodeInstanceRole-P7qjC7RqXaZr/i-085482f0c0bae4f88 is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-east-1:1234567890:table/eks-workshop-carts/index/idx_global_customerId because no identity-based policy allows the dynamodb:Query action (Service: DynamoDb, Status Code: 400, Request ID: SD7IOMHAD7VL31S3M8K80A7EI3VV4KQNSO5AEMVJF66Q9ASUAAJG)
+Check that the DynamoDB table has been created and your IAM credentials are configured with the appropriate access.
 ```
 
-The application is generating an error which indicates that the IAM Role our Pod is using to access DynamoDB does not have the required permissions. This is happening because by default, if no IAM Roles or Policies are linked to our Pod, it use the IAM Role linked to the Instance Profile assigned to the EC2 instance on which its running, in this case this Role does not have an IAM Policy that allows access to DynamoDB.
+The application is generating an error which indicates that the IAM role our Pod is using to access DynamoDB does not have the required permissions. This is happening because by default, if no IAM roles or policies are linked to our Pod, it use the IAM role linked to the instance profile assigned to the EC2 instance on which its running, in this case this role does not have an IAM policy that allows access to DynamoDB.
 
-One way we could solve this is to expand the IAM permissions of our EC2 Instance Profile, but this would allow any Pod that runs on them to access our DynamoDB table which is not secure, and not a good practice of granting the principle of least privilege. Instead we'll using EKS Pod Identity to allow specific access required by the `carts` application at Pod level.
+One way we could solve this is to expand the IAM permissions of our EC2 instance profile, but this would allow any Pod that runs on them to access our DynamoDB table which is not secure, and not a good practice of granting the principle of least privilege. Instead we'll use EKS Pod Identity to allow the specific access required by the `carts` application at Pod level.

website/docs/security/amazon-eks-pod-identity/use-pod-identity.md

@@ -40,7 +40,7 @@ eks-pod-identity-agent-hslc5   1/1     Running   0          3d21h
 eks-pod-identity-agent-thvf5   1/1     Running   0          3d21h
 ```
 
-An IAM Role which provides the required permissions for the `carts` service to read and write to DynamoDB table has been created when you ran the `prepare-environment` script in the first step of this module. You can view the policy as shown below.
+An IAM role which provides the required permissions for the `carts` service to read and write to DynamoDB table has been created when you ran the `prepare-environment` script in the first step of this module. You can view the policy as shown below.
 
 ```bash
 $ aws iam get-policy-version \

website/docs/security/amazon-eks-pod-identity/using-dynamo.md

@@ -62,21 +62,22 @@ metadata:
   namespace: carts
 ```
 
-We'll need to recycle all the Pods of the `carts` application to pick up our new ConfigMap contents.
+Now we need to recycle all the carts Pods to pick up our new ConfigMap contents:
 
-```bash hook=enable-dynamo hookTimeout=430
-$ kubectl -n carts rollout restart deployment/carts
+```bash expectError=true hook=enable-dynamo
+$ kubectl rollout restart -n carts deployment/carts
 deployment.apps/carts restarted
-$ kubectl -n carts rollout status deployment/carts
+$ kubectl rollout status -n carts deployment/carts --timeout=20s
+Waiting for deployment "carts" rollout to finish: 1 old replicas are pending termination...
+error: timed out waiting for the condition
 ```
 
-So now our application should be using DynamoDB.
-
-But we run the following command we will see that the `carts` pods are failing:
+It looks like our change failed to deploy properly, we can confirm this by looking at the Pods:
 
 ```bash
 $ kubectl -n carts get pod
 NAME                              READY   STATUS             RESTARTS        AGE
+carts-5d486d7cf7-8qxf9            1/1     Running            0               5m49s
 carts-df76875ff-7jkhr             0/1     CrashLoopBackOff   3 (36s ago)     2m2s
 carts-dynamodb-698674dcc6-hw2bg   1/1     Running            0               20m
 ```

website/docs/security/amazon-eks-pod-identity/verifying-dynamo.md

@@ -3,7 +3,7 @@ title: "Verifying DynamoDB Access"
 sidebar_position: 35
 ---
 
-Now, with the `carts` Service Account associated with the authorized IAM Role, the `carts` Pod has permission to access the DynamoDB table. Access the web store again and navigate to the shopping cart.
+Now, with the `carts` Service Account associated with the authorized IAM role, the `carts` Pod has permission to access the DynamoDB table. Access the web store again and navigate to the shopping cart.
 
 ```bash
 $ kubectl -n ui get service ui-nlb -o jsonpath='{.status.loadBalancer.ingress[*].hostname}{"\n"}'

website/docs/security/iam-roles-for-service-accounts/add_irsa.md

@@ -8,7 +8,6 @@ To use IAM roles for service accounts in your cluster, an `IAM OIDC Identity Pro
 
 ```bash
 $ aws iam list-open-id-connect-providers
-
 {
     "OpenIDConnectProviderList": [
         {
@@ -22,7 +21,6 @@ Validate its association with our Amazon EKS cluster.
 
 ```bash
 $ aws eks describe-cluster --name ${EKS_CLUSTER_NAME} --query 'cluster.identity'
-
 {
     "oidc": {
         "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/7185F12D2B62B8DA97B0ECA713F66C86"
@@ -79,22 +77,22 @@ $ aws iam get-role \
 }
 ```
 
-All thats left is to re-configure the Service Account object associated with the `carts` application adding the required annotation to it, so IRSA can provide the correct authorization for Pods using the IAM Role above.
+All thats left is to re-configure the Service Account object associated with the `carts` application adding the required annotation to it, so IRSA can provide the correct authorization for Pods using the IAM role above.
 Let's validate the SA associated with the `carts` Deployment.
 
 ```bash
 $ kubectl -n carts describe deployment carts | grep 'Service Account'
   Service Account:  cart
 ```
 
-Now lets check the value of `CARTS_IAM_ROLE` which will provide the ARN of the IAM Role for the Service Account annotation.
+Now lets check the value of `CARTS_IAM_ROLE` which will provide the ARN of the IAM role for the Service Account annotation.
 
 ```bash
 $ echo $CARTS_IAM_ROLE
 arn:aws:iam::1234567890:role/eks-workshop-carts-dynamo
 ```
 
-Once we've verified the IAM Role to be used, we can run Kustomize to apply the change on the Service Account.
+Once we've verified the IAM role to be used, we can run Kustomize to apply the change on the Service Account.
 
 ```bash
 $ kubectl kustomize ~/environment/eks-workshop/modules/security/irsa/service-account \
@@ -121,4 +119,6 @@ With the ServiceAccount updated now we just need to recycle the carts Pod so it
 $ kubectl rollout restart -n carts deployment/carts
 deployment.apps/carts restarted
 $ kubectl rollout status -n carts deployment/carts
+Waiting for deployment "carts" rollout to finish: 1 old replicas are pending termination...
+deployment "carts" successfully rolled out
 ```

website/docs/security/iam-roles-for-service-accounts/index.md

@@ -24,6 +24,6 @@ You can view the Terraform that applies these changes [here](https://github.com/
 
 :::
 
-Applications in a Pod’s containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. For example, applications may need to upload files to an S3 bucket or query a DynamoDB table. To do so applications must sign their AWS API requests with AWS credentials. [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA) provide the ability to manage credentials for your applications, similar to the way that IAM Instance Profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or relying on the Amazon EC2 Instance Profile for authorization, you associate an IAM Role with a Kubernetes Service Account and configure your Pods to use that Service Account.
+Applications in a Pod’s containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. For example, applications may need to upload files to an S3 bucket or query a DynamoDB table. To do so applications must sign their AWS API requests with AWS credentials. [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA) provide the ability to manage credentials for your applications, similar to the way that IAM Instance Profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or relying on the Amazon EC2 Instance Profile for authorization, you associate an IAM role with a Kubernetes Service Account and configure your Pods to use that Service Account.
 
 In this chapter we'll re-configure one of the sample application components to leverage an AWS API and provide it with the appropriate authentication.

website/docs/security/iam-roles-for-service-accounts/tests/hook-enable-dynamo.sh

@@ -7,19 +7,10 @@ before() {
 after() {
   sleep 10
 
-  kubectl rollout status deployment/carts -n carts --timeout 180s
+  if [[ $TEST_OUTPUT != *"timed out waiting"* ]]; then
+    echo "Failed to match expected output"
+    echo $TEST_OUTPUT
 
-  export endpoint=$(kubectl -n kube-system get svc -n ui ui-nlb -o json | jq -r '.status.loadBalancer.ingress[0].hostname')
-
-  EXIT_CODE=0
-
-  timeout -s TERM 400 bash -c \
-    'while [[ "$(curl -s -o /dev/null -L -w ''%{http_code}'' ${endpoint}/home)" != "500" ]];\
-    do echo "Waiting for ${endpoint}" && sleep 30;\
-    done' || EXIT_CODE=$?
-
-  if [ $EXIT_CODE -ne 0 ]; then
-    >&2 echo "Load balancer did not become available or return HTTP 500 for 180 seconds"
     exit 1
   fi
 }

website/docs/security/iam-roles-for-service-accounts/tests/hook-pod-logs.sh

@@ -0,0 +1,18 @@
+set -Eeuo pipefail
+
+before() {
+  echo "noop"
+}
+
+after() {
+  sleep 10
+
+  if [[ $TEST_OUTPUT != *"An error occurred when accessing Amazon DynamoDB"* ]]; then
+    echo "Failed to match expected output"
+    echo $TEST_OUTPUT
+
+    exit 1
+  fi
+}
+
+"$@"

website/docs/security/iam-roles-for-service-accounts/understanding.md

@@ -5,18 +5,25 @@ sidebar_position: 23
 
 The first place to look for the issue is the logs of the `carts` service:
 
-```bash
-$ kubectl logs -n carts deployment/carts
-```
+```bash hook=pod-logs
+$ LATEST_POD=$(kubectl get pods -n carts --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].metadata.name}')
+$ kubectl logs -n carts -p $LATEST_POD
+[...]
+***************************
+APPLICATION FAILED TO START
+***************************
+
+Description:
+
+An error occurred when accessing Amazon DynamoDB:
+
+User: arn:aws:sts::1234567890:assumed-role/eksctl-eks-workshop-nodegroup-defa-NodeInstanceRole-rjjGEigUX8KZ/i-01f378b057326852a is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-west-2:1234567890:table/eks-workshop-carts/index/idx_global_customerId because no identity-based policy allows the dynamodb:Query action (Service: DynamoDb, Status Code: 400, Request ID: PUIFHHTQ7SNQVERCRJ6VHT8MBBVV4KQNSO5AEMVJF66Q9ASUAAJG)
 
-This will return a lot of logs, so lets filter it to get a succinct view of the problem:
+Action:
 
-```bash
-$ kubectl -n carts logs deployment/carts \
-  | grep DynamoDb
-User: arn:aws:sts::1234567890:assumed-role/eksctl-eks-workshop-nodegroup-def-NodeInstanceRole-P7qjC7RqXaZr/i-085482f0c0bae4f88 is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-east-1:1234567890:table/eks-workshop-carts/index/idx_global_customerId because no identity-based policy allows the dynamodb:Query action (Service: DynamoDb, Status Code: 400, Request ID: SD7IOMHAD7VL31S3M8K80A7EI3VV4KQNSO5AEMVJF66Q9ASUAAJG)
+Check that the DynamoDB table has been created and your IAM credentials are configured with the appropriate access.
 ```
 
-Our application is generating an error which indicates that the IAM Role our Pod is using to access DynamoDB does not have the required permissions. This is happening because our Pod is by default using the IAM Role assigned to the EC2 worker node on which its running, which does not have an IAM Policy that allows access to DynamoDB.
+The application is generating an error which indicates that the IAM role our Pod is using to access DynamoDB does not have the required permissions. This is happening because by default, if no IAM roles or policies are linked to our Pod, it use the IAM role linked to the instance profile assigned to the EC2 instance on which its running, in this case this role does not have an IAM policy that allows access to DynamoDB.
 
-One way we could solve this is to expand the IAM permissions of our EC2 worker nodes, but this would allow any Pod that runs on them to access our DynamoDB table which is not a good practice, and also not secure. Instead we'll using IAM Roles for Service Accounts (IRSA) to specifically allow the Pods in our `carts` service access.
+One way we could solve this is to expand the IAM permissions of our EC2 worker nodes, but this would allow any Pod that runs on them to access our DynamoDB table, but this does not reflect security best practices. Instead we'll using IAM Roles for Service Accounts (IRSA) to specifically allow the Pods in our `carts` service access.

website/docs/security/iam-roles-for-service-accounts/using-dynamo.md

@@ -53,18 +53,20 @@ metadata:
 
 Now we need to recycle all the carts Pods to pick up our new ConfigMap contents:
 
-```bash hook=enable-dynamo hookTimeout=430
+```bash expectError=true hook=enable-dynamo
 $ kubectl rollout restart -n carts deployment/carts
 deployment.apps/carts restarted
-$ kubectl rollout status -n carts deployment/carts
+$ kubectl rollout status -n carts deployment/carts --timeout=20s
+Waiting for deployment "carts" rollout to finish: 1 old replicas are pending termination...
+error: timed out waiting for the condition
 ```
 
-So now our application should be using DynamoDB.
-But if we run the following command we will see that the `carts` pods are failing:
+It looks like our change failed to deploy properly, we can confirm this by looking at the Pods:
 
 ```bash
 $ kubectl -n carts get pod
 NAME                              READY   STATUS             RESTARTS        AGE
+carts-5d486d7cf7-8qxf9            1/1     Running            0               5m49s
 carts-df76875ff-7jkhr             0/1     CrashLoopBackOff   3 (36s ago)     2m2s
 carts-dynamodb-698674dcc6-hw2bg   1/1     Running            0               20m
 ```