CHANGELOG-1.4.md

---

v1.4.8 (2020-07-20)

See code changes.

cmd

• Add cmd/ecr-utils.

eksconfig

• Make AddOnPHPApache ECR image configurable.
• Add AddOnCNIVPC.
  • https://github.com/aws/amazon-vpc-cni-k8s/tree/master/config.
  • Requires at least AddOnCNIVPC.Version v1.7.

eks

• Add eks/cni-vpc.
  • eks/cni-vpc is installed before worker nodes and not deleted.
  • Requires at least AddOnCNIVPC.Version v1.7.
• Fix [eks/hollow-nodes/remote]
  • Deploys to kubemark namespace
  • Uses a replication controller instead of deployment
  • Kubemark pods now equal corresponding hollow node names
  • Adds labels to hollow nodes for CA discovery
  • Idempotently Create/Update kubemark resources.

pkg

• Change pkg/aws/ecr.Check to return ok bool.
  • Set true if the repository exists.
• Add pkg/aws/ecr.Create.
• Add pkg/aws/ecr.Delete.

Dependency

• Upgrade github.com/aws/aws-sdk-go from v1.33.7 to v1.33.8.

Go

• Compile with Go 1.14.6.

---

v1.4.7 (2020-07-17)

See code changes.

eks

• Valid China region service principals in eks/ng and eks/mng.
• Fix eks/prometheus-grafana.

Dependency

• Upgrade github.com/aws/aws-sdk-go from v1.33.6 to v1.33.7.

Go

• Compile with Go 1.14.6.

---

v1.4.6 (2020-07-16)

See code changes.

eksconfig

• Add SkipDeleteClusterAndNodes to skip EKS cluster and nodes deletion.
  • Useful for testing add-ons with existing clusters.
  • If true, delete operation keeps all resources created before cluster (e.g. IAM role, VPC, CMK, etc.).
  • If true, delete operation keeps all node groups and managed node groups!
  • Set via AWS_K8S_TESTER_EKS_SKIP_DELETE_CLUSTER_AND_NODES=true.
  • Default AWS_K8S_TESTER_EKS_SKIP_DELETE_CLUSTER_AND_NODES=false.
  • aws-k8s-tester eks create cluster --path must be passed a valid configuration file with existing cluster information in order to use the existing cluster. Create cluster once, and delete all add-ons, and keep the cluster related fields in the YAML with all other fields removed.
  • We will have better workflow and separate abstraction for add-ons. This is a short term solution.
  • See #123 for more.
  • To delete the cluster, SkipDeleteClusterAndNodes must be set to "false" manually.
  • Create a cluster with add-ons AWS_K8S_TESTER_EKS_SKIP_DELETE_CLUSTER_AND_NODES=true aws-k8s-tester eks create cluster --auto-path and delete add-ons "only" with aws-k8s-tester eks delete cluster --path [PATH] (make sure the YAML config file is set skip-delete-cluster-and-nodes to false), and aws-k8s-tester eks create cluster --path [PATH] to test more add-ons. And repeat.
• Change AddOnHollowNodesRemote.DeploymentReplicas to ReplicationControllerReplicas.
  • AWS_K8S_TESTER_EKS_ADD_ON_HOLLOW_NODES_REMOTE_DEPLOYMENT_REPLICAS is now AWS_K8S_TESTER_EKS_ADD_ON_HOLLOW_NODES_REMOTE_REPLICATION_CONTROLLER_REPLICAS.

eks

• Use ReplicationController for eks/hollow-nodes/remote.
• Skip cluster and prerequisite resource deletion if AWS_K8S_TESTER_EKS_SKIP_DELETE_CLUSTER_AND_NODES=true.

Dependency

• Upgrade github.com/kubernetes/kubernetes from v1.18.6-rc.0 to v1.18.7-rc.0.
• Upgrade github.com/kubernetes/client-go from v0.18.6-rc.0 to v0.18.7-rc.0.
• Upgrade github.com/aws/aws-sdk-go from v1.33.5 to v1.33.6.

Go

• Compile with Go 1.14.6.

---

v1.4.5 (2020-07-14)

See code changes.

cmd

• Add cmd/s3-utils for IRSA tests.
• Add cmd/sts-utils for IRSA tests.

ec2

• Tag resources with user information.

eksconfig

• Remove AddOnIRSA.RoleManagedPolicyARNs.
• Rename SSHConfig to NodeInfo.
• AWS_K8S_TESTER_EKS_PARAMETERS_TAGS must be set in map[string]string.
  • 'a=b;c;d,e=f' should be {"a":"b","c":"d"}.

eks

• Clean up ELB resources from services in eks/wordpress,nlb-guestbook,jupyter-hub,prometheus-grafana.
• Tag resources with user information.
• Set timeouts for eks/irsa and eks/irsa-fargate S3 requests.
• Allow eks/irsa tester failures, only requires minimum 1 Pod success, debugging....
• Add kubectl logs --timestamps flags.
• Remove arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess from default eks/irsa-fargate IAM role.
• Fix eks/irsa when worker nodes have disabled SSH access (e.g. Bottlerocket OS).
• Fix typos in kubectl describe job commands for eks/*/remote testers.
• Fix wrong buckets for eks/*/remote testers.
• Add more debugging logs to remote testers for failed pods.
• Add more debugging logs to eks/irsa.
• Fix remote tester job Pod.
  • Change pod RestartPolicy from v1.RestartPolicyOnFailure to v1.RestartPolicyNever.
  • ref. kubernetes/kubernetes#54870.
• Use pkg/k8s-client.WaitForDeploymentCompletes.
• Fix eks/irsa and eks/irsa-fargate role ARN query tests.
  • Now using s3-utils.
• Use regional STS endpoint by default.
• Add iam:SetSecurityTokenServicePreferences to worker node role IAM policy.
  • ref. https://docs.aws.amazon.com/cli/latest/reference/iam/set-security-token-service-preferences.html.
  • To use regional STS endpoint.
• Fix STS regional endpoint dial timeouts.
  • e.g. "RequestError: send request failed\ncaused by: Post \"https://iam.amazonaws.com/\": dial tcp: i/o timeout"} failed to create AWS session RequestError: send request failed caused by: Post "https://sts.us-west-2.amazonaws.com/": dial tcp: i/o timeout"
• Allow eks/fluentd namespace deletion timeouts.

pkg

• Add pkg/user.
• Add pkg/aws/s3.WithTimeout.
• Fix pkg/fileutil.IsDirWriteable os.RemoveAll.
  • Fix "failed to write dir remove /var/log/.touch: no such file or directory" in remote testers.
• Add pkg/k8s-client.WaitForDeploymentCompletes.
• Add pkg/k8s-client.WithPodFunc to debug job pod failures.
• Use regional STS endpoint by default in pkg/aws.

# Upgrading Kubernetes to "v1.19" is reverted...
-### Dependency
-
-- Upgrade [`github.com/kubernetes/kubernetes`](https://github.com/kubernetes/kubernetes/releases) from [`v1.18.6-rc.0`](https://github.com/kubernetes/kubernetes/releases/tag/v1.18.6-rc.0) to [`v1.19.0-rc.0`](https://github.com/kubernetes/kubernetes/releases/tag/v1.19.0-rc.0).
-- Upgrade [`github.com/kubernetes/client-go`](https://github.com/kubernetes/client-go/releases) from [`v0.18.6-rc.0`](https://github.com/kubernetes/clienthttps://github.com/kubernetes/client-go/releases/tag/v0.18.6-rc.0) to [`v0.19.0-rc.0`](https://github.com/kubernetes/client-go/releases/tag/v0.19.0-rc.0).
-  - See [commit `0e4cbc8e` for all the `eks` changes](https://github.com/aws/aws-k8s-tester/commit/0e4cbc8e0a3b7c7f3808e40205ecf5dc6d3ddbe9).
-  - See [commit `f1a984e3` for all the `vendor` changes](https://github.com/aws/aws-k8s-tester/commit/f1a984e394c880a1864327f97bb54ffab94e48f8).
-  - ref. https://github.com/kubernetes/kubernetes/pull/90552 changes `k8s.io/kubernetes/pkg/kubelet/remote` to `k8s.io/kubernetes/pkg/kubelet/cri/remote`.

# github.com/containerd/containerd/sys
vendor/github.com/containerd/containerd/sys/proc.go:33:34: undefined: system.GetClockTicks
github.com/google/cadvisor/container/raw
# github.com/google/cadvisor/container/raw
vendor/github.com/google/cadvisor/container/raw/handler.go:62:20: undefined: "github.com/opencontainers/runc/libcontainer/cgroups/fs".Manager
# github.com/google/cadvisor/container/docker
vendor/github.com/google/cadvisor/container/docker/handler.go:140:20: undefined: "github.com/opencontainers/runc/libcontainer/cgroups/fs".Manager
# github.com/google/cadvisor/container/containerd
vendor/github.com/google/cadvisor/container/containerd/handler.go:72:20: undefined: "github.com/opencontainers/runc/libcontainer/cgroups/fs".Manager
github.com/google/cadvisor/container/crio
# github.com/google/cadvisor/container/crio
vendor/github.com/google/cadvisor/container/crio/handler.go:74:23: undefined: "github.com/opencontainers/runc/libcontainer/cgroups/fs".Manager
vendor/github.com/google/cadvisor/container/crio/handler.go:98:20: undefined: "github.com/opencontainers/runc/libcontainer/cgroups/fs".Manager
github.com/aws/aws-k8s-tester/pkg/aws
github.com/aws/aws-k8s-tester/eks/cluster-loader
github.com/aws/aws-k8s-tester/pkg/k8s-client
# helm.sh/helm/v3/pkg/kube
vendor/helm.sh/helm/v3/pkg/kube/client.go:180:26: too many arguments in call to helper.Get
vendor/helm.sh/helm/v3/pkg/kube/client.go:180:58: info.Export undefined (type *"k8s.io/cli-runtime/pkg/resource".Info has no field or method Export)
vendor/helm.sh/helm/v3/pkg/kube/client.go:380:31: too many arguments in call to helper.Get
vendor/helm.sh/helm/v3/pkg/kube/client.go:380:69: target.Export undefined (type *"k8s.io/cli-runtime/pkg/resource".Info has no field or method Export)
vendor/helm.sh/helm/v3/pkg/kube/client.go:485:11: undefined: "k8s.io/client-go/tools/watch".ListWatchUntil

Dependency

• Upgrade helm.sh/helm/v3 from v3.2.3 to v3.2.4.
  • v3.3.0-rc.1 does not work...
  • ref. kubectl -n grafana logs pod/grafana-test shows [ "$code" == "200" ]' failed.

Go

• Compile with Go 1.14.4.

---

v1.4.4 (2020-07-10)

See code changes.

eksconfig

• Make EKS 1.17 default version.
  • aws/containers-roadmap#697

eks

• Fix node label for BottleRocket OS worker nodes.
• Increase ListCSRs batch limit to 1,000.
• Increase ListNodes batch limit to 1,000.
• Do not print spinner if not supported.
• Set upper limit for WaitForJobCompletes.

Go

• Compile with Go 1.14.4.

---

v1.4.3 (2020-07-10)

See code changes.

ec2

• Only print SSMCommands when total nodes are less than 10.

eksconfig

• Fix AddOnConformance sonobuoy-result/plugins/e2e/results/global/e2e.log upload path.

eks

• Clean up worker nodes polling outputs.
• Only print SSMCommands when total nodes are less than 10.
• Fix eks/conformance sonobuoy-result/plugins/e2e/results/global/e2e.log upload.
• Add "spinner" to all polling functions.

pkg

• Add pkg/spinner.
• Pass log writer to pkg/cfn.Poll for "spinner".
• Increase list pod batch limit and reduce batch interval for pkg/k8s-client.WaitForJobCompletes.
• Retry pkg/k8s-client.ListPod when a paginated response is stale.
  • Fix "The provided continue parameter is too old to display a consistent list result. You can start a new list without the continue parameter, or use the continue token in this response to retrieve the remainder of the results. Continuing with the provided token results in an inconsistent list - objects that were created, modified, or deleted between the time the first chunk was returned and now may show up in the list.".

Dependency

• Add github.com/briandowns/spinner v1.11.1.

Go

• Compile with Go 1.14.4.

---

v1.4.2 (2020-07-09)

See code changes.

ec2

• Handle duplicate EC2 key pair creation error.

eks

• Fix eks/ng ASG launch template for multiple instance types.
• Handle duplicate EC2 key pair creation error.
• Wait AddOnClusterVersionUpgrade.WaitBeforeUpgrade before running cluster version upgrades.
• Fix instance state polling in eks/ng and eks/mng.

eksconfig

• Add AddOnClusterVersionUpgrade.WaitBeforeUpgrade.
  • Set via AWS_K8S_TESTER_EKS_ADD_ON_CLUSTER_VERSION_UPGRADE_WAIT_BEFORE_UPGRADE=5m.

pkg

• Pass context.Context and chan struct{} to pkg/aws/ec2.PollUntilRunning.

Dependency

• Upgrade github.com/aws/aws-sdk-go from v1.33.3 to v1.33.5.

Go

• Compile with Go 1.14.4.

---

v1.4.1 (2020-07-07)

See code changes.

ec2

• Fix log fetch, make it work run multiple times.

eks

• Add eks/cw-agent and eks/fluentd.
• Add eks/php-apache.
• Support ECR repository for busybox images to minimize docker hub dependency in eks/cron-jobs and eks/jobs-echo.
• Handle NotFound errors in delete operations.
• Fix log fetch, make it work run multiple times.
• Increase MNG update timeouts.
• Create regional ECR client to all remote testers based on RepositoryRegion.

eksconfig

• Set AddOnNodeGroups.FetchLogs to false by default, to reduce the test runtime for a large number of worker nodes.
  • Set AWS_K8S_TESTER_EKS_ADD_ON_NODE_GROUPS_FETCH_LOGS=true to enable.
  • To stream pod logs to CloudWatch, set AWS_K8S_TESTER_EKS_ADD_ON_FLUENTD_ENABLE=true.
• Set AddOnManagedNodeGroups.FetchLogs to false by default, to reduce the test runtime for a large number of worker nodes.
  • Set AWS_K8S_TESTER_EKS_ADD_ON_MANAGED_NODE_GROUPS_FETCH_LOGS=true to enable.
  • To stream pod logs to CloudWatch, set AWS_K8S_TESTER_EKS_ADD_ON_FLUENTD_ENABLE=true.
• Add AddOnCWAgent and AddOnFluentd.
• Add AddOnPHPApache.
• Support ECR repository for busybox images to minimize docker hub dependency in AddOnCronJobs and AddOnJobsEcho.
• Add RepositoryRegion to all remote testers.
• Reduce AddOnIRSA default replicas to 1.

pkg/aws/ecr

• Add region checks to Check.

Dependency

• Upgrade github.com/aws/aws-sdk-go from v1.32.11 to v1.33.3.

Go

• Compile with Go 1.14.4.

---

v1.4.0 (2020-06-29)

See code changes.

ec2-utils

• ec2-utils --auto-path now automatically uses the generated cluster name for local file paths, instead of random string.

ec2config

• Enable S3BucketCreate and S3BucketCreateKeep by default, error if no S3 bucket is specified.

aws-k8s-tester

• aws-k8s-tester --auto-path now automatically uses the generated cluster name for local file paths, instead of random string.
• Remove --block flags.
• Print JSON body for (managed) node groups.

ec2

• Clean up S3 uploads.
• Clean up colorstring printf.

eks

• Set timeouts for "aws sts get-caller-identity" for eks/irsa and eks/irsa-fargate.
• Improve and clean up all Poll functions with contexts.
• Add missing "configmaps" to eks/stresser/remote RBAC.
• Clean up eks/irsa-fargate.
• Use multi-writer to pipe stderr logging to log file.
• Run query function while checking eks/mng version upgrades.
• Improve and clean up eks/irsa.
• Add clusterloader --provider=eks flag to eks/cluster-loader.
• Add eks/cluster-loader CL2UseHostNetworkPods support.
• Explicitly set RestartPolicy in all pod objects.
• Run eks/mng/scale.Scale after creating add-ons.
• Fix eks/mng/scale.
• Update nodes after eks/mng/scale.
• Remove eks/tester.Tester.AggregateResults.
• eks/cluster-loader with Job object to run remote testers "once", instead of Deployment.
• eks/cluster-loader to remove log fetcher dependency for remote tester, use S3 instead.
• eks/configmaps with Job object to run remote testers "once", instead of Deployment.
• eks/configmaps to remove log fetcher dependency for remote tester, use S3 instead.
• eks/csrs with Job object to run remote testers "once", instead of Deployment.
• eks/csrs to remove log fetcher dependency for remote tester, use S3 instead.
• eks/secrets with Job object to run remote testers "once", instead of Deployment.
• eks/secrets to remove log fetcher dependency for remote tester, use S3 instead.
• eks/stresser with Job object to run remote testers "once", instead of Deployment.
• eks/stresser to remove log fetcher dependency for remote tester, use S3 instead.
• Clean up S3 uploads.
• Compare raw data points for regression tests.
  • Used for Kolmogorov–Smirnov test.
• Publish performance data to CloudWatch.
• Add compare tests for all stressing tests, useful for regression tests.
• Improve eks/configmaps, eks/csrs, eks/secrets results collect with S3.
• Add eks/tester.Tester.Name method.
• Fix eks/stresser collect metrics.
• Clean up colorstring printf.
• Clean up polling operation error handling.
  • Rename eks/cluster/version-upgrade.Poll to eks/cluster/wait.PollUpdate.
• Discard HTTP download progress for URL checks.
• Increase cluster-loader timeout and error if output is not expected.
• Run AWS_K8S_TESTER_EKS_ADD_ON_CLUSTER_LOADER before AWS_K8S_TESTER_EKS_ADD_ON_HOLLOW_NODES.
  • TODO: Handle 23 resource_gather_worker.go:63] error while reading data from hollowdreamprimehaze8iw5ul: the server is currently unable to handle the request (get nodes hollowdreamprimehaze8iw5ul:10250).
• Improve eks/cluster-loader/remote result download, use "scp".
• Store kube-apiserver /metrics output.
• Add eks/cluster-loader.ParsePodStartupLatency.
• Add eks/cluster-loader.MergePodStartupLatency.
• Record PodStartupLatency in eksconfig.AddOnClusterLoaderLocal via eks/cluster-loader/local.
• Record PodStartupLatency in eksconfig.AddOnClusterLoaderRemote via eks/cluster-loader/remote.
• Fix eks/cluster-loader error handling.
• Add S3 access policies to worker node roles.
• Improve eks/stresser/remote results fetch.
• Fix multiple eks/cluster-loader runs.
• Add extra namespace force-deletion function to eks/stresser/remote.
• eks/mng/scale added to scale mngs up and down
  • See https://docs.aws.amazon.com/cli/latest/reference/eks/update-nodegroup-config.html.

eksconfig

• AWS_K8S_TESTER_EKS_ADD_ON_CSRS_REMOTE_DEPLOYMENT_REPLICAS is now AWS_K8S_TESTER_EKS_ADD_ON_CSRS_REMOTE_COMPLETES and AWS_K8S_TESTER_EKS_ADD_ON_CSRS_REMOTE_PARALLELS.
• AWS_K8S_TESTER_EKS_ADD_ON_CONFIGMAPS_REMOTE_DEPLOYMENT_REPLICAS is now AWS_K8S_TESTER_EKS_ADD_ON_CONFIGMAPS_REMOTE_COMPLETES and AWS_K8S_TESTER_EKS_ADD_ON_CONFIGMAPS_REMOTE_PARALLELS.
• AWS_K8S_TESTER_EKS_ADD_ON_SECRETS_REMOTE_DEPLOYMENT_REPLICAS is now AWS_K8S_TESTER_EKS_ADD_ON_SECRETS_REMOTE_COMPLETES and AWS_K8S_TESTER_EKS_ADD_ON_SECRETS_REMOTE_PARALLELS.
• AWS_K8S_TESTER_EKS_ADD_ON_STRESSER_REMOTE_DEPLOYMENT_REPLICAS is now AWS_K8S_TESTER_EKS_ADD_ON_STRESSER_REMOTE_COMPLETES and AWS_K8S_TESTER_EKS_ADD_ON_STRESSER_REMOTE_PARALLELS.
• Use Job for all remote testers.
• Enable S3BucketCreate and S3BucketCreateKeep by default, error if no S3 bucket is specified.
• Configure S3 directory.
• Add ClusterAutoscaler add-on per node-group using AWS_K8S_TESTER_EKS_ADD_ON_NODE_GROUPS_ASGS={"GetRef.Name-...":{..."cluster-autoscaler":{"enable":false}...}}.
• Fix typo in eksconfig.AddOnManagedNodeGroups.LogsTarGzPath.
• Add Status.PrivateDNSToNodeInfo for node SSH access.
• Record PodStartupLatency in eksconfig.AddOnClusterLoaderLocal via eks/cluster-loader/local.
• Record PodStartupLatency in eksconfig.AddOnClusterLoaderRemote via eks/cluster-loader/remote.
• Add RequestsSummaryWritesCompareS3Dir and RequestsSummaryReadsCompareS3Dir.
• Add AddOnClusterLoader* CL2UseHostNetworkPods support.

ssh

• Move "scp" executable binary path check before creating context timeouts.
• Fix and improve retries.

pkg

• Add pkg/k8s-client.WaitForJobCompletes,WaitForCronJobCompletes.
• Add pkg/aws/s3.
• Add pkg/k8s-client.EKSConfig.MetricsRawOutputDir to store kube-apiserver /metrics output.
• Add pkg/k8s-client.WithForceDelete option for DeleteNamespaceAndWait.
  • Fix #100.
  • See kubernetes/kubernetes#60807.
• Add pkg/metrics.RequestsCompare.

Dependency

• Upgrade github.com/aws/aws-sdk-go from v1.32.2 to v1.32.11.
• Upgrade github.com/kubernetes/kubernetes from v1.18.2 to v1.18.6-rc.0.
• Upgrade github.com/kubernetes/client-go from v0.18.3 to v0.18.6-rc.0.
  • See commit fc93a579 for all the changes.
• Add k8s.io/perf-tests.
  • See 1aea23d3 for commit.

Go

• Compile with Go 1.14.4.

---