Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(eks): add support of Helm charts located in ECR of AWS CN region #29778

Merged
merged 4 commits into from
Apr 17, 2024

Conversation

yubingjiaocn
Copy link
Contributor

Issue # (if applicable)

Closes #28460.

Reason for this change

Current implementation will not identity helm charts stored in ECR on AWS CN regions, and will treat ECR as simple, unauthorized repository.

Description of changes

This change add support of ECR on AWS CN region by adding a optional suffix of .cn to the regex.

Description of how you validated changes

Run the affected regex against helm repo in ECR China (123456789012.dkr.ecr.cn-northwest-1.amazonaws.com.cn)

import re
repository = 'oci://123456789012.dkr.ecr.cn-northwest-1.amazonaws.com.cn'
private_ecr_pattern = 'oci://(?P<registry>\d+\.dkr\.ecr\.(?P<region>[a-z0-9\-]+)\.amazonaws\.com(\.cn)?)*'
private_registry = re.match(private_ecr_pattern, repository).groupdict()
print(private_registry['registry'])

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added the beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK label Apr 10, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team April 10, 2024 06:44
@github-actions github-actions bot added bug This issue is a bug. effort/small Small work item – less than a day of effort p2 labels Apr 10, 2024
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@yubingjiaocn yubingjiaocn changed the title Fix(eks): Add support of Helm charts located in ECR of AWS CN region fix(eks): add support of Helm charts located in ECR of AWS CN region Apr 10, 2024
@yubingjiaocn
Copy link
Contributor Author

Exemption Request: This change is related to custom resources handler. Test is separated and no test file can be provided in code repo.

@aws-cdk-automation aws-cdk-automation added pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Apr 10, 2024
@pahud
Copy link
Contributor

pahud commented Apr 10, 2024

Thank you for your PR.

Can you write a simple CDK App like sample.ts in packages/@aws-cdk-testing/framework-integ/test/aws-eks/test as described in the contributing guide and verify it with successful deployments for both regions in China?

@yubingjiaocn
Copy link
Contributor Author

yubingjiaocn commented Apr 14, 2024

Hi,

I have added integrated testing for this PR. Unfortunately I can't put it into repository due to it contains my AWS account info. So I'll describe my testing procedure here.

I added a sample stack integ.eks-helm-asset-oci-cn.ts with following code:

/// !cdk-integ pragma:disable-update-workflow
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import { App, Stack } from 'aws-cdk-lib';
import * as integ from '@aws-cdk/integ-tests-alpha';
import { getClusterVersionConfig } from './integ-tests-kubernetes-version';
import * as eks from 'aws-cdk-lib/aws-eks';

class EksClusterStack extends Stack {
  private cluster: eks.Cluster;
  private vpc: ec2.IVpc;

  constructor(scope: App, id: string) {
    super(scope, id);

    // allow all account users to assume this role in order to admin the cluster
    const mastersRole = new iam.Role(this, 'AdminRole', {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    // just need one nat gateway to simplify the test
    this.vpc = new ec2.Vpc(this, 'Vpc', { natGateways: 1, restrictDefaultSecurityGroup: false });

    // create the cluster with a default nodegroup capacity
    this.cluster = new eks.Cluster(this, 'Cluster', {
      vpc: this.vpc,
      mastersRole,
      defaultCapacity: 2,
      ...getClusterVersionConfig(this),
      tags: {
        foo: 'bar',
      },
    });

    this.assertHelmChartAsset();
  }

  private assertHelmChartAsset() {
    // get helm chart from Asset
    this.cluster.addHelmChart('test-oci-cn-chart', {
      chart: 'hello-world',
      release: 'hello-world-chart',
      repository: 'oci://<REDACTED>.dkr.ecr.cn-northwest-1.amazonaws.com.cn/charts/hello-world',
      version: '0.1.0',
      namespace: 'default',
      createNamespace: true,
    });;
  }
}

const app = new App();

const stack = new EksClusterStack(app, 'aws-cdk-eks-helm-test');
new integ.IntegTest(app, 'aws-cdk-eks-helm', {
  testCases: [stack],
  // Test includes assets that are updated weekly. If not disabled, the upgrade PR will fail.
  diffAssets: false,
});

app.synth();

Then I run integration testing command against the stack:

yarn integ integ.eks-helm-asset-oci-cn.js --force --directory ./test/aws-eks/test/ --update-on-failed --no-clean --parallel-regions cn-north-1

and get the following output:

[ec2-user@ip-172-31-0-241 test]$ yarn integ integ.eks-helm-asset-oci-cn.js --force --directory ./test/aws-eks/test/ --update-on-failed --no-clean --parallel-regions cn-north-1 
yarn run v1.22.22
$ integ-runner --language javascript integ.eks-helm-asset-oci-cn.js --force --directory ./test/aws-eks/test/ --update-on-failed --no-clean --parallel-regions cn-north-1

Verifying integration test snapshots...

  NEW        integ.eks-helm-asset-oci-cn 0.764s

Snapshot Results: 

Tests:    1 failed, 1 total
Failed: /home/ec2-user/aws-cdk/packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-helm-asset-oci-cn.js

Running integration tests for failed tests...

Running in parallel across regions: cn-north-1
Running test /home/ec2-user/aws-cdk/packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-helm-asset-oci-cn.js in cn-north-1
  SUCCESS    integ.eks-helm-asset-oci-cn-aws-cdk-eks-helm/DefaultTest 836.377s
       NO ASSERTIONS

Test Results: 

Tests:    1 passed, 1 total
Not cleaning up stacks since "--no-clean" was used
Done in 837.65s.

The screenshot of deployed CFN stack shows Clustercharttestocicnchart is deployed successfully:

cfn-cn-north-1

helm CLI also shows the deployed helm release:

helm-cn-north-1

These screenshot can confirm the deployment is succeed with a helm chart hosted on ECR of AWS China (Beijing) region (cn-north-1).

I also tested on China (Ningxia) region (cn-northwest-1) with a success. Here are the screenshots:

CFN stack:

cfn-cn-northwest-1

Helm:

helm-cn-northwest-1

@pahud
Copy link
Contributor

pahud commented Apr 15, 2024

OK.

I have successfully deployed the following code into cn-north-1 and cn-northwest-1 with my custom helm chart I built according to this doc which is pushed to China ECR.

import {
  App, Stack,
  aws_eks as eks,
  aws_ec2 as ec2,
  aws_iam as iam,
} from 'aws-cdk-lib';

import { KubectlV29Layer } from '@aws-cdk/lambda-layer-kubectl-v29';

const app = new App();

const stack = new Stack(app, 'my-test-stack4');

const mastersRole = new iam.Role(stack, 'AdminRole', {
  assumedBy: new iam.AccountRootPrincipal(),
});

const vpc = new ec2.Vpc(stack, 'Vpc', { natGateways: 1 });

const cluster = new eks.Cluster(stack, 'Cluster', {
  vpc,
  mastersRole,
  version: eks.KubernetesVersion.V1_29,
  kubectlLayer: new KubectlV29Layer(stack, 'KubectlLayer'),
  defaultCapacity: 2,
});

cluster.addHelmChart('test-oci-cn-chart', {
  chart: 'helm-test-chart',
  release: 'helm-test-chart-chart',
  repository: 'oci://<deducted>.dkr.ecr.cn-north-1.amazonaws.com.cn/helm-test-chart',
  version: '0.1.0',
  namespace: 'default',
  createNamespace: true,
});;

And verified the ConfigMap of the Helm chart with

$ kubectl get configmap helm-test-chart-configmap -o jsonpath='{.data}'
{"myvalue":"Hello World"}

I can confirm this works in both China regions now.

And the following code in us-east-1 and ap-northeast-1

import {
  App, Stack,
  aws_eks as eks,
  aws_ec2 as ec2,
  aws_iam as iam,
} from 'aws-cdk-lib';

import { KubectlV29Layer } from '@aws-cdk/lambda-layer-kubectl-v29';

const app = new App();

const stack = new Stack(app, 'my-test-stack4');

const mastersRole = new iam.Role(stack, 'AdminRole', {
  assumedBy: new iam.AccountRootPrincipal(),
});

const vpc = new ec2.Vpc(stack, 'Vpc', { natGateways: 1 });

const cluster = new eks.Cluster(stack, 'Cluster', {
  vpc,
  mastersRole,
  version: eks.KubernetesVersion.V1_29,
  kubectlLayer: new KubectlV29Layer(stack, 'KubectlLayer'),
  defaultCapacity: 2,
});

cluster.addHelmChart('test-oci-chart', {
  chart: 'helm-test-chart',
  release: 'helm-test-chart-chart',
  repository: 'oci://<deducted>.dkr.ecr.us-east-1.amazonaws.com/helm-test-chart',
  version: '0.1.0',
  namespace: 'default',
  createNamespace: true,
});;

And I have verified successful deployments in us-east-1 and ap-northeast-1 as well.

I think this PR is safe to ship.

@GavinZZ GavinZZ added pr-linter/exempt-integ-test The PR linter will not require integ test changes pr-linter/exempt-test The PR linter will not require test changes and removed pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. labels Apr 15, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review April 17, 2024 19:00

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@TheRealAmazonKendra
Copy link
Contributor

@Mergifyio update

Copy link
Contributor

mergify bot commented Apr 17, 2024

update

❌ Mergify doesn't have permission to update

For security reasons, Mergify can't update this pull request. Try updating locally.
GitHub response: refusing to allow a GitHub App to create or update workflow .github/workflows/request-cli-integ-test.yml without workflows permission

Copy link
Contributor

@TheRealAmazonKendra TheRealAmazonKendra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Copy link
Contributor

mergify bot commented Apr 17, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Apr 17, 2024
Copy link
Contributor

mergify bot commented Apr 17, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@TheRealAmazonKendra
Copy link
Contributor

@Mergifyio update

Copy link
Contributor

mergify bot commented Apr 17, 2024

update

❌ Mergify doesn't have permission to update

For security reasons, Mergify can't update this pull request. Try updating locally.
GitHub response: refusing to allow a GitHub App to create or update workflow .github/workflows/request-cli-integ-test.yml without workflows permission

Copy link
Contributor

mergify bot commented Apr 17, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 77fcaba
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 0da25e5 into aws:main Apr 17, 2024
9 checks passed
Copy link
Contributor

mergify bot commented Apr 17, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p2 pr-linter/exempt-integ-test The PR linter will not require integ test changes pr-linter/exempt-test The PR linter will not require test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EKS: Unable to log in ECR on AWS China partition when adding helm chart
5 participants