-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SAM] Makes S3 Bucket parameter optional and creates bucket automatically in the deployment region #3040
Conversation
…ng captured by CLI either way
…ptured as part of CLI already
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is super useful. Thanks for doing it.
It might be useful to let the user control bucket name by adding another parameter --create-bucket
which will optionally create a bucket with name --s3-bucket
if it does not exist. If the bucket exists, this will be a no-op.
What do you think?
def _run_main(self, parsed_args, parsed_globals): | ||
region = parsed_globals.region if parsed_globals.region else "us-east-1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we error out if region is not supplied? Last thing we want is surprises. If they are using AWS CLI, they have most likely set the region. So this will be raising the error for the 1% case
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That’s a good idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was our initial surprise when hacking that during the hackathon - If you don't supply "--region" during the package command "parsed_globals.region" will be "None" and we were expecting to be whatever was set in the AWS CLI - That's the reason we had this one-liner if.
What's set in the AWS CLI like regions for a profile (default, lab, etc.) only seems to work when you initiate a connection with a service (self._session <- contains a dict of regions set in a profile and likely use those in the absence of one).
Given that we need the region set as a parameter to run some additional logic we have two options here:
- Error out if not supplied and end execution there as simple as that
- Try capture the default region configured in the AWS CLI (there's gotta be an easy way other than parsing self._session from 'default' or from a 'profile' if set)
Implemented 1st option for now and commit to follow
Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Heitor, I see your point. But give how buckets are global, this might do more harm than good.
@jamesls Do you know of any option to do Option-2 above (reliably get the Region set in their creds chain)?
template_path=template_path) | ||
template_path=template_path) | ||
|
||
if (parsed_args.s3_bucket is not None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bracket not needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
def _get_bucket_region(self, s3_bucket, s3_client): | ||
s3_loc = s3_client.get_bucket_location( | ||
Bucket=s3_bucket).get("LocationConstraint", "us-east-1") | ||
return s3_loc.get |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why another .get
? Isn't s3_loc
just a string?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It allows the function to be mocked for the tests easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sanath is right.. This code seems incomplete, I'm download the latest source code and will send another commit after checking tests, etc. That part should've been:
def _get_bucket_region(self, s3_bucket, s3_client):
s3_loc = s3_client.get_bucket_location(Bucket=s3_bucket)
return s3_loc.get("LocationConstraint", "us-east-1")
instead of:
def _get_bucket_region(self, s3_bucket, s3_client):
s3_loc = s3_client.get_bucket_location(
Bucket=s3_bucket).get("LocationConstraint", "us-east-1")
return s3_loc.get
if (parsed_args.s3_bucket is not None): | ||
bucket = parsed_args.s3_bucket | ||
s3_bucket_region = self._get_bucket_region(bucket, s3_client) | ||
if not s3_bucket_region == region: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if s3_bucket_region is not region:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's an exact word comparison and has to be "==" as "is not" isn't exactly the same here and can lead to surprised and make tests to fail.
}, | ||
"us_standard": { | ||
"Bucket": bucket, | ||
"CreateBucketConfiguration": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand. I thought the location constraint was for non-us-standard regions. Am I missing something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right - Fixed.
I think adding the new flag would be redundant. There’s already a dry run flag to perform no-ops.
|
Thanks for all comments @sanathkr - We'll go through them by max EOW. We discussed having that option on s3 bucket name and after considering all scenarios and trying in practice we found that it only adds more complexity -- Main idea was to make developers to remember less parameters and get going more quickly. The ideal scenario we thought was for SAM to support a "project_name" variable of some sort, of which it'd allow the developer to have the S3 bucket automatically created and named using the project_name + uuid. Happy to hear your thoughts on this though |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented Sanath's suggestions and replied to other comments. One thought to be discussed though on --region flag
def _get_bucket_region(self, s3_bucket, s3_client): | ||
s3_loc = s3_client.get_bucket_location( | ||
Bucket=s3_bucket).get("LocationConstraint", "us-east-1") | ||
return s3_loc.get |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sanath is right.. This code seems incomplete, I'm download the latest source code and will send another commit after checking tests, etc. That part should've been:
def _get_bucket_region(self, s3_bucket, s3_client):
s3_loc = s3_client.get_bucket_location(Bucket=s3_bucket)
return s3_loc.get("LocationConstraint", "us-east-1")
instead of:
def _get_bucket_region(self, s3_bucket, s3_client):
s3_loc = s3_client.get_bucket_location(
Bucket=s3_bucket).get("LocationConstraint", "us-east-1")
return s3_loc.get
template_path=template_path) | ||
template_path=template_path) | ||
|
||
if (parsed_args.s3_bucket is not None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
if (parsed_args.s3_bucket is not None): | ||
bucket = parsed_args.s3_bucket | ||
s3_bucket_region = self._get_bucket_region(bucket, s3_client) | ||
if not s3_bucket_region == region: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's an exact word comparison and has to be "==" as "is not" isn't exactly the same here and can lead to surprised and make tests to fail.
def _run_main(self, parsed_args, parsed_globals): | ||
region = parsed_globals.region if parsed_globals.region else "us-east-1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was our initial surprise when hacking that during the hackathon - If you don't supply "--region" during the package command "parsed_globals.region" will be "None" and we were expecting to be whatever was set in the AWS CLI - That's the reason we had this one-liner if.
What's set in the AWS CLI like regions for a profile (default, lab, etc.) only seems to work when you initiate a connection with a service (self._session <- contains a dict of regions set in a profile and likely use those in the absence of one).
Given that we need the region set as a parameter to run some additional logic we have two options here:
- Error out if not supplied and end execution there as simple as that
- Try capture the default region configured in the AWS CLI (there's gotta be an easy way other than parsing self._session from 'default' or from a 'profile' if set)
Implemented 1st option for now and commit to follow
Thoughts?
}, | ||
"us_standard": { | ||
"Bucket": bucket, | ||
"CreateBucketConfiguration": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right - Fixed.
Codecov Report
@@ Coverage Diff @@
## develop #3040 +/- ##
===========================================
+ Coverage 95.93% 96.01% +0.08%
===========================================
Files 166 166
Lines 12610 12640 +30
===========================================
+ Hits 12097 12136 +39
+ Misses 513 504 -9
Continue to review full report at Codecov.
|
@heitorlessa Can you fix the test failures? |
sorry @sanathkr but which failures? I ran them before pushing the latest commits and the only one that is complaining now is a health check from Codecov that I don't know how to improve - If that's the case, any suggestions that I can go and fix? 20:27:59 ➜ aws-cli git:(sam-package-enhanced) venv:(aws-cli-hackathon) python:(🐍 aws-cli-hackathon) python -m unittest -v tests/unit/customizations/cloudformation/test_package.py
test_main (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ...
Successfully packaged artifacts and wrote output template to file ./oputput.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/lessa/new-packaging-hackathon/aws-cli/oputput --stack-name <YOUR STACK NAME>
/Users/lessa/new-packaging-hackathon/aws-cli/tests/unit/customizations/cloudformation/test_package.py:81: DeprecationWarning: Please use assertEqual instead.
self.assertEquals(rc, 0)
Successfully packaged artifacts and wrote output template to file ./oputput.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/lessa/new-packaging-hackathon/aws-cli/oputput --stack-name <YOUR STACK NAME>
ok
test_main_bucket_different_deployment_region (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ... ok
test_main_empty_region_error (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ... ok
test_main_error (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ... ok
test_main_without_bucket (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ...
Successfully packaged artifacts and wrote output template to file ./oputput.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/lessa/new-packaging-hackathon/aws-cli/oputput --stack-name <YOUR STACK NAME>
Successfully packaged artifacts and wrote output template to file ./oputput.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/lessa/new-packaging-hackathon/aws-cli/oputput --stack-name <YOUR STACK NAME>
ok
test_write_output_to_stdout (tests.unit.customizations.cloudformation.test_package.TestPackageCommand) ... ok
----------------------------------------------------------------------
Ran 6 tests in 0.014s
OK |
AFAIK, I think it is complaining about code coverage drop.. |
into sam-package-enhanced * 'sam-package-enhanced' of https://github.com/adhorn/aws-cli: refactor to increase code coverage in unit test
@sanathkr Did some refactor in the original tests including mine and reached 100% according to There is one conflict on New coverage results according to Coverage lib 21:18:33 ➜ aws-cli git:(sam-package-enhanced) venv:(aws-cli-hackathon) python:(🐍 aws-cli-hackathon) coverage report
Name Stmts Miss Branch BrPart Cover
---------------------------------------------------------------------------------------------
...
awscli/customizations/cloudformation/__init__.py 7 3 0 0 57%
awscli/customizations/cloudformation/artifact_exporter.py 193 123 46 0 29%
awscli/customizations/cloudformation/deploy.py 74 51 20 0 24%
awscli/customizations/cloudformation/deployer.py 89 71 20 0 17%
awscli/customizations/cloudformation/exceptions.py 26 0 0 0 100%
awscli/customizations/cloudformation/package.py 72 0 18 0 100% <--- from 93.75%
awscli/customizations/cloudformation/yamlhelper.py 26 19 8 0 21%
...
---------------------------------------------------------------------------------------------
TOTAL 3713 2652 1163 6 22% |
@sanathkr Any update on this? I'd really love to see this getting merged. |
* sam pipeline bootstrap (aws#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <crehn@outlook.com> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <crehn@outlook.com> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <3804518+aahung@users.noreply.github.com> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: _sam <3804518+aahung@users.noreply.github.com> * sam pipeline init command (aws#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (aws#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (aws#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (aws#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (aws#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (aws#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (aws#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (aws#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (aws#2976) * Sam pipelines ux round 3 (aws#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (aws#2982) * Adding AWS_ prefix to displayed out. (aws#2993) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Add region to pipeline bootstrap interactive flow (aws#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (aws#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * List detected env names in pipeline init when prompt to input the env name (aws#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <crehn@outlook.com> * Adding account id to bootstrap message. (aws#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Cfn creds fix (aws#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Ux bootstrap revamp 20210706 (aws#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <crehn@outlook.com> * Adding account condition for CFN execution role. (aws#3027) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * pipeline UX revamp 20210707 (aws#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (aws#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix toml file incorrect read/write in init --bootstrap (aws#3037) * Temporarily removing account fix. (aws#3038) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Rename environment to stage (aws#3040) * Improve account source selection (aws#3042) * Fixing various cosmetics UX issues with pipeline workflow. (aws#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Auto skip questions if stage detected (aws#3045) * Autofill question if default value is presented * Allow to use index to select stage names (aws#3051) * Updating message when bootstrap stages are missing. (aws#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fixing bootstrap integ tests. (aws#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (aws#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (aws#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix pipeline init integ test because of pipelineconfig file exists (aws#3067) * Make stage name randomized to avoid race condition among multi canary runs (aws#3078) * Load number of stages from pipeline template (aws#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (aws#2950) * pipeline init UX: Ask to confirm when file exists (aws#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (aws#3087) * Adding accidentally removed tests back. (aws#3088) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> Co-authored-by: elbayaaa <72949274+elbayaaa@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Ahmed Elbayaa <elbayaaa@amazon.com> Co-authored-by: Tarun <c2tarun@users.noreply.github.com> Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>
Hi @heitorlessa, thanks for creating this PR and for your patience. Could you please rebase this PR against the latest branch for the team’s further consideration going forward? |
Going to close this PR for the time being. If someone later decides they want to pick this up, rebase it and it will be reopened for consideration. |
Hi everyone,
This PR makes it possible to omit
--s3-bucket
and AWS CLI will create a S3 bucket in the deployment region specified - This makes the following possible:--s3-bucket
was passedThis is a non-breaking change and those who are currently using
--s3-bucket
should continue to work just fine.Usage: Deploy SAM stack in Ireland region without specifying the bucket
cc'ing contributors @sthulb @igngar