Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pubsys: add SSM parameter publishing #1060

Merged
merged 1 commit into from
Aug 24, 2020

Conversation

zmrow
Copy link
Contributor

@zmrow zmrow commented Aug 21, 2020

cargo make ssm will use the amis.json from cargo make ami to populate
parameter name/value templates from files in policies/ssm. The parameters
are set in SSM using the full build version, e.g. "0.5.0-abcdef", and it won't
overwrite by default.

cargo make promote-ssm -e SSM_TARGET=VERSION will promote (copy) those values
from the full-build-version name to a more general one. It's recommended to
promote from the full version to a short version like "0.5.0", then to a
well-known pointer like "latest". This allows for easy rollback by promoting
from an older version, using -e SSM_SOURCE=OLD_VERSION.

Testing done:

  • Tested setting more than 100 parameters in a region - request chunking and throttling worked, parameters got through
  • Tested the "new namespace" issue in opt-in region - error was caught properly and it was treated as a new parameter
  • Tested many regions at once
  • Confirmed no overwrite allowed unless ALLOW_CLOBBER
  • Confirmed that adding regions / parameters will add the new ones without considering it a clobber (existing parameters show no change; idempotent)
  • Confirmed that override templates work, and it doesn't run overrides not matching current values
  • Confirmed that set_parameter retries work, or at least that bad requests are retried and eventually fail, and good requests pass, and the output is clear about failures and failure count.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@tjkirch
Copy link
Contributor

tjkirch commented Aug 21, 2020

^ This push fixes an issue Ben found. There's a convenience function to join together SSM prefix and parameter name, making sure there's only one slash between them, and we removed a character in the wrong branch.

@tjkirch tjkirch changed the base branch from develop to sts-region August 23, 2020 16:51
@tjkirch
Copy link
Contributor

tjkirch commented Aug 23, 2020

⬆️ This force-push rebases on #1064 to get the STS region fix.

⬇️ The force-push below incorporates the changes from #1064 into the SSM code from this PR.

@tjkirch tjkirch changed the base branch from sts-region to develop August 23, 2020 21:18
Copy link
Contributor

@webern webern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@@ -38,6 +38,8 @@ regions = ["us-west-2", "us-east-1", "us-east-2"]
profile = "my-profile"
# If specified, we assume this role before making any API calls.
role = "arn:aws:iam::012345678901:role/assume-global"
# If specified, this string will be prefixed on all parameter names published to SSM.
ssm_prefix = "/your/prefix/here"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path to the ssm policy is not specified in Infra.toml?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though we could consider moving it, and there's also some separate discussion about combining some of our configuration sources...

The way we were thinking about the configuration sources is that the Makefile would have default values/paths for things that should work for most people and rarely need to change; Infra.toml would have user-specific settings that rarely need to change; parameters handle things that always need to be specified or frequently change; and JSON files as a "runtime store" for things that change each time, like the AMI IDs.

tools/pubsys/src/aws/ami/mod.rs Show resolved Hide resolved
tools/pubsys/src/aws/ssm/template.rs Outdated Show resolved Hide resolved
tools/pubsys/src/aws/ssm/template.rs Show resolved Hide resolved
tools/pubsys/src/aws/ssm/ssm.rs Show resolved Hide resolved
name = "{variant}/{arch}/{image_version}/image_id"
value = "{image_id}"

[[parameter]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we have one of these for current? Should that be in the defaults.toml as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For cargo ssm, the image_version in the template is replaced by the version you're building, and for promote-ssm, by the version you specify with -e SSM_TARGET=. So when we want to publish latest, we'd do cargo make promote-ssm -e SSM_TARGET=latest. In our release process we're planning to promote once from the full version (e.g. 0.5.0-abcdef) to the short version (0.5.0) and again to latest.

`cargo make ssm` will use the amis.json from `cargo make ami` to populate
parameter name/value templates from files in `policies/ssm`.  The parameters
are set in SSM using the full build version, e.g. "0.5.0-abcdef", and it won't
overwrite by default.

`cargo make promote-ssm -e SSM_TARGET=VERSION` will promote (copy) those values
from the full-build-version name to a more general one.  It's recommended to
promote from the full version to a short version like "0.5.0", then to a
well-known pointer like "latest".  This allows for easy rollback by promoting
from an older version, using `-e SSM_SOURCE=OLD_VERSION`.

Co-authored-by: Zac Mrowicki <mrowicki@amazon.com>
Co-authored-by: Tom Kirchner <tjk@amazon.com>
@tjkirch
Copy link
Contributor

tjkirch commented Aug 24, 2020

^ This force-push fixes a typo caught by @webern.

@tjkirch tjkirch merged commit 27755a0 into bottlerocket-os:develop Aug 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants