diff --git a/README.md b/README.md index 765fc294..88564960 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,17 @@ # Chamber -Chamber is a tool for managing secrets. Currently it does so by storing +Chamber is a tool for managing secrets. Currently it does so by storing secrets in SSM Parameter Store, an AWS service for storing secrets. For detailed info about using chamber, read [The Right Way To Manage Secrets](https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/) ## 2.0 Breaking Changes -Starting with version 2.0, chamber uses parameter store's path based API by default. Chamber pre-2.0 supported this API using the `CHAMBER_USE_PATHS` environment variable. The paths based API has performance benefits and is the recommended best practice by AWS. +Starting with version 2.0, chamber uses parameter store's path based API by default. Chamber pre-2.0 supported this API using the `CHAMBER_USE_PATHS` environment variable. The paths based API has performance benefits and is the recommended best practice by AWS. -As a side effect of this change, if you didn't use path based secrets before 2.0, you will need to set `CHAMBER_NO_PATHS` to enable the old behavior. This option is deprecated, and We recommend only using this setting for supporting existing applications. +As a side effect of this change, if you didn't use path based secrets before 2.0, you will need to set `CHAMBER_NO_PATHS` to enable the old behavior. This option is deprecated, and We recommend only using this setting for supporting existing applications. -To migrate to the new format, you can take advantage of the `export` and `import` commands. For example, if you wanted to convert secrets for service `foo` to the new format using chamber 2.0, you can do: +To migrate to the new format, you can take advantage of the `export` and `import` commands. For example, if you wanted to convert secrets for service `foo` to the new format using chamber 2.0, you can do: ```bash CHAMBER_NO_PATHS=1 chamber export foo | chamber import foo - @@ -24,6 +24,7 @@ If you have a functional go environment, you can install with: ```bash go install github.com/segmentio/chamber/v2@latest ``` + for Go >= 1.17; or @@ -62,7 +63,7 @@ alias chamberprod='aws-vault exec production -- chamber' ## Setting up KMS Chamber expects to find a KMS key with alias `parameter_store_key` in the -account that you are writing/reading secrets. You can follow the [AWS KMS +account that you are writing/reading secrets. You can follow the [AWS KMS documentation](http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) to create your key, and [follow this guide to set up your alias](http://docs.aws.amazon.com/kms/latest/developerguide/programming-aliases.html). @@ -134,25 +135,28 @@ Event Version Date User Created 1 06-09 17:30:19 daniel-fuentes Updated 2 06-09 17:30:56 daniel-fuentes ``` + The `history` command gives a historical view of a given secret. This view is useful for auditing changes, and can point you toward the user who made the change so it's easier to find out why changes were made. ### Exec + ```bash $ chamber exec -- ``` `exec` populates the environment with the secrets from the specified services -and executes the given command. Secret keys are converted to upper case (for +and executes the given command. Secret keys are converted to upper case (for example a secret with key `secret_key` will become `SECRET_KEY`). -Secrets from services are loaded in the order specified in the command. For +Secrets from services are loaded in the order specified in the command. For example, if you do `chamber exec app apptwo -- ...` and both apps have a secret named `api_key`, the `api_key` from `apptwo` will be the one set in your environment. ### Reading + ```bash $ chamber read service key Key Value Version LastModified User @@ -167,6 +171,7 @@ the `--version/-v` flag to read can print older versions of the secret. Default version (-1) is the latest secret. ### Exporting + ```bash $ chamber export [--format ] [--output-file ] {"key":"secret"} @@ -175,43 +180,86 @@ $ chamber export [--format ] [--output-file ] `export` provides ability to export secrets in various file formats. The following file formats are supported: -* json (default) -* yaml -* java-properties -* csv -* tsv -* dotenv -* tfvars +- json (default) +- yaml +- java-properties +- csv +- tsv +- dotenv +- tfvars File is written to standard output by default but you may specify an output file. -To set env vars in your terminal you can use the `chamber env` command. For example, +To set env vars in your terminal you can use the `chamber env` command. For example, + ```shell source <(chamber env service)` printf "%s" "$SERVICE_VAR" ``` ### Importing + ```bash -$ chamber import +$ chamber import [--normalize-keys] ``` `import` provides the ability to import secrets from a json or yaml file (like the kind you get from `chamber export`). + +> __Note__ +> By default, `import` will **not** normalize key inputs, meaning that keys will +> be written to the secrets backend in the format they exist in the source file. +> In order to normalize keys on import, provide the `--normalize-keys` flag + +When normalizing keys, before write, the key will be be first converted to lowercase +to match how `chamber write` handles keys. + +Example: `DB_HOST` will be converted to `db_host`. + You can set `filepath` to `-` to instead read input from stdin. ### Deleting + ```bash -$ chamber delete service key +$ chamber delete [--exact-key] service key ``` `delete` provides the ability to remove a secret from chamber permanently, including the secret's additional metadata. There is no way to recover a secret once it has been deleted so care should be taken with this command. + +> __Note__ +> By default, `delete` will normalize any provided keys. To change that behavior, +> provide the `--exact-key` flag to attempt to delete the raw provided key. + +Example: Given the following setup, + +```bash +$ chamber list service +Key Version LastModified User +apikey 2 06-09 17:30:56 daniel-fuentes +APIKEY 1 06-09 17:30:34 daniel-fuentes +``` + +Calling + +```bash +$ chamber delete --exact-key service APIKEY +``` + +will delete only `APIKEY` from the service and leave only + +```bash +$ chamber list service +Key Version LastModified User +apikey 2 06-09 17:30:56 daniel-fuentes +``` + ### Finding + ```bash $ chamber find key ``` @@ -221,6 +269,7 @@ $ chamber find key ```bash $ chamber find value --by-value ``` + Passing `--by-value` or `-v` will search the values of all secrets and return the services and keys which match. @@ -249,9 +298,9 @@ If you'd like to use a custom SSM endpoint for chamber, you can use `CHAMBER_AWS ## S3 Backend (experimental) -By default, chamber store secrets in AWS Parameter Store. We now also provide an experimental S3 backend for storing secrets in S3 instead. +By default, chamber store secrets in AWS Parameter Store. We now also provide an experimental S3 backend for storing secrets in S3 instead. -To configure chamber to use the S3 backend, use `chamber -b s3 --backend-s3-bucket=mybucket`. Preferably, this bucket should reject uploads that do not set the server side encryption header ([see this doc for details how](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)) +To configure chamber to use the S3 backend, use `chamber -b s3 --backend-s3-bucket=mybucket`. Preferably, this bucket should reject uploads that do not set the server side encryption header ([see this doc for details how](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)) This feature is experimental, and not currently meant for production work. @@ -260,18 +309,19 @@ This feature is experimental, and not currently meant for production work. This backend is similar to the S3 Backend but uses KMS Key Encryption to encrypt your documents at rest, similar to the SSM Backend which encrypts your secrets at rest. You can read how S3 Encrypts documents with KMS [here](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). The highlights of SSE-KMS are: + - You can choose to create and manage encryption keys yourself, or you can choose to use your default service key uniquely generated on a customer by service by region level. - The ETag in the response is not the MD5 of the object data. - The data keys used to encrypt your data are also encrypted and stored alongside the data they protect. - Auditable master keys can be created, rotated, and disabled from the AWS KMS console. - The security controls in AWS KMS can help you meet encryption-related compliance requirements. + Source https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html -To configure chamber to use the S3 KMS backend, use `chamber -b s3-kms --backend-s3-bucket=mybucket --kms-key-alias=alias/keyname`. You must also supply an environment variable of the KMS Key Alias to use CHAMBER_KMS_KEY_ALIAS, by default "alias/parameter_store_key" will be used. +To configure chamber to use the S3 KMS backend, use `chamber -b s3-kms --backend-s3-bucket=mybucket --kms-key-alias=alias/keyname`. You must also supply an environment variable of the KMS Key Alias to use CHAMBER_KMS_KEY_ALIAS, by default "alias/parameter_store_key" will be used. Preferably, this bucket should reject uploads that do not set the server side encryption header ([see this doc for details how](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)) - When changing secrets between KMS Keys, you must first delete the Chamber secret with the existing KMS Key, then write it again with new KMS Key. If services contain multiple KMS Keys, `chamber list` and `chamber exec` will only show Chamber secrets encrypted with KMS Keys you have access to. @@ -284,13 +334,12 @@ If it's preferred to not use any backend at all, use `chamber -b null`. Doing so This feature is experimental, and not currently meant for production work. - ## Analytics -`chamber` includes some usage analytics code which Segment uses internally for tracking usage of internal tools. This analytics code is turned off by default, and can only be enabled via a linker flag at build time, which we do not set for public github releases. +`chamber` includes some usage analytics code which Segment uses internally for tracking usage of internal tools. This analytics code is turned off by default, and can only be enabled via a linker flag at build time, which we do not set for public github releases. ## Releasing To cut a new release, just push a tag named `v` where `` is a -valid semver version. This tag will be used by Github Actions to automatically publish +valid semver version. This tag will be used by Github Actions to automatically publish a github release. diff --git a/cmd/delete.go b/cmd/delete.go index e5f32517..43793501 100644 --- a/cmd/delete.go +++ b/cmd/delete.go @@ -1,10 +1,9 @@ package cmd import ( - "strings" - "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -17,17 +16,24 @@ var deleteCmd = &cobra.Command{ RunE: delete, } +var exactKey bool + func init() { + deleteCmd.Flags().BoolVar(&exactKey, "exact-key", false, "Prevent normalization of the provided key in order to delete any keys that match the exact provided casing.") RootCmd.AddCommand(deleteCmd) } func delete(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } - key := strings.ToLower(args[1]) + key := args[1] + if !exactKey { + key = utils.NormalizeKey(key) + } + if err := validateKey(key); err != nil { return errors.Wrap(err, "Failed to validate key") } diff --git a/cmd/env.go b/cmd/env.go index dfb2546a..841607ac 100644 --- a/cmd/env.go +++ b/cmd/env.go @@ -6,6 +6,7 @@ import ( "strings" "github.com/pkg/errors" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -27,7 +28,7 @@ func init() { } func env(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } diff --git a/cmd/export.go b/cmd/export.go index b0d21a91..120badfe 100644 --- a/cmd/export.go +++ b/cmd/export.go @@ -12,6 +12,7 @@ import ( "github.com/magiconair/properties" "github.com/pkg/errors" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" "gopkg.in/yaml.v3" @@ -59,11 +60,12 @@ func runExport(cmd *cobra.Command, args []string) error { } params := make(map[string]string) for _, service := range args { + service = utils.NormalizeService(service) if err := validateService(service); err != nil { return errors.Wrapf(err, "Failed to validate service %s", service) } - rawSecrets, err := secretStore.ListRaw(strings.ToLower(service)) + rawSecrets, err := secretStore.ListRaw(service) if err != nil { return errors.Wrapf(err, "Failed to list store contents for service %s", service) } diff --git a/cmd/history.go b/cmd/history.go index 123a9e55..c8054fc0 100644 --- a/cmd/history.go +++ b/cmd/history.go @@ -3,11 +3,11 @@ package cmd import ( "fmt" "os" - "strings" "text/tabwriter" "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -25,12 +25,12 @@ func init() { } func history(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } - key := strings.ToLower(args[1]) + key := utils.NormalizeKey(args[1]) if err := validateKey(key); err != nil { return errors.Wrap(err, "Failed to validate key") } diff --git a/cmd/import.go b/cmd/import.go index d65136b0..53f34dad 100644 --- a/cmd/import.go +++ b/cmd/import.go @@ -4,10 +4,10 @@ import ( "fmt" "io" "os" - "strings" "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" "gopkg.in/yaml.v3" @@ -20,14 +20,16 @@ var ( Args: cobra.ExactArgs(2), RunE: importRun, } + normalizeKeys bool ) func init() { + importCmd.Flags().BoolVar(&normalizeKeys, "normalize-keys", false, "Normalize keys to match how `chamber write` would handle them. If not specified, keys will be written exactly how they are defined in the import source.") RootCmd.AddCommand(importCmd) } func importRun(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } @@ -70,6 +72,9 @@ func importRun(cmd *cobra.Command, args []string) error { } for key, value := range toBeImported { + if normalizeKeys { + key = utils.NormalizeKey(key) + } secretId := store.SecretId{ Service: service, Key: key, diff --git a/cmd/list-services.go b/cmd/list-services.go index 48532aff..b322dd09 100644 --- a/cmd/list-services.go +++ b/cmd/list-services.go @@ -4,10 +4,10 @@ import ( "fmt" "os" "sort" - "strings" "text/tabwriter" "github.com/pkg/errors" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" ) @@ -32,7 +32,7 @@ func listServices(cmd *cobra.Command, args []string) error { if len(args) == 0 { service = "" } else { - service = strings.ToLower(args[0]) + service = utils.NormalizeService(args[0]) } secretStore, err := getSecretStore() diff --git a/cmd/list.go b/cmd/list.go index 0bea6a83..79fc16b6 100644 --- a/cmd/list.go +++ b/cmd/list.go @@ -9,6 +9,7 @@ import ( "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -37,7 +38,7 @@ func init() { } func list(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateServiceWithLabel(service); err != nil { return errors.Wrap(err, "Failed to validate service") } diff --git a/cmd/read.go b/cmd/read.go index 6e8bcb3b..464b7394 100644 --- a/cmd/read.go +++ b/cmd/read.go @@ -3,11 +3,11 @@ package cmd import ( "fmt" "os" - "strings" "text/tabwriter" "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -32,12 +32,12 @@ func init() { } func read(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } - key := strings.ToLower(args[1]) + key := utils.NormalizeKey(args[1]) if err := validateKey(key); err != nil { return errors.Wrap(err, "Failed to validate key") } diff --git a/cmd/write.go b/cmd/write.go index b922960b..3a53d419 100644 --- a/cmd/write.go +++ b/cmd/write.go @@ -8,6 +8,7 @@ import ( "github.com/pkg/errors" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" "github.com/spf13/cobra" analytics "gopkg.in/segmentio/analytics-go.v3" ) @@ -32,12 +33,12 @@ func init() { } func write(cmd *cobra.Command, args []string) error { - service := strings.ToLower(args[0]) + service := utils.NormalizeService(args[0]) if err := validateService(service); err != nil { return errors.Wrap(err, "Failed to validate service") } - key := strings.ToLower(args[1]) + key := utils.NormalizeKey(args[1]) if err := validateKey(key); err != nil { return errors.Wrap(err, "Failed to validate key") } diff --git a/environ/environ.go b/environ/environ.go index 9fc36eab..52f40001 100644 --- a/environ/environ.go +++ b/environ/environ.go @@ -5,6 +5,7 @@ import ( "strings" "github.com/segmentio/chamber/v2/store" + "github.com/segmentio/chamber/v2/utils" ) // environ is a slice of strings representing the environment, in the form "key=value". @@ -87,7 +88,7 @@ func normalizeEnvVarName(k string) string { // collisions will be populated with any keys that get overwritten // noPaths enables the behavior as if CHAMBER_NO_PATHS had been set func (e *Environ) load(s store.Store, service string, collisions *[]string, noPaths bool) error { - rawSecrets, err := s.ListRaw(strings.ToLower(service)) + rawSecrets, err := s.ListRaw(utils.NormalizeService(service)) if err != nil { return err } @@ -134,7 +135,7 @@ func (e *Environ) LoadStrictNoPaths(s store.Store, valueExpected string, pristin func (e *Environ) loadStrict(s store.Store, valueExpected string, pristine bool, noPaths bool, services ...string) error { for _, service := range services { - rawSecrets, err := s.ListRaw(strings.ToLower(service)) + rawSecrets, err := s.ListRaw(utils.NormalizeService(service)) if err != nil { return err } diff --git a/utils/utils.go b/utils/utils.go new file mode 100644 index 00000000..b5c0dd91 --- /dev/null +++ b/utils/utils.go @@ -0,0 +1,13 @@ +package utils + +import "strings" + +// NormalizeService normalizes a provided service to a common format +func NormalizeService(service string) string { + return strings.ToLower(service) +} + +// NormalizeKey normalizes a provided secret key to a common format +func NormalizeKey(key string) string { + return strings.ToLower(key) +} diff --git a/utils/utils_test.go b/utils/utils_test.go new file mode 100644 index 00000000..f5f10bfd --- /dev/null +++ b/utils/utils_test.go @@ -0,0 +1,46 @@ +package utils + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestNormalizeService(t *testing.T) { + testCases := []struct { + service string + expected string + }{ + {"service", "service"}, + {"service-with-hyphens", "service-with-hyphens"}, + {"service_with_underscores", "service_with_underscores"}, + {"UPPERCASE_SERVICE", "uppercase_service"}, + {"mIXedcase-SERvice", "mixedcase-service"}, + {".complex/service-CASE", ".complex/service-case"}, + } + + for _, testCase := range testCases { + t.Run(testCase.service, func(t *testing.T) { + assert.Equal(t, testCase.expected, NormalizeService(testCase.service)) + }) + } +} + +func TestNormalizeKey(t *testing.T) { + testCases := []struct { + key string + expected string + }{ + {"key", "key"}, + {"key-with-hyphens", "key-with-hyphens"}, + {"key_with_underscores", "key_with_underscores"}, + {"UPPERCASE_KEY", "uppercase_key"}, + {"mIXedcase-Key", "mixedcase-key"}, + } + + for _, testCase := range testCases { + t.Run(testCase.key, func(t *testing.T) { + assert.Equal(t, testCase.expected, NormalizeKey(testCase.key)) + }) + } +}