Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: New S3 Backend for using KMS Keys as S3 Encryption Method #205

Merged
merged 26 commits into from
Jul 8, 2019

Commits on Mar 11, 2019

  1. Use KMS Encryption instead of AES256

    Switch from using AES256 to KMS for storing individual keys and the latest.json file. The SSM Store supports KMS Encryption and allows more segmentation for which keys a user is allowed to read.
    Matthew Borden committed Mar 11, 2019
    Configuration menu
    Copy the full SHA
    697238d View commit details
    Browse the repository at this point in the history
  2. Store Latest values for a service by KMS Key

    Store the latest keys for each service by the KMS key they were written with. Doing so allows for Chamber to use KMS Encryption on the latest file as well as the individual keys in the service.
    Matthew Borden committed Mar 11, 2019
    Configuration menu
    Copy the full SHA
    93e038b View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2019

  1. Store KMS Key and Version in __latest.json File

    Store the KMS Key and the version in the latest file. This allows us to combine the latest files together (even though there is a __latest.json file per KMS Key).
    Matthew Borden committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    deea34d View commit details
    Browse the repository at this point in the history
  2. Don't allow writing or deleting with a different KMS Key.

    Instead of attempting to migrate between KMS Keys and leaving the first KMS's latest file in an inaccurate state, we ask users to delete and recreate the chamber secret with the new key.
    Matthew Borden committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    d34076b View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2019

  1. Print current KMS key when asking users to delete Chamber secrets

    Before writing an existing secret with a new KMS Key, ask the user to delete the existing Chamber secret with the current key. Tell them which KMS key the secret is currently encrypted with.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    7101301 View commit details
    Browse the repository at this point in the history
  2. Handle errors attempting to read per KMS key latest file.

    When attempting to read the __latest.json for a KMS Key, treat AccessDenied as the file not existing. We do this because the user may not be able to read keys in the service that are encrypted with a KMS Key they don't have access to.
    
    If we get a different kind of error then panic, this isn't my prefered way to handle the error but the S3 pages API doesn't allow to pass back err as a result.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    3a7c572 View commit details
    Browse the repository at this point in the history
  3. Create new S3-KMS Backend

    The changes I've made to the __latest.json format (both schema changes and made multiple of these files, one per KMS Key used) are not backwards compadible with the previous implementation of the S3 backend.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    7e7b3c5 View commit details
    Browse the repository at this point in the history
  4. Add S3-KMS Backend to README.md

    Detail how to use it and it's features.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    a9cd871 View commit details
    Browse the repository at this point in the history
  5. Run go fmt

    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    02e5cc3 View commit details
    Browse the repository at this point in the history
  6. Use traditional error handling instead of panic.

    This seems to be more traditional and I was just working around the lack of err in the AWS ListObjectsPages method.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    05e1b23 View commit details
    Browse the repository at this point in the history
  7. Prefix Latest File Names with __kms

    To avoid treating any secret names that start with kms as index files.
    Matthew Borden committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    9adfc92 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2019

  1. Remove copy-pasta'd legacy NewS3KMSStore method

    I coppied this from the S3Store interface but this was only being used as a proxy for NewS3StoreWithBucket. Since this backend is brand new it can just have the NewS3KMSStoreWithBucket method and doesn't need to expose the legacy method.
    Matthew Borden committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    e55aa70 View commit details
    Browse the repository at this point in the history
  2. Remove duplicated functions that can be shared with S3Store

    There is a lot of overlap bettween the S3Store and S3KMSStore (because I coppied and pasted the entire thing to begin with). I'm removing a lot of the shared code and just using method overriding for the methods that have changes.
    Matthew Borden committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    cbb5b4d View commit details
    Browse the repository at this point in the history
  3. Cleanup KMS Key Alias to S3KMSStore

    Previously we used an environment variable or set a default. However, in order to added a flag to the UI it would be cleaner to pass in into the store interface.
    Matthew Borden committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    8d5bd49 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2019

  1. Allow setting KMS key via --kms-key-alias command line arguement

    To support making the command line more usable (and removing the setup of environment being the main interface) the S3-KMS backend now allows setting the KMS Key Alias via the command line interface. The environment variable will be prefered.
    Matthew Borden committed Jun 23, 2019
    Configuration menu
    Copy the full SHA
    d518950 View commit details
    Browse the repository at this point in the history
  2. Revert "Remove duplicated functions that can be shared with S3Store"

    This reverts commit cbb5b4d.
    
    This broke the List operations (because it removed the overriding of the base class. This caused an issue because the List operation now searches for keys in multiple kms key latest index files).
    Matthew Borden committed Jun 23, 2019
    Configuration menu
    Copy the full SHA
    72b1f25 View commit details
    Browse the repository at this point in the history
  3. Remove duplicated history method

    This can be shared with the S3Store interface via function overriding.
    Matthew Borden committed Jun 23, 2019
    Configuration menu
    Copy the full SHA
    e735a8a View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2019

  1. Establish an S3Store within S3KMS Store and remove more duplicated fu…

    …nctions
    
    Instead of maintaining two sets of code for the S3KMSStore and S3Store, instead share functions that are similar between the modules (eg, history, read).
    The biggest differences are in List (Where we index of multiple latest files in the KMS version but only one in the S3Store), Write and Delete (where we must handle writing with different keys).
    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    6e4ecae View commit details
    Browse the repository at this point in the history
  2. Remove more shareable functions with S3Store

    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    603b6a7 View commit details
    Browse the repository at this point in the history
  3. Allow setting KMS without "alias/" prefix

    All kms key aliases need to be prefixed with "alias/"keyname
    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    7cdc6f3 View commit details
    Browse the repository at this point in the history
  4. GOFMT

    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    6106e51 View commit details
    Browse the repository at this point in the history
  5. Use --kms-key-alias for SSM Backend too

    Allow passing in the KMS Key Alias to write and delete keys with to the SSM Backend as well as the S3KMSStore
    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    0cfab16 View commit details
    Browse the repository at this point in the history
  6. Use the default parameter store key when KMS key not set.

    Matthew Borden committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    4967538 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2019

  1. Revert "Use the default parameter store key when KMS key not set."

    This reverts commit 4967538.
    Matthew Borden committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    bda34b2 View commit details
    Browse the repository at this point in the history
  2. Revert "Use --kms-key-alias for SSM Backend too"

    This reverts commit 0cfab16.
    Matthew Borden committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    20cfb62 View commit details
    Browse the repository at this point in the history
  3. Explode if --kms-key-alias is used with unsupported backends.

    Explode when --kms-key-alias is used with other backends that don't support it.
    Matthew Borden committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    79a2569 View commit details
    Browse the repository at this point in the history