diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ab3fb2c..7d2d7e9f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,16 @@ # Changelog -## master (unreleased) +## v0.9.0 (February 6, 2021) + +Big thank you to [mattlqx](https://github.com/mattlqx) for the great enhancements. ENHANCEMENTS: +* Proper arg parsing with help text for subcommands ([#73](https://github.com/fishi0x01/vsh/pull/73) - Thank you for implementation [mattlqx](https://github.com/mattlqx)) +* Add replace command ([#69](https://github.com/fishi0x01/vsh/pull/69) - Thank you for implementation [mattlqx](https://github.com/mattlqx)) +* Add key selector to replace command ([#72](https://github.com/fishi0x01/vsh/pull/72) - Thank you for implementation [mattlqx](https://github.com/mattlqx)) * Allow limiting scope of grep to keys or values ([#66](https://github.com/fishi0x01/vsh/pull/66) - Thank you for implementation [mattlqx](https://github.com/mattlqx)) -* Do not show and operate on KV2 metadata ([#68](https://github.com/fishi0x01/vsh/pull/68) +* Do not show and operate on KV2 metadata ([#68](https://github.com/fishi0x01/vsh/pull/68)) ## v0.8.0 (January 27, 2021) diff --git a/README.md b/README.md index a388da66..e2758275 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,10 @@ ![vsh usage](https://user-images.githubusercontent.com/10799507/66355982-9872a980-e969-11e9-8ca4-6a2ff215f835.gif) -`vsh` is an interactive [HashiCorp Vault](https://www.vaultproject.io/) shell which treats paths and keys like directories and files. +`vsh` is an interactive [HashiCorp Vault](https://www.vaultproject.io/) shell and cli tool. It comes with multiple common operations and treats paths like directories and files. Core features are: -- recursive operations on paths with `cp`, `mv` or `rm` +- recursive operations on paths for many operations, e.g., `cp`, `rm`, `mv` - search with `grep` (substring or regular-expression) - substitute patterns in keys and/or values (substring or regular-expression) with `replace` - transparency towards differences between KV1 and KV2, i.e., you can freely move/copy secrets between both @@ -37,115 +37,33 @@ Download latest static binaries from [release page](https://github.com/fishi0x01 ## Supported commands -```text -append [flag] -cat -cd -cp -grep [-e|--regexp] [-k|--keys] [-v|--values] -ls -mv -replace [-e|--regexp] [-k|--keys] [-v|--values] [-y|--confirm] [-n|--dry-run] -rm -``` - -`cp`, `grep`, `replace` and `rm` command always have the `-r/-R` flag implied, i.e., every operation works recursively. - -### append - -Append operation reads secrets from `` and merges it to ``. -The `` will be created with a placeholder value if it does not exists. -Both `` and `` must be leaves (path cannot end with `/`). - -By default, `append` does not overwrite secrets if the `` already contains a key. -The default behavior can be explicitly set using flag: `-s` or `--skip`. Example: - -```bash -> cat /secret/from - -fruit=apple -vegetable=tomato - -> cat /secret/to - -fruit=pear -tree=oak - -> append /secret/from /secret/to -s - -> cat /secret/to - -fruit=pear -vegetable=tomato -tree=oak -``` - -Setting flag `-f` or `--force` will cause the conflicting keys from the `` to be overwritten with keys from the `. Example: - -```bash -> cat /secret/from - -fruit=apple -vegetable=tomato - -> cat /secret/to - -fruit=pear -tree=oak - -> append /secret/from /secret/to -f - -> cat /secret/to - -fruit=apple -vegetable=tomato -tree=oak -``` - -Setting flag `-r` or `--rename` will cause the conflicting keys from the `` to be kept as they are. Instead the keys from the ` will be stored under a renamed key. Example: +- [append](doc/commands/append.md) merges secrets with different strategies (allows recursive operation on paths) +- [cat](doc/commands/cat.md) shows the key/value pairs of a path +- [cd](doc/commands/cd.md) allows interactive navigation through the paths +- [cp](doc/commands/cp.md) copies secrets from one location to another (allows recursive operation on paths) +- [grep](doc/commands/grep.md) searches for substrings or regular expressions (allows recursive operation on paths) +- [ls](doc/commands/ls.md) shows the subpaths of a given path +- [mv](doc/commands/mv.md) moves secrets from one location to another (allows recursive operation on paths) +- [replace](doc/commands/replace.md) substrings or regular expressions (allows recursive operation on paths) +- [rm](doc/commands/rm.md) removes secret(s) (allows recursive operation on paths) -```bash -> cat /secret/from - -fruit=apple -vegetable=tomato - -> cat /secret/to - -fruit=pear -tree=oak - -> append /secret/from /secret/to -r - -> cat /secret/to - -fruit=pear -fruit_1=apple -vegetable=tomato -tree=oak -``` - -### grep - -`grep` recursively searches the given substring in key and value pairs. To treat the search string as a regular-expression, add `-e` or `--regexp` to the end of the command. By default, both keys and values will be searched. If you would like to limit the search, you may add `-k` or `--keys` to the end of the command to search only a path's keys, or `-v` or `--values` to search only a path's values. - If you are looking for copies or just trying to find the path to a certain string, this command might come in handy. - -### replace +## Setting the vault token -`replace` works similarly to `grep` above, but has the ability to mutate data inside Vault. By default, confirmation is required before writing data. You may skip confirmation by using the `-y`/`--confirm` flags. Conversely, you may use the `-n`/`--dry-run` flags to skip both confirmation and any writes. Changes that would be made are presented in red (delete) and green (add) coloring. +In order to get a valid token, `vsh` uses vault's TokenHelper mechanism. +That means `vsh` supports setting vault tokens via `~/.vault-token`, `VAULT_TOKEN` and external [token-helper](https://www.vaultproject.io/docs/commands/token-helper). -## Setting the vault token +## Token permission requirements -In order to get a valid token, `vsh` uses vault's TokenHelper mechanism (`github.com/hashicorp/vault/command/config`). -That means `vsh` supports setting vault tokens via `~/.vault-token`, `VAULT_TOKEN` and external `token_helper`. +`vsh` requires `List` permission on the operated paths. +This is necessary to determine if a path points to a node or leaf in the path tree. +Further, it is needed to gather auto-completion data. -## Secret Backend Discovery +Commands which alter the data like `cp` or `mv`, additionally require `Read` and `Write` permissions on the operated paths. -`vsh` attempts to reliably discover all available backends. -Ideally, the vault token used by `vsh` has `list` permissions on `sys/mount`. -If this is not the case, then `vsh` does not know the available backends beforehand. +In order to reliably discover all available backends, ideally the vault token used by `vsh` has `List` permission on `sys/mount`. However, this is not a hard requirement. +If the token doesn't have `List` permission on `sys/mount`, then `vsh` does not know the available backends beforehand. That means initially there won't be path auto-completion on the top (backend) level. -However, `vsh` will try with best-effort strategy to reliably determine the kv version of every entered path. +Regardless, `vsh` will try with best-effort strategy to reliably determine the kv version of every entered path. ## Interactive mode @@ -160,7 +78,7 @@ http://localhost:8080 /secret/> **Note:** the given token is used for auto-completion, i.e., `List()` queries are done with that token, even if you do not `rm` or `mv` anything. `vsh` caches `List()` results to reduce the amount of queries. However, after execution of each command the cache is cleared in order to do accurate tab-completion. -If your token has a limited number of uses, then consider using the non-interactive mode to avoid auto-completion queries. +If your token has a limited number of uses, then consider using the non-interactive mode or toggle auto-completion off, to avoid `List()` queries. ### Toggle auto-completion @@ -190,27 +108,23 @@ export VAULT_TOKEN= ./vsh -c "rm secret/dir/to/remove/" ``` -## Permission requirements - -`vsh` requires `List` permission on the operated paths. -This is necessary to determine if a path points to a node or leaf in the path tree. -Further, it is needed to gather auto-completion data. - -For operations like `cp` or `mv`, `vsh` additionally requires `Read` and `Write` permissions on the operated paths. - -## Quality +## Some words about the quality Working on vault secrets can be critical, making quality and correct behavior a first class citizen for `vsh`. -That being said, `vsh` is still a small open source project, meaning we cannot make any guarantees. -However, we put strong emphasis on [TDD](https://en.wikipedia.org/wiki/Test-driven_development). +That being said, `vsh` is still a small open source project, meaning we cannot give any guarantees. +However, we put strong emphasis on test-driven development. Every PR is tested with an extensive [suite](test/suites) of integration tests. -Most tests run on KV1 and KV2 and every test runs against vault `1.0.0` and `1.6.1`, i.e., versions in between should also be compatible. +Vast majority of tests run on KV1 and KV2 and every test runs against vault `1.0.0` and `1.6.2`, i.e., vault versions in between should also be compatible. + +## Contributions + +Contributions in any form are always welcome! Without contributions from the community, `vsh` wouldn't be the tool it is today. -## Local Development +### Local Development Requirements: -- `golang` (compiled and tested with `v1.15.3`) +- `golang` (compiled and tested with `v1.15.7`) - `docker` for integration testing - `make` for simplified commands @@ -220,6 +134,6 @@ make get-bats make integration-tests ``` -## Debugging +### Debugging `-v DEBUG` sets debug log level, which also creates a `vsh_trace.log` file to log any error object from the vault API. diff --git a/doc/commands/append.md b/doc/commands/append.md new file mode 100644 index 00000000..3bcd57ff --- /dev/null +++ b/doc/commands/append.md @@ -0,0 +1,77 @@ +# append + +```text +append [-f|--force] [-s|--skip] [-r|--rename] SOURCE TARGET +``` + +Append operation reads secrets from `SOURCE` and merges it to `TARGET`. +The `TARGET` will be created with a placeholder value if it does not exists. +Both `SOURCE` and `TARGET` must be leaves (path cannot end with `/`). + +By default, `append` does not overwrite secrets if the `TARGET` already contains a key. +The default behavior can be explicitly set using flag: `-s` or `--skip`. Example: + +```bash +> cat /secret/from + +fruit=apple +vegetable=tomato + +> cat /secret/to + +fruit=pear +tree=oak + +> append --skip /secret/from /secret/to + +> cat /secret/to + +fruit=pear +vegetable=tomato +tree=oak +``` + +Setting flag `-f` or `--force` will cause the conflicting keys from the `` to be overwritten with keys from the `. Example: + +```bash +> cat /secret/from + +fruit=apple +vegetable=tomato + +> cat /secret/to + +fruit=pear +tree=oak + +> append -f /secret/from /secret/to + +> cat /secret/to + +fruit=apple +vegetable=tomato +tree=oak +``` + +Setting flag `-r` or `--rename` will cause the conflicting keys from the `` to be kept as they are. Instead the keys from the ` will be stored under a renamed key. Example: + +```bash +> cat /secret/from + +fruit=apple +vegetable=tomato + +> cat /secret/to + +fruit=pear +tree=oak + +> append -r /secret/from /secret/to + +> cat /secret/to + +fruit=pear +fruit_1=apple +vegetable=tomato +tree=oak +``` diff --git a/doc/commands/cat.md b/doc/commands/cat.md new file mode 100644 index 00000000..b942bfe3 --- /dev/null +++ b/doc/commands/cat.md @@ -0,0 +1,7 @@ +# cat + +```text +cat PATH +``` + +Show the keys/values of the given `PATH`. diff --git a/doc/commands/cd.md b/doc/commands/cd.md new file mode 100644 index 00000000..7c1dd373 --- /dev/null +++ b/doc/commands/cd.md @@ -0,0 +1,7 @@ +# cd + +```text +cd PATH +``` + +Interactively navigate to given `PATH`. diff --git a/doc/commands/cp.md b/doc/commands/cp.md new file mode 100644 index 00000000..07d61b42 --- /dev/null +++ b/doc/commands/cp.md @@ -0,0 +1,7 @@ +# cp + +```text +cp SOURCE TARGET +``` + +Copy `SOURCE` path to `TARGET` path. If executed on a node, (i.e., a path ending with `/`), then copy is applied recursively. diff --git a/doc/commands/grep.md b/doc/commands/grep.md new file mode 100644 index 00000000..22aca442 --- /dev/null +++ b/doc/commands/grep.md @@ -0,0 +1,8 @@ +# grep + +```text +grep [-e|--regexp] [-k|--keys] [-v|--values] SEARCH PATH +``` + +`grep` recursively searches the given `SEARCH` substring in key and value pairs of given `PATH`. To treat the search string as a regular-expression, add `-e` or `--regexp` to the end of the command. By default, both keys and values will be searched. If you would like to limit the search, you may add `-k` or `--keys` to the end of the command to search only a path's keys, or `-v` or `--values` to search only a path's values. + If you are looking for copies or just trying to find the path to a certain string, this command might come in handy. diff --git a/doc/commands/ls.md b/doc/commands/ls.md new file mode 100644 index 00000000..f819fd79 --- /dev/null +++ b/doc/commands/ls.md @@ -0,0 +1,7 @@ +# ls + +```text +ls PATH +``` + +List subpaths of given `PATH`. diff --git a/doc/commands/mv.md b/doc/commands/mv.md new file mode 100644 index 00000000..9ca1b80d --- /dev/null +++ b/doc/commands/mv.md @@ -0,0 +1,7 @@ +# mv + +```text +mv SOURCE TARGET +``` + +Move `SOURCE` path to `TARGET` path. If executed on a node, (i.e., a path ending with `/`), then move is applied recursively. diff --git a/doc/commands/replace.md b/doc/commands/replace.md new file mode 100644 index 00000000..4d7f80cf --- /dev/null +++ b/doc/commands/replace.md @@ -0,0 +1,3 @@ +# replace + +`replace` works similarly to `grep`, but has the ability to mutate data inside Vault. By default, confirmation is required before writing data. You may skip confirmation by using the `-y`/`--confirm` flags. Conversely, you may use the `-n`/`--dry-run` flags to skip both confirmation and any writes. Changes that would be made are presented in red (delete) and green (add) coloring. diff --git a/doc/commands/rm.md b/doc/commands/rm.md new file mode 100644 index 00000000..2b6856ac --- /dev/null +++ b/doc/commands/rm.md @@ -0,0 +1,7 @@ +# rm + +```text +rm PATH +``` + +Remove `PATH`. If executed on a node, (i.e., a path ending with `/`), then remove is applied recursively.