feat(tools): add yq (#2406)

## Description Adds `zarf tools yq` ## Related Issue Fixes #2384 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [x] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed
defenseunicorns · Mar 27, 2024 · 9740ba8 · 9740ba8
1 parent ac0c058
commit 9740ba8
Show file tree

Hide file tree

Showing 13 changed files with 410 additions and 8 deletions.
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools.md
@@ -38,3 +38,4 @@ Collection of additional tools to make airgap easier
 * [zarf tools sbom](zarf_tools_sbom.md)	 - Generates a Software Bill of Materials (SBOM) for the given package
 * [zarf tools update-creds](zarf_tools_update-creds.md)	 - Updates the credentials for deployed Zarf services. Pass a service key to update credentials for a single service
 * [zarf tools wait-for](zarf_tools_wait-for.md)	 - Waits for a given Kubernetes resource to be ready
+* [zarf tools yq](zarf_tools_yq.md)	 - yq is a lightweight and portable command-line data file processor.
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq.md
@@ -0,0 +1,81 @@
+# zarf tools yq
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+yq is a lightweight and portable command-line data file processor.
+
+## Synopsis
+
+yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
+
+```
+zarf tools yq [flags]
+```
+
+## Examples
+
+```
+
+# yq defaults to 'eval' command if no command is specified. See "zarf tools yq eval --help" for more examples.
+
+# read the "stuff" node from "myfile.yml"
+zarf tools yq '.stuff' < myfile.yml
+
+# update myfile.yml in place
+zarf tools yq -i '.stuff = "foo"' myfile.yml
+
+# print contents of sample.json as idiomatic YAML
+zarf tools yq -P sample.json
+
+```
+
+## Options
+
+```
+  -C, --colors                        force print with colors
+      --csv-auto-parse                parse CSV YAML/JSON values (default true)
+      --csv-separator char            CSV Separator character (default ,)
+  -e, --exit-status                   set exit status if there are no matches or null or false is returned
+      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string              Load expression from specified file.
+  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
+      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
+  -h, --help                          help for yq
+  -I, --indent int                    sets indent level for output (default 2)
+  -i, --inplace                       update the file in place of first file given.
+  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l] parse format for input. (default "auto")
+      --lua-globals                   output keys as top-level global variables
+      --lua-prefix string             prefix (default "return ")
+      --lua-suffix string             suffix (default ";\n")
+      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
+  -M, --no-colors                     force print with no colors
+  -N, --no-doc                        Don't print document separators (---)
+  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
+  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
+  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l] output format type. (default "auto")
+  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
+      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string   separator to use between keys and values (default " = ")
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
+      --split-exp-file string         Use a file to specify the split-exp expression.
+      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -v, --verbose                       verbose mode
+  -V, --version                       Print version information and quit
+      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
+      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
+```
+
+## SEE ALSO
+
+* [zarf tools](zarf_tools.md)	 - Collection of additional tools to make airgap easier
+* [zarf tools yq eval](zarf_tools_yq_eval.md)	 - (default) Apply the expression to each document in each yaml file in sequence
+* [zarf tools yq eval-all](zarf_tools_yq_eval-all.md)	 - Loads _all_ yaml documents of _all_ yaml files and runs expression once
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq_eval-all.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq_eval-all.md
@@ -0,0 +1,91 @@
+# zarf tools yq eval-all
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+Loads _all_ yaml documents of _all_ yaml files and runs expression once
+
+## Synopsis
+
+yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
+
+# Evaluate All ##
+This command loads _all_ yaml documents of _all_ yaml files and runs expression once
+Useful when you need to run an expression across several yaml documents or files (like merge).
+Note that it consumes more memory than eval.
+
+
+```
+zarf tools yq eval-all [expression] [yaml_file1]... [flags]
+```
+
+## Examples
+
+```
+
+# Merge f2.yml into f1.yml (inplace)
+zarf tools yq eval-all --inplace 'select(fileIndex == 0) * select(fileIndex == 1)' f1.yml f2.yml
+# the same command and expression using shortened names:
+zarf tools yq ea -i 'select(fi == 0) * select(fi == 1)' f1.yml f2.yml
+
+
+# Merge all given files
+zarf tools yq ea '. as $item ireduce ({}; . * $item )' file1.yml file2.yml ...
+
+# Pipe from STDIN
+# use '-' as a filename to pipe from STDIN
+cat file2.yml | zarf tools yq ea '.a.b' file1.yml - file3.yml
+
+```
+
+## Options
+
+```
+  -h, --help   help for eval-all
+```
+
+## Options inherited from parent commands
+
+```
+  -C, --colors                        force print with colors
+      --csv-auto-parse                parse CSV YAML/JSON values (default true)
+      --csv-separator char            CSV Separator character (default ,)
+  -e, --exit-status                   set exit status if there are no matches or null or false is returned
+      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string              Load expression from specified file.
+  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
+      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
+  -I, --indent int                    sets indent level for output (default 2)
+  -i, --inplace                       update the file in place of first file given.
+  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l] parse format for input. (default "auto")
+      --lua-globals                   output keys as top-level global variables
+      --lua-prefix string             prefix (default "return ")
+      --lua-suffix string             suffix (default ";\n")
+      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
+  -M, --no-colors                     force print with no colors
+  -N, --no-doc                        Don't print document separators (---)
+  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
+  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
+  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l] output format type. (default "auto")
+  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
+      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string   separator to use between keys and values (default " = ")
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
+      --split-exp-file string         Use a file to specify the split-exp expression.
+      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -v, --verbose                       verbose mode
+      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
+      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
+```
+
+## SEE ALSO
+
+* [zarf tools yq](zarf_tools_yq.md)	 - yq is a lightweight and portable command-line data file processor.
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq_eval.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_yq_eval.md
@@ -0,0 +1,93 @@
+# zarf tools yq eval
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+(default) Apply the expression to each document in each yaml file in sequence
+
+## Synopsis
+
+yq is a portable command-line data file processor (https://github.com/mikefarah/yq/) 
+See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.
+
+# Evaluate Sequence ##
+This command iterates over each yaml document from each given file, applies the 
+expression and prints the result in sequence.
+
+```
+zarf tools yq eval [expression] [yaml_file1]... [flags]
+```
+
+## Examples
+
+```
+
+# Reads field under the given path for each file
+zarf tools yq e '.a.b' f1.yml f2.yml 
+
+# Prints out the file
+zarf tools yq e sample.yaml 
+
+# Pipe from STDIN
+# use '-' as a filename to pipe from STDIN
+cat file2.yml | zarf tools yq e '.a.b' file1.yml - file3.yml
+
+# Creates a new yaml document
+# Note that editing an empty file does not work.
+zarf tools yq e -n '.a.b.c = "cat"' 
+
+# Update a file inplace
+zarf tools yq e '.a.b = "cool"' -i file.yaml 
+
+```
+
+## Options
+
+```
+  -h, --help   help for eval
+```
+
+## Options inherited from parent commands
+
+```
+  -C, --colors                        force print with colors
+      --csv-auto-parse                parse CSV YAML/JSON values (default true)
+      --csv-separator char            CSV Separator character (default ,)
+  -e, --exit-status                   set exit status if there are no matches or null or false is returned
+      --expression string             forcibly set the expression argument. Useful when yq argument detection thinks your expression is a file.
+      --from-file string              Load expression from specified file.
+  -f, --front-matter string           (extract|process) first input as yaml front-matter. Extract will pull out the yaml content, process will run the expression against the yaml content, leaving the remaining data intact
+      --header-preprocess             Slurp any header comments and separators before processing expression. (default true)
+  -I, --indent int                    sets indent level for output (default 2)
+  -i, --inplace                       update the file in place of first file given.
+  -p, --input-format string           [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|lua|l] parse format for input. (default "auto")
+      --lua-globals                   output keys as top-level global variables
+      --lua-prefix string             prefix (default "return ")
+      --lua-suffix string             suffix (default ";\n")
+      --lua-unquoted                  output unquoted string keys (e.g. {foo="bar"})
+  -M, --no-colors                     force print with no colors
+  -N, --no-doc                        Don't print document separators (---)
+  -0, --nul-output                    Use NUL char to separate values. If unwrap scalar is also set, fail if unwrapped scalar contains NUL char.
+  -n, --null-input                    Don't read input, simply evaluate the expression given. Useful for creating docs from scratch.
+  -o, --output-format string          [auto|a|yaml|y|json|j|props|p|csv|c|tsv|t|xml|x|base64|uri|toml|shell|s|lua|l] output format type. (default "auto")
+  -P, --prettyPrint                   pretty print, shorthand for '... style = ""'
+      --properties-array-brackets     use [x] in array paths (e.g. for SpringBoot)
+      --properties-separator string   separator to use between keys and values (default " = ")
+  -s, --split-exp string              print each result (or doc) into a file named (exp). [exp] argument must return a string. You can use $index in the expression as the result counter.
+      --split-exp-file string         Use a file to specify the split-exp expression.
+      --string-interpolation          Toggles strings interpolation of \(exp) (default true)
+      --tsv-auto-parse                parse TSV YAML/JSON values (default true)
+  -r, --unwrapScalar                  unwrap scalar, print the value with no quotes, colors or comments. Defaults to true for yaml (default true)
+  -v, --verbose                       verbose mode
+      --xml-attribute-prefix string   prefix for xml attributes (default "+@")
+      --xml-content-name string       name for xml content (if no attribute name is present). (default "+content")
+      --xml-directive-name string     name for xml directives (e.g. <!DOCTYPE thing cat>) (default "+directive")
+      --xml-keep-namespace            enables keeping namespace after parsing attributes (default true)
+      --xml-proc-inst-prefix string   prefix for xml processing instructions (e.g. <?xml version="1"?>) (default "+p_")
+      --xml-raw-token                 enables using RawToken method instead Token. Commonly disables namespace translations. See https://pkg.go.dev/encoding/xml#Decoder.RawToken for details. (default true)
+      --xml-skip-directives           skip over directives (e.g. <!DOCTYPE thing cat>)
+      --xml-skip-proc-inst            skip over process instructions (e.g. <?xml version="1"?>)
+      --xml-strict-mode               enables strict parsing of XML. See https://pkg.go.dev/encoding/xml for more details.
+```
+
+## SEE ALSO
+
+* [zarf tools yq](zarf_tools_yq.md)	 - yq is a lightweight and portable command-line data file processor.