Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add file type variables to bundles #631

Merged
merged 33 commits into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bf334e6
feat: add file type variables to bundles
TristanHoladay May 22, 2024
fbf8229
add FileVariables to BundleDeployOptions.
TristanHoladay May 22, 2024
3a63013
adding loadFileContents() to loadVariables().
TristanHoladay May 22, 2024
0b6f219
add file handling from loadVariables()
TristanHoladay May 23, 2024
1e271cd
fix deploy_test call of loadVariables()
TristanHoladay May 23, 2024
405327e
test for zarf var setting; adding file handling to processOverrideVar…
TristanHoladay May 24, 2024
d5b375b
small refactor; attempting helm.FileValues (bad format errors)
TristanHoladay May 24, 2024
5c97215
cleanup helm file handling; base64 test pub key.
TristanHoladay May 28, 2024
9a835a3
lint fixes
TristanHoladay May 28, 2024
7dd839b
merge main and resolve; refactor file var handling
TristanHoladay May 29, 2024
39d2187
fix loadViperConfig duplicate; fix file handling in override add
TristanHoladay May 29, 2024
cd23c1a
refactor ChartVariableType; refactor tests.
TristanHoladay May 29, 2024
bec07c9
lint fix
TristanHoladay May 29, 2024
a2fe420
file path handling made more reusable; fixed tests.
TristanHoladay May 29, 2024
dc8613c
Merge branch 'main' into add-file-type-var
TristanHoladay May 29, 2024
3a407ec
fix typo in test
TristanHoladay May 29, 2024
a5afc3e
Merge branch 'main' into add-file-type-var
TristanHoladay May 29, 2024
513ed8e
lint fix'
TristanHoladay May 29, 2024
daf9076
add docs; make test less brittle.
TristanHoladay May 29, 2024
05080fa
add to docs; rename test
TristanHoladay May 29, 2024
a2b4276
move formAndCheckFilePath and annotate
TristanHoladay May 30, 2024
2d11b46
file handling and docs refactors
TristanHoladay May 31, 2024
bc7d4d0
handle different value source paths.
TristanHoladay Jun 3, 2024
18b13ec
refactor tests.
TristanHoladay Jun 3, 2024
ee2e768
update schema and fix test conflicts
TristanHoladay Jun 3, 2024
3c420ce
change ValueSource to Source to remove confusion with type.
TristanHoladay Jun 3, 2024
6d2914a
fix wrong property name in unit test
TristanHoladay Jun 3, 2024
d72cde9
revert e2e secret test removals
TristanHoladay Jun 3, 2024
33daef1
clean up tests from previous iterations
TristanHoladay Jun 4, 2024
861f15f
lifted unit tests up to processOverrideVariables
TristanHoladay Jun 4, 2024
85a5ad3
check override map in unit test for correct FileValues
TristanHoladay Jun 4, 2024
e995e65
Update src/pkg/bundle/deploy_test.go
TristanHoladay Jun 5, 2024
9b457c9
switch Equals to Equal
TristanHoladay Jun 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/overrides.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,56 @@ Variable precedence is as follows:
1. `uds-config.yaml` variables
1. Variables `default` in the`uds-bundle.yaml`

#### Variable Types
Variables can be of either type `raw` or `file`. The type will default to raw if not set explicitly.

> [!WARNING]
> If a variable is set to accept a file as its value, but is missing the `file` type, then the file will not be processed.

```yaml
kind: UDSBundle
metadata:
name: example-bundle
version: 0.0.1

packages:
- name: helm-overrides-package
path: "../../packages/helm"
ref: 0.0.1
overrides:
podinfo-component:
unicorn-podinfo:
variables:
- name: UI_COLOR
path: "ui.color"
description: "variable UI_COLOR accepts a raw value (e.g. a string, int, map) like "purple", which is passed to the ui.color helm path"
type: raw
- name: test_secret
path: "testSecret"
description: "variable TEST_SECRET will resolve to the contents of a file (e.g. test.cert), which gets passed to the testSecret helm path"
type: file
```
UncleGedd marked this conversation as resolved.
Show resolved Hide resolved

**File Paths**

If a file path is not absolute, it will be set as relative to the `uds-config.yaml` directory.

e.g. the following `uds-config.yaml` is in [`src/test/bundles/07-helm-overrides/variable-files/`](../src/test/bundles/07-helm-overrides/variable-files/uds-config.yaml)
```yaml
variables:
helm-overrides:
test_secret: test.cert
```

This means when `test.cert` is evalutated it will first be appended to the config path like so `src/test/bundles/07-helm-overrides/variable-files/test.cert`.

If the file path is already set to the same relative path as the config, then no merging will take place.

> [!NOTE]
> UDS CLI does not encrypt or base64 encode any file contents before passing said data to Zarf or Helm.
> For example, if the file contains a key to be used in a Kubernetes secret, it must be base64 encoded before being ingested by UDS CLI.


### Namespace
It's also possible to specify a namespace for a packaged Helm chart to be installed in. For example, to deploy the a chart in the `custom-podinfo` namespace, you can specify the `namespace` in the `overrides` block:

Expand Down
5 changes: 5 additions & 0 deletions src/cmd/uds.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ var deployCmd = &cobra.Command{
bundleCfg.DeployOpts.Source = chooseBundle(args)
configureZarf()

// set DeployOptions.Config if exists
if config := v.ConfigFileUsed(); config != "" {
bundleCfg.DeployOpts.Config = config
}

// create new bundle client and deploy
bndlClient := bundle.NewOrDie(&bundleCfg)
defer bndlClient.ClearPaths()
Expand Down
87 changes: 80 additions & 7 deletions src/pkg/bundle/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,11 @@ func (b *Bundle) Deploy() error {
if len(userSpecifiedPackages) != len(packagesToDeploy) {
return fmt.Errorf("invalid zarf packages specified by --packages")
}
return deployPackages(packagesToDeploy, resume, b)
} else {
packagesToDeploy = b.bundle.Packages
}

return deployPackages(b.bundle.Packages, resume, b)
return deployPackages(packagesToDeploy, resume, b)
}

func deployPackages(packages []types.Package, resume bool, b *Bundle) error {
Expand Down Expand Up @@ -340,7 +341,7 @@ func (b *Bundle) processOverrideNamespaces(overrideMap sources.NamespaceOverride
func (b *Bundle) processOverrideValues(overrideMap *map[string]map[string]*values.Options, values *[]types.BundleChartValue, componentName string, chartName string, pkgVars map[string]string) error {
for _, v := range *values {
// Add the override to the map, or return an error if the path is invalid
if err := addOverrideValue(*overrideMap, componentName, chartName, v.Path, v.Value, pkgVars); err != nil {
if err := addOverride(*overrideMap, componentName, chartName, v, v.Value, pkgVars); err != nil {
return err
}
}
Expand All @@ -361,41 +362,47 @@ func (b *Bundle) processOverrideVariables(overrideMap *map[string]map[string]*va
setVal := strings.Split(k, ".")
if setVal[0] == pkgName && strings.ToUpper(setVal[1]) == v.Name {
overrideVal = val
v.Source = b.getSourcePath(types.CLI)
}
} else if strings.ToUpper(k) == v.Name {
overrideVal = val
v.Source = b.getSourcePath(types.CLI)
}
}

// check for override in env vars if not in --set
if envVarOverride, exists := os.LookupEnv(strings.ToUpper(config.EnvVarPrefix + v.Name)); overrideVal == nil && exists {
overrideVal = envVarOverride
v.Source = b.getSourcePath(types.Env)
}

// if not in --set or an env var, use the following precedence: configFile, sharedConfig, default
if overrideVal == nil {
if configFileOverride, existsInConfig := b.cfg.DeployOpts.Variables[pkgName][v.Name]; existsInConfig {
overrideVal = configFileOverride
v.Source = b.getSourcePath(types.Config)
} else if sharedConfigOverride, existsInSharedConfig := b.cfg.DeployOpts.SharedVariables[v.Name]; existsInSharedConfig {
overrideVal = sharedConfigOverride
v.Source = b.getSourcePath(types.Config)
} else if v.Default != nil {
overrideVal = v.Default
v.Source = b.getSourcePath(types.Bundle)
} else {
continue
}
}

// Add the override to the map, or return an error if the path is invalid
if err := addOverrideValue(*overrideMap, componentName, chartName, v.Path, overrideVal, nil); err != nil {
if err := addOverride(*overrideMap, componentName, chartName, v, overrideVal, nil); err != nil {
return err
}

}
return nil
}

// addOverrideValue adds a value to a PkgOverrideMap
func addOverrideValue(overrides map[string]map[string]*values.Options, component string, chart string, valuePath string, value interface{}, pkgVars map[string]string) error {
// addOverride adds a value or variable to a PkgOverrideMap
func addOverride[T types.ChartOverride](overrides map[string]map[string]*values.Options, component string, chart string, override T, value interface{}, pkgVars map[string]string) error {
// Create the component map if it doesn't exist
if _, ok := overrides[component]; !ok {
overrides[component] = make(map[string]*values.Options)
Expand All @@ -406,6 +413,23 @@ func addOverrideValue(overrides map[string]map[string]*values.Options, component
overrides[component][chart] = &values.Options{}
}

var valuePath string

switch v := any(override).(type) {
case types.BundleChartValue:
valuePath = v.Path
case types.BundleChartVariable:
valuePath = v.Path
if v.Type == types.File {
if fileVals, err := addFileValue(overrides[component][chart].FileValues, value.(string), v); err == nil {
overrides[component][chart].FileValues = fileVals
} else {
return err
}
return nil
}
}

// Add the value to the chart map
switch v := value.(type) {
case []interface{}:
Expand Down Expand Up @@ -443,13 +467,31 @@ func addOverrideValue(overrides map[string]map[string]*values.Options, component
templatedVariable := fmt.Sprintf("%v", v)
value = setTemplatedVariables(templatedVariable, pkgVars)
}
// handle default case of simple values like strings and numbers

// Handle default case of simple values like strings and numbers
helmVal := fmt.Sprintf("%s=%v", valuePath, value)
overrides[component][chart].Values = append(overrides[component][chart].Values, helmVal)
}
return nil
}

// getSourcePath returns the path from where a value is set
func (b *Bundle) getSourcePath(pathType types.ValueSources) string {
var sourcePath string
switch pathType {
case types.CLI:
sourcePath, _ = os.Getwd()
case types.Env:
sourcePath, _ = os.Getwd()
case types.Bundle:
sourcePath = filepath.Dir(b.cfg.DeployOpts.Source)
case types.Config:
sourcePath = filepath.Dir(b.cfg.DeployOpts.Config)
}

return sourcePath
}

// setTemplatedVariables sets the value for the templated variables
func setTemplatedVariables(templatedVariables string, pkgVars map[string]string) string {
// Use ReplaceAllStringFunc to handle all occurrences of templated variables
Expand All @@ -464,3 +506,34 @@ func setTemplatedVariables(templatedVariables string, pkgVars map[string]string)
})
return replacedValue
}

// addFileValue adds a key=filepath string to helm FileValues
func addFileValue(helmFileVals []string, filePath string, override types.BundleChartVariable) ([]string, error) {
verifiedPath, err := formFilePath(override.Source, filePath)
if err != nil {
return nil, err
}
helmVal := fmt.Sprintf("%s=%v", override.Path, verifiedPath)
return append(helmFileVals, helmVal), nil
}

// formFilePath merges relative paths together to form full path and checks if the file exists
func formFilePath(anchorPath string, filePath string) (string, error) {
if !filepath.IsAbs(filePath) {
// set path relative to anchorPath (i.e. cwd or config), unless they are the same
if anchorPath != filepath.Dir(filePath) {
filePath = filepath.Join(anchorPath, filePath)
}
}

if helpers.InvalidPath(filePath) {
return "", fmt.Errorf("unable to find file %s", filePath)
}

_, err := helpers.IsTextFile(filePath)
if err != nil {
return "", err
}

return filePath, nil
}
Loading
Loading