Overlay configs for custom generators do not override base configs

[snarlysodboxer]

I'm using Kustomize 3.2.0.

It seems to be impossible to use a base/overlay setup for custom generator configs. Generator configs in overlays do not override generator configs in base, at least not prior to running the custom plugin. E.G.

Base (for a staging environment):

kustomization.yaml:

generators:
- mySecretGetter.yaml

mySecretGetter.yaml:

apiVersion: crd.myteam.com/v1
kind: SecretGetter
metadata:
  name: my-name
secretsProjectID: 1234
...

Overlay (for a production environment):

kustomization.yaml:

resources:
- ../../base

generators:
- mySecretGetter.yaml

mySecretGetter.yaml:

apiVersion: crd.myteam.com/v1
kind: SecretGetter
metadata:
  name: my-name
secretsProjectID: 5678

What I did:

• Tried to kustomize build the overlay.

What happened:

• The plugin tries to get secret values using secretsProjectID: 1234.
• This happens during the resource inclusion of ../../base.
• My plugin fails here because the production credentials are not authorized to the staging secrets project 1234.

What I expected to happen:

• The plugin tries to get secret values using secretsProjectID: 5678.

Additional Notes

It's possible to move mySecretGetter.yaml into the resources section in the base, and patch it in the patches section in the overlay before using it as a generator config, however this produces objects of kind SecretGetter in the build output, as well as makes the base an incomplete configuration on it's own.

The work-around I'm currently using is to use separate overlays, one for staging and one for production, and a Kustomize structure like the following. This is not ideal because the base is again not a complete configuration on it's own, and because it requires nearly identical complete copies of mySecretGetter.yaml, which deviates from Kustomize's base/overlay strategy.

Staging Overlay (work-around):

resources:
- ../../base
- ../../secrets/staging

Production Overlay (work-around):

resources:
- ../../base
- ../../secrets/production

kustomization.yaml in secrets/staging and secrets/production:

generators:
- mySecretGetter.yaml

[Liujingfang1]

In the overlay, instead of adding mySecretGetter.yaml as generators, add it as a patch.

resources:
- ../../base

patchesStrategicMerge:
- mySecretGetter.yaml

[snarlysodboxer]

@Liujingfang1 Thank you. I tried that. The problem is it first runs the plugin with secretsProjectID: 1234, even though the patch later tries to change it to secretsProjectID: 5678. This fails because where my production overlay is being run, it's credentials are not authenticated to the staging project ID 1234, only the production project ID 5678. Then, since the base didn't produce the object from mySecretGetter.yaml, the patch also fails with failed to find unique target.

[snarlysodboxer]

Is there any plan to support not running plugins until after their configs have been patched by overlays?

[tehmoon]

I ended up in the same situation and try to play with the suggested answer with patchesStrategicMerge in the overlay.
I tried to add a flag enable: true in the base/mySecretGetter.yaml that will output the current templated configuration so it could be merged to your staging/mySecretGetter.yaml. The the goal would to have enable: true in the overlay so the plugin runs correctly (there is additional logic in the plugin).

That is patching the base/mySecretGetter.yaml correctly, but not executes the generator after :/ If there is a way to execute the generator generated by by the output of the patchesStrategicMerge that would solve my issue.

[snarlysodboxer]

@Liujingfang1 Additionally, unless you add mySecretGetter.yaml to both the resources and the generators in your base, (and are ok with your plugin config being output by kustomize build) you cannot have mySecretGetter.yaml in patchesStrategicMerge section in your overlay, because kustomize will fail with failed to find unique target for patch crd.myteam.com_v1_SecretGetter|example-login.

This is quite the problem as various workarounds are all also problematic, and it seems like a common scenario for certain types of custom plugins. We need a way to patch plugin configs before the plugin gets run against them.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Aenima4six2]

Running into the same issue. Any newer ways to resolve this ?

[tehmoon]

@Aenima4six2 I ended up writing a transformer that executes a JQ filter, which suits my needs better as JQ is way more flexible. But it is more verbose as well.
The good thing about the transformers is that they are scoped in the current kustomization.yaml, meaning that, if you have some hierarchical kustomizations, they will be executed from bottom to top.

[Aenima4six2]

@Aenima4six2 I ended up writing a transformer that executes a JQ filter, which suits my needs better as JQ is way more flexible. But it is more verbose as well.
The good thing about the transformers is that they are scoped in the current kustomization.yaml, meaning that, if you have some hierarchical kustomizations, they will be executed from bottom to top.

Yeah, I basically have my plugin YAML replicated across each kutomize override directory. Would be nice to perform a strategic merge patch override, so only the bits that change between environments need to be present, but we can live with this for now.

[hrobertson]

This is an old closed issue but it came up when I was searching for the solution to this exact same problem. I'm therefore posting this here in case it helps anyone...

Rather than duplicating the generator config, you can patch it if you use this sort of structure:

base/
  kustomization.yaml
  resource.yaml
generators-base/
  generator-config-foo.yaml
  generator-config-bar.yaml
production/
  generator-config/
    kustomization.yaml
  kustomization.yaml
staging/
  generator-config/
    kustomization.yaml
  kustomization.yaml
etc

production/generator-config/kustomization.yaml

resources:
- ../../generators-base

patches:
- target:
    kind: MyCustomGeneratorConfig
  patch: |-
    - op: replace
      path: /some/path
      value: something-specific-to-production

production/kustomization.yaml

resources:
- ../base

generators:
- ./generator-config

[paololazzari]

@hrobertson any chance you could update the example to show base/kustomization.yaml, base/resource.yaml and generators-base/generator-config-foo.yaml ? thank you