Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for dot-separated properties in values files #437

Closed
dav-garcia opened this issue May 7, 2024 · 4 comments
Closed

Support for dot-separated properties in values files #437

dav-garcia opened this issue May 7, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@dav-garcia
Copy link

Is your feature request related to a problem? Please describe.
My KafkaTopic resources use default config properties from a values file like this:

topics:
  defaults:
    partitions: 6
    replicas: 3
    configs:
      retention.ms: 2419200000 # 28 days

Applying those values in the template with the toyaml filter results in a YAML resource with the dot-separated properties split like this:

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaTopic"
metadata:
  ...
spec:
  partitions: 6
  replicas: 3
  configs:
    retention:
      ms: 2419200000

And then, the tool fails with this error:

|  #1 | TopicConfigKeysValidation | Config key 'retention' for topic    | {}      |
|     |                           | '...' is not valid                  |         |

For reference, here's the code in the template:

...
{% set defaults = values.topics.defaults %}
...
spec:
  partitions: {{ customizations.partitions | default(defaults.partitions) }}
  replicas: {{ customizations.replicas | default(defaults.replicas) }}
  configs:
    {% set configs = {} %}
    {% do configs.update(defaults.configs) %}
    {% do configs.update(customizations.configs) %}
    {{ configs | toyaml | indent(4) }}

Describe the solution you'd like
Honestly, I am quite surprised no one has faced this problem. Either I'm doing something wrong or my use case isn't that common (FWIW, we are moving from an imperfect but working Helm chart with Julie).

So, my request is to either:
a) Don't split YAML keys in values files.
b) Dot-join the nested config properties under configs in the KafkaTopic resource before actually validating and applying them.
c) Enable recursive macro support in JinJava (I've defined a recursive macro to "flatten" the nested maps but recursion is disabled):
JinjavaConfig.newBuilder().withEnableRecursiveMacroCalls(true)...

Describe alternatives you've considered
As I said above, maybe my template is wrong. If so, any advice is greatly appreciated.

Additional context
Thanks very much.

@fhussonnois
Copy link
Member

Hi @dav-garcia, you should now be able to test the early-access release which provide a fix for your needs (i.e., configs will be automatically flatten internally). In addition, I've added the capability to enable/disable recursive macro calls.

Out of curiosity, what motivated your migration from JulieOps to Jikkou? If you found any missing feature feel free to open github issues. Thank you.

@dav-garcia
Copy link
Author

Great! Thanks a lot @fhussonnois!

Regarding Julie, our main concerns are: a) It's no longer maintained. b) No template support (we relied on Helm for it).
Anyway, I really like the Jikkou approach of declaring configurations as k8s resources; it fits nicely.

@dav-garcia
Copy link
Author

I can confirm that flattening the nested maps works. 👌
I haven't tested the recursive macros feature.
Thank you.

@fhussonnois
Copy link
Member

Thank you very much for your feedback @dav-garcia. Don't forget to add a star to the Github project. It's really rewarding and useful for promoting the project ^^. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants