Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple additive templates #49

Merged
merged 3 commits into from
Aug 28, 2023
Merged

Allow multiple additive templates #49

merged 3 commits into from
Aug 28, 2023

Conversation

ryannedolan
Copy link
Collaborator

@ryannedolan ryannedolan commented Aug 26, 2023

Summary

Changed template resolution logic to support finding multiple templates with the same name.

Changed template rendering to support multiple output documents -- one for each found template.

Changed the build to avoid using fat application jars, enabling multiple adapters to implement templates with the same name.

Details

Occasionally it is useful for two adapters to provide a template with the same name. For example, a Flink adapter and a Spark adapter may both provide SqlJob.yaml.template. In such cases, having both adapters in the classpath would result in racy behavior, where nuances in the build would result in only one of the adapters actually being used.

Instead, we want adapters to be additive. Having both a Flink and Spark adapter in the classpath should result in both Flink and Spark jobs -- not one or the other. This requires using a normal application distribution -- not a fat jar -- since a fat jar cannot have multiple files with the same name.

This fixes #41 by allowing adapters to just not provide a template. In that case, Resources are rendered to an empty set of YAML files, and the operator does nothing.

Testing Done

There are currently no adapters that provide additive templates as described above, so to test this I added a dummy SqlJob.yaml.template to the Kafka adapter in my local build. This meant that both the Kafka adapter and Flink adapter ostensibly implemented SqlJobs. I was able to verify that the operator attempts to create K8s objects corresponding to the output of both templates.

@ryannedolan ryannedolan merged commit 6e1b6c0 into main Aug 28, 2023
1 check passed
@ryannedolan ryannedolan deleted the multiple-resources branch August 28, 2023 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support no-op Resources
2 participants