-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merging Butane configs #118
Comments
* Accept Fedora CoreOS Config (FCC) version v1.1.0 inputs and render Ignition v3.1.0 format * Continue to accept FCC v1.0.0 inputs and render Ignition v3.0.0 format * Support merging FCC snippets with either v1.1.0 or v1.0.0 versions. However, all snippets mut use a version that matches the version in the FCC content. Version skew among snippets is not supported Related: * coreos/butane#118
* Accept Fedora CoreOS Config (FCC) version v1.1.0 inputs and render Ignition v3.1.0 format * Continue to accept FCC v1.0.0 inputs and render Ignition v3.0.0 format * Support merging FCC snippets with either v1.1.0 or v1.0.0 versions. However, all snippets mut use a version that matches the version in the FCC content. Version skew among snippets is not supported Related: * coreos/butane#118
I needed to extend terraform-provider-ct with a similar ability to add FCC With no fragments, fcct's Translate is used. With fragments, the main FCC content is parsed to pick an FCC/Ignition version, then each fragment is Translated and parsed into Ignition to be able to merge Ignition Config struct's that are all of the same version. For now enforcing the FCC and any fragments are on matching versions. I'm interested in what this might look like in fcct. Which seems to have much nicer internal primitives. My strategy of falling back through the different Ignition versions, calling Parse isn't great. |
I can see two main approaches. In any case. Making it simpler to divide the FCC config into several files would be appreciated! |
Personally prefer the first approach as it makes the dependency between fcc files trackable. |
To add to this, it would be very good if the mechanism to specify |
@Okeanos Hmm, I don't see files and merged configs as neatly divided into two distinct namespaces. For example, I think there's a reasonable argument that each merged config might want its own files-dir, at which point we'd have N separate namespaces. The general expectation is that if you have snippets that are commonly reused, you'd render each of them into a separate Ignition config (with its own files-dir), host it at a well-defined URL, and merge it via HTTPS at provisioning time. That allows the snippets to be independently updated without rerendering the parent config. |
That thought actually occurred to me later as well; forgot to update the comment, though. |
Honestly, I don't have a problem with any of these approaches, however, it would be great to have this. In general, I would like to use the Butane configs a bit as roles and playbooks in Ansible and I'm doing that but having Makefile for this is weird. I really think that this functionality should be already there. Personally, I would go the simplest path. When the Butane found |
It'd seem really natural to me to just support: |
@cgwalters We've avoided supporting that for the same reason we try to avoid command-line options affecting the semantics of the output. The instructions for assembling the final Ignition config would now reside ephemerally in your bash history, rather than persistently in a config file. |
The idea is more one could easily write a |
Yeah, understood. Even in that case, though, the final config would now be specified in a mix of two languages/locations. |
Also, I expect a lot of simple cases are e.g. |
That wildcard is problematic for another reason: the semantics of the resulting config are dependent on the merge order. The usual solution is to add sequence numbers to filenames, but users would need to know to do that. Even if we were to support ad-hoc merging from the command line, we'd also need to support principled merging of Butane configs via Butane syntax. And it'd be a good idea to add that support first, to avoid encouraging poor config hygiene. |
@bgilbert Good point about the order! I tried to find documentation about the exact functionality of the merge key but cannot seem to find it. |
In my case, I have multiple machines in wildly different infrastructure (e.g. one in a public cloud, one on my home network) and I want to factor out common Ignition bits like my SSH key. They may not be able to reach a common URL, and even if they could doing this introduces a whole new level of complexity (e.g. to correctly do this you want to use
I'm a bit confused; are you arguing against the concept of
Do you have an example case in mind where someone might be depending on the merge order in a problematic way? |
I'm doing that even now but it's not something I would like to do really. |
@tkarls Butane doesn't currently have any docs about config merging because Butane doesn't do the merging itself. See the Ignition operator notes for more info.
Yup, to be clear, client-side merging makes a lot of sense for smaller environments. Merging independent config sources at runtime is the more general case, and might make more sense in an enterprise setting where multiple teams independently maintain configs. I was arguing specifically against supporting multiple ( As an aside, in your use case you may not need
Sure, but merge semantics are more subtle than object linking. An analogy might be a C program with a lot of
A hardware-specific or workload-specific config might want to override pretty much anything in a site-wide config: the contents of a config file, whether to enable a systemd unit, the size of a root or data partition. (Also, if |
Exactly because of this I think that the Butane should benefit from the existing Ignition merge feature. What Butane should do is to look on the config and do the translation of the pointed '.bu' files in the |
Is there anything up-to-date on how to do this? I see mentions of makefiles but the current docs are in such a state around this topic that I cannot figure out how to do merging at all. |
@alvarlagerlof Right now, the inputs to config merging are Ignition configs, not Butane configs. You can use Butane to generate both configs, but the child config referenced by the parent |
Ah, Thank you for the pointer. |
After that just use the standard Makefile which will check that the file does exists and solve the issues about what changed for you. |
Why this is usefulI fully agree with the proposal to combine multiple Butane files into one Ignition file at build time, for two reasons:
Both of these use cases might be quite common in the real world, so I assume that a feature that allows to import/include other Butane files will be of high value to the community, especially to those who build Butane configs that are not enterprise-level but still quite complex. Discussion summaryTo get this going, I'd first like to summarize the current state of the discussion:
ConceptAn idea how to implement both approaches (independently from each other as I wrote above):
|
My original aim was just for This would allow Butane tools to implement merging the same way. Phrased another way, we need to agree on function that can merge a list of butane snippets. Discussions about specific flag-based tools or how to expose the feature follow from that. |
To be honest I can't follow. Isn't this the repo for the
Do you mean purely from the technical perspective? I.e. Butane gets two or more YAML structures to merge (however it may have gotten them) and needs to output a single YAML structure? I'd say the algorithm for this should be exactly the same one as in Ignition, provided we do use the same syntax in the end. This leaves us with importing/including, which is a related feature but doesn't need the algorithm for config merging. |
Given multiple Ignition Once that is in place, |
For those who want to use including and merging right away, I have created a ###
# Copyright: 2021 Lukas Bestle
# License: https://opensource.org/licenses/MIT
###
files := $(shell find files -type file)
yaml := $(shell find . -name '*.yml')
# Final build step
dist/ignition.json: $(yaml) $(files) dist/butane.bu
butane -d files dist/butane.bu -o dist/ignition.json
# Combines all YAML files into the merged Butane YAML
# Each merging pass resolves the `!include` and `!merge` tags:
# `!include` replaces the tag with the referenced file contents
# `!merge` merges the parent with the referenced file contents
# Multiple passes are used to resolve recursive includes/merges
dist/butane.bu: $(yaml) dist
cp main.yml dist/.butane.bu
for number in 1 2 3; do \
echo "Merging pass $$number"; \
yq eval '(.. | select(tag == "!include")) |= load(.)' -i dist/.butane.bu; \
yq eval 'with(.. | select(tag == "!merge"); parent = (parent *+ load(.)) | del(.))' -i dist/.butane.bu; \
done
mv dist/.butane.bu dist/butane.bu
# Creates the dist folder if it doesn't exist
dist:
mkdir -p dist
# Deletes all dist files
.PHONY: clean
clean:
rm -r dist
# Spins up a temporary HTTP server to serve the ignition config
.PHONY: serve
serve: dist/ignition.json
cd dist; python3 -m http.server UsageThe
Here's an example for the YAML syntax you would use: variant: fcos
version: 1.4.0
passwd:
groups:
!include your-custom-structure/groups.yml
users:
- !merge your-custom-structure/users.yml
- name: core
groups:
- wheel Output: variant: fcos
version: 1.4.0
passwd:
groups:
- name: test
users:
- name: core
groups:
- wheel
- name: user1
... |
We really need the merging feature... 😅 Thanks @lukasbestle for sharing your work! |
Here's another example of how to (naively) merge Butane config snippets: https://github.com/LorbusChris/butane-config-template |
So useful! Thanks! |
@dghubble Hmm, I'm not sure I understand the API issue. On the Ignition side, there are currently a couple ways to merge two byte slices containing Ignition configs:
Both approaches support fragments with mixed spec versions. We've used both in different places, depending on the situation; e.g. the first one can be implemented in non-Go code. The first one is probably too obscure to add as a helper function in Ignition, and the second one doesn't seem worth a helper because it's basically two function calls. Maybe I'm missing something though? On the Butane side, I think programmatic merging will happen for free as part of the user-facing implementation. The caller will be able to create a Butane config struct with the appropriate merge directives (which may mean we should support both In any event, let's keep this issue focused on user-facing config merging. If you'd like to continue discussion of the API side, feel free to open a separate issue. |
@lukasbestle Thanks for the writeup and proposal! Import/include and merge via YAML tagsI have substantial reservations about this approach.
Native Ignition mergeI think it makes more sense to extend the existing merge semantics to Butane configs. But adding a variant: fcos
version: 9.9.9-experimental
ignition:
config:
merge:
- local_butane: child.bu
- inline_butane: |
variant: fcos
version: 1.1.0
[...] I'd think users would mostly use As you point out, I agree that in all cases, child configs should use the same Another option: variant: fcos
version: 9.9.9-experimental
ignition:
config:
merge:
- butane: true
local: child.bu
- butane: true
inline: |
variant: fcos
version: 1.1.0
[...] Early or late bindingTo be completely consistent with Ignition config merging, we'd need to recursively render child configs to Ignition and include them as data URLs ("late binding"). That allows Ignition itself to handle the actual merging at runtime, and allows child configs to use a newer config spec than the parent. It's also space-inefficient and awkward, and the output is difficult to manually inspect. (I ended up building this for debugging If we want Butane itself to handle the merging ("early binding"), Butane would recursively evaluate the children, evaluate the parent without the merge directives, and then merge the two together. Child configs couldn't have a newer spec version than the parent, since the parent would presumably define the output Ignition config version, and the newer spec might contain fields that are unrepresentable in the older one. We'd also need to think about validation semantics: if a child Arguably, early binding is a separable issue, since we might conceivably want to support it even for the current Ignition-centric semantics of |
Thanks for sharing your thoughts!
Fair enough. :)
I'm combining these three points in my reply as they are related. I feel like merging full Butane configs and importing snippets are features for two very distinct use cases. One major use case for importing snippets is a "configuration template" that can be shared as a Git repo. So you would have a Keep in mind that snippet importing is much less complex regarding the behavior and handling. E.g. the Butane version issue you describe for config merging wouldn't be an issue here as the preprocessor can rightfully assume that the imported snippet uses the same Butane version. I'm generally a huge fan of solutions with low complexity and a huge impact. This doesn't mean that importing can replace config merging (it really can't), I just think it would be a simple and powerful solution for this use case.
OK, that makes sense. Maybe it's a question of expectations for the Butane tool. When I read that there is a tool to generate Ignition configs, I had really thought that it included all sorts of convenience functions to make the life working with Ignition configs easier. But if I understand you correctly, Butane is really just meant as a simple low-level tool to convert YAML to JSON and validate the document structure. Of course it does more internally, but that's how it feels from the user perspective. To be honest I feel like that's a bit of a lost opportunity: There is already
Why is that? I'd say that the output should use the newest version of all used Butane configs. This is the only way to represent all config data. And if a newer version ever removes a feature from a previous version, there's likely a conflict anyway and Butane should fail. Or can there be a case where a user would deliberately want to choose an older version for their parent config even though they want to merge in configs with a newer version?
Yes, please. If Butane doesn't fail, Ignition will and that won't help anyone. I'd even say that Butane should always fail if it encounters configs from different Butane variants (reduces complexity and avoids bugs in weird edge cases). This restriction excludes use cases where one would want to use a global base config for multiple Butane variants. But I really wonder if this use case is even viable without edge cases.
I agree. It all gets incredibly complex. I'm worried it will be very hard to understand and especially to debug in the end. |
This comment has been minimized.
This comment has been minimized.
Let's keep this issue focused on config merging UX, please. @gui-don, I've moved your comment to a separate issue #301. |
@lukasbestle Thanks for the response!
I agree with the "configuration template" use case, but disagree that merging full configs doesn't make sense here. To make this more concrete, I've included some examples at the bottom of this comment: a unified config that creates a file and two users, the same config broken out using inclusion as I understand the proposal, and the same config broken out using Butane config merging. In my view, the merged configs are substantially cleaner than the included ones. The intentions of the child configs are clear at a glance; the child configs declare their variant/version and thus have well-defined semantics; and the extra boilerplate is fairly trivial. Also (not shown in the example) it's possible for a single child config to declare a related user, systemd service, and file, without switching to a different config format (from inclusion to a Butane child config). And in the parent config, the child configs are all listed in one place, rather than scattered through the file.
That's okay for the example you gave. However, if the feature existed, it would surely be used in contexts where the parent and child configs are managed independently. In that case, bumping the version of the parent config could change the semantics of the child — possibly even invalidating it. (Semantic changes are allowed in major spec version bumps.) There'd be no metadata in the child indicating which spec it was written for, so a human would need to manually fix up the child based on knowledge of the relevant spec versions. I don't think saving a few lines is worth the ambiguity. Butane's predecessor
Butane is indeed intended to be a higher-level tool, with convenience functions etc. But it's also aligned with the philosophy of the rest of the provisioning stack (Ignition/Afterburn) and to some degree the rest of FCOS, which favors small, correct, opinionated tools rather than adding every possible feature. IMO that's consistent with adding useful config merging, but in a form that favors explicitness and correctness.
There are a couple things tied up here. On the one hand, it's convenient to allow parent and child to be versioned independently. On the other hand, we currently have a well-defined mapping from Butane spec version to Ignition spec version. That's mainly important because configs need to be able to set an upper bound on the output Ignition version, since they may be targeting OSes with older versions of Ignition. So actually, I think it would be okay to loosen that requirement, as long as we don't spontaneously generate a config version newer than any of the merge inputs. We could emit the max version of all merged configs, or we could have the parent config version limit the versions of its children. The latter is more explicit and also somewhat harder to use. Example unified configvariant: fcos
version: 1.4.0
storage:
files:
- path: /a
contents:
inline: hello world
passwd:
users:
- name: user1
- name: user2 Example inclusionParent configvariant: fcos
version: 1.4.0
storage:
files:
- !include 'file.yml'
passwd:
users: !include 'users.yml' file.ymlpath: /a
contents:
inline: hello world users.yml- name: user1
- name: user2 Example config inclusionParent configvariant: fcos
version: 1.4.0
ignition:
config:
merge:
- local_butane: file.bu
- local_butane: users.bu file.buvariant: fcos
version: 1.4.0
storage:
files:
- path: /a
contents:
inline: hello world users.buvariant: fcos
version: 1.4.0
passwd:
users:
- name: user1
- name: user2 |
@bgilbert Thank you, that is entirely convincing. I agree that config merging is the best way forward also for this use case.
Both ways make sense, but considering:
I'd say that the safest way would be to always use the Butane version as defined in the parent config. Child configs with a lower version would be upgraded. If a child config uses a higher version than the parent config, ideally the following would happen:
In case such a compatibility check between different Butane versions and the downgrade are not easily possible, the alternative would be to always throw a fatal error if the child config version is higher. With a good and specific error message this would still be easy to fix by the user and the compatibility check with downgrade could still be added as an enhancement later. |
Yeah, I generally agree. If we start with the most restrictive model (fail if the child version is newer than the parent), we can always loosen it later. Note that there's no mechanism for version downgrades (other than ign-converter, which is unsupported), and no mechanism for upgrading/downgrading Butane configs at all. Any version translation would be applied to Ignition configs after they've been transpiled. The translation code is maintained by hand, and I don't think we should add translations that are only used in certain corner cases; rules will inevitably be missed. |
Thanks @bgilbert, I missed this addition. It handles the case I had for merges of varying configs via the API. |
#301 (comment) discusses a use case where child configs would like to have a different |
We could support reading a child config in either of two modes:
We could distinguish these implicitly by checking whether |
This feature would be extremely beneficial for those of us working with infrastructure as code, where modularity, reusability, and templating are key practices. Implementing this would greatly enhance Butane's utility by allowing us to construct more modular and maintainable configurations that can be easily templated and reused across various projects and environments. |
So I had some free to play with open source tools and got stuck on this issue. I made a package that works for me for now but it surfaced a number of issues that will probably be useful to think about long term? https://github.com/nveeser/butanex Summary (as I understand it)Users value some form of config parameterization and/or composition to allow building an Ignition file from multiple composable "providers" (ie Terraform modules, etc). Two different paths are "import" and "template". This bug is about "import". As with all abstractions there is a tension. The more sophisticated you make the set of tools, the easier it is to build something really complex which is very hard to reason about ("why does only host Y get storage.files.foo to X, it should be Y"). Early / Late bindingThis could be implemented in Ignition (aka late binding) or in Butane before transformation (aka early binding). The latter makes Ignition configs easier to read and has better ergonomics and possibly has other benefits VersioningButane files are versioned which allows the implementation to change semantics while allowing users to migrate intentionally. With multiple files together there is an important challenge of merging the same object from two files where the objects have different semantics. NotesWorking through this I found a few corner cases but I suspect there are more. Especially since I have no experience with tools like Terraform and how that could be used to partition this problem. Here are some notes. Mapping Nodes - Maps vs StructsThis operates on generic Perhaps the translater interface could be an interface type rather than a function type? Local PathsI made the choice to update relative paths (ie Resource.Local) where possible. When merging a file, the local paths in that file are updated to be relative to the new file based on the existing one. This may make assumptions on the underlying filesystem layout. Merge VersionsGiven Configurable semanticsThe challenging with merging config trees like this is subtle details like sequences. For example when merging two Sequences, does the source sequence or replace destination sequence. Sometimes you want one file to contain defaults and the other file to overwrite the defaults. I suspect the semantics can be per file or per tag? For tag specific behaviors one could store them on struct tag of the schema struct. For file-level behaviors this could be a command line argument or a node with tags in the Butane YAML itself? Likely some user will come asking to fine-tune that for the specific use case. Again the challenge is addressing needs without handing out foot-guns. I added a config with patterns that can be applied to parts of the tree. If the pattern matches the context path, apply that behavior (overwrite, replace filepaths, etc) YAML Sequence ambiguityIf I understand the spec YAML is permissive on Sequence types. It's possible to write a YAML key that only contains a single element; if the target is a sequence type (aka slice), then YAML (at construction?) will add single element to the list. Looking at the file there is no way to determine if that field is a list field or not. Examples
Merging two files which use both for the same key is challenging to get right? (I guess sequence should always win?). I did not solve this issue here. My guess is that |
It's convenient for users to write multiple Butane fragments and then merge them together into one config. This is awkward right now: users must separately transpile each fragment and then use Ignition's
merge
directive to merge them at runtime. Or, they can transpile each Butane fragment and use a wrapper script to build a top-level Butane config that inlines the Ignition fragments usingmerge
directives.Provide a mechanism to include Butane fragments into another Butane config and produce a unified Ignition config as output. This might be done by extending
merge
to allow referencing local Butane configs, transpiling each piece, and then performing the Ignition config merge at transpile time.The text was updated successfully, but these errors were encountered: