-
Notifications
You must be signed in to change notification settings - Fork 706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not possible to represent absence of Taints when marshalling kubeadm configuration #1358
Comments
that is true. applying a taint if the list is empty by default is not ideal. but also our comment in the API is incorrect as the taint is not applied to JoinConfiguration, yet JoinConfiguration also embeds NodeRegistrationOptions. is this happening in the JoinConfiguration (bug?) or InitConfiguration? in terms of actions that the logic performs - for
but this also skips the label: |
@neolit123 This is happening in the To be clear, I am using a Go program to generate the |
understood, so even if you set the list of taints to but, when executing
()note that this will skip the label too) wouldn't that solve the problem? |
Actually, kubeadm will not taint the master if the marshalled configuration sets taints to
It's possible to create such a configuration by hand, just not using a Go program that imports kubeadm types. If |
while this library is part of the pipeline currently, consumers of the kubeadm config should be using the serializers from api-machinery or the new
i need to verify this. the marshaling problem aside. how about this as a workaround?
|
Ugh, I'm sorry, I mixed up nil and empty slice. kubeadm will apply default taints if the taints slice is nil, and will not apply taints if the taints field is an empty slice. If I set taints to an empty slice in a Go program, it the field will be omitted, because it's tagged "omitempty." I'll make corrections to the comments. |
I updated the example program to illustrate what exactly what happens. I create the struct, set Taints to an empty slice, marshal it, then unmarshal to a fresh config, and now Taints is nil, not an empty slice. |
thank you for the update.
since our config is in beta, we won't be able to modify this field for a while. |
to re-iterate, while this seems like a problem in the config, it appears to be only a problem when marshaling and not if the users are writing a config by hand. |
@neolit123 That's correct, it's a problem only when marshalling. |
My Cluster API provider marshals kubeadm configuration. I think doing this is quite natural when kubeadm is a part of some larger automation that is written in Go. Now that kubeadm is GA, I expect more tools to marshal kubeadm configuration. |
Looks like this issue has come up before. Here's what the API conventions say:
|
thanks for the link ^. for now a workaround is to patch the marshaled yaml string, which isn't a great solution. |
Specifically for users who want |
/assign @timothysc |
I don't think, that this can be fixed in the context of
As of now (with Therefore, to continue to taint the master, one must add the taint manually in the config like so:
This is a breaking change and unacceptable in On the other hand it is worthy of consideration and probably the best approach for a future config version. |
/kind api-change |
I don't understand what you are trying to say. Can you please clarify? Removing the
Output:
|
As recommended by the API conventions, the issue can be resolved by removing the "omitempty" removed from the Taints field. As far as I understand, removing the "omitempty" tag changes only how the field is marshalled. It does not change how the field is unmarshalled. It does not change the default value kubeadm sets. Finally, it does not change the field's type. Given the above, is there any reason why removing the "omitempty" tag would break v1beta1 API compatibility? Thanks in advance for feedback. |
@dlipovetsky I am really sorry for the late response. Say kubeadm or third party tool marshales to v1beta1 config with 1.13.3 vendored in. If it doesn't care about taints (which is the usual case) it will simply not initialize that field and when the config is marshalled to YAML it won't have Now, consider, that after some time the same piece of software (be it kubeadm or third party) vendors in newer k8s version, where v1beta1 does not have omitempty on the Taints key. In that case the same piece of software will generate a line of So the same piece of marshaller software, using the same config version, but with a different k8s version causes different output and different behavior when kubeadm uses that config. |
Question for you: if this piece of software marshals |
The answer is no and I know, that this will be the normal case. Actually I did not express myself clearly in the quoted lines above. What I meant is that people may depend on that omitempty misplacement bug and not know or care about it. Consider this:
Now let's assume, that we didn't add any taints. Yes, I admit, this is a corner case. It's not written to the full spec. But I bet, that there is someone who has written their code in that manner and they don't expect the config behavior to change for them, unless they flip the config version. That will be a nasty surprise for them and they'll notice it when random pods start to get scheduled on their control plane nodes. We just have to be careful with public APIs. Fixing a bug somewhere for someone does not mean, that this won't break someone else's code. That's why we have to do the fixes in new versions. |
@rosti Thanks for giving more detail. I think we agree that, if omitempty is removed, then by default Taints will be marshalled as
The code snippet you share assigns an empty slice to the Taints field, which in v1beta means no taints should be applied. If I understand correctly, you are saying that some tool both
I do not think this is a corner case; I think this tool has a bug. To fix the bug, the tool should leave the Taints field as nil. The omitempty tag in v1beta1 hides the tool's bug. And removing the omitempty tag in v1beta1 will expose the tool's bug. Is that what worries you? |
@dlipovetsky you are correct. It's a bug on our side and, possibly, relies on a bug on the other side for this to happen. But sometimes we need to be bug compatible and support our own bugs (especially if we are shipping a public API). @fabriziopandini @timothysc WDYT? |
we can make the decision to change or not change next cycle (change/break)... ⚖️ for now, is it worth documenting |
@rosti, I think we agree that a tool that uses the v1beta1 API correctly will not be affected by the bug fix. If such a tool were affected, I would agree that we should bump the version before fixing the bug. Should we hold off fixing an API bug, even if the fix only affects tools that use the API incorrectly? I'm not sure. I think this is a subtle and interesting question. |
@neolit123 Good idea! I'll file a PR on the website repo. |
please ping me on that. |
It's probably best to bring this topic on next week's kubeadm office hours and gather some opinions. |
i think it was discussed already during a cluster api meeting and possibly a kubeadm office hours. |
I haven't discussed this in kubeadm office hours; I'll try to attend next week. @rosti, I appreciate all of your feedback! Believe it or not, I recently voiced a similar concern on another PR:
|
Versions
kubeadm API v1alpha3 and v1beta1 (releases 1.11, 1.12, 1.13)
What happened?
I wanted to use a Go program to generate kubeadm configuration, such that no taints are applied to the node. I imported the kubeadm types, initialized an
InitConfiguration
struct, and setNodeRegistration.Taints
to an empty slice. I then marshalled the struct YAML and wrote this to a configuration file. I observed thatNodeRegistration.Taints
was not present in the file. When I passed the configuration file tokubeadm init
, I observed that kubeadm applied the default set of taints to the node.What you expected to happen?
The Taints field is explained in the code as follows:
I want to use the kubeadm types in a Go program and set the
Taints
field to an empty slice. However, when I marshal the struct, the field is omitted (because it is taggedomitempty
).How to reproduce it (as minimally and precisely as possible)?
Here is a Go program that generates
InitConfiguration
withNodeRegistration.Taints
set to an empty slice, then marshals the configuration to YAML, then unmarshals it into a freshInitConfiguration
struct.. Note that, in the fresh configuration struct,NodeRegistration.Taints
is nil, not an empty slice.Output:
The text was updated successfully, but these errors were encountered: