Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

capacity: fix handling of topology changes with immediate binding SCs #475

Merged
merged 1 commit into from
Sep 2, 2020

Conversation

pohly
Copy link
Contributor

@pohly pohly commented Aug 31, 2020

What type of PR is this?
/kind bug

What this PR does / why we need it:

A copy-and-pasted "return" instead of "continue" caused processing to
stop after encountering a storage class with immediate binding, which
had the effect that further CSIStorageCapacity objects for other
storage classes never got created.

This only happened when topology changed at runtime. When topology
was already known when the controller started, all expected objects
got created.

Does this PR introduce a user-facing change?:

CSIStorageCapacity objects were potentially incomplete when at least one storage class used "immediate binding" and topology changed while the controller was already running.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 31, 2020
@k8s-ci-robot k8s-ci-robot requested review from lpabon and msau42 August 31, 2020 08:46
@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Aug 31, 2020
@pohly
Copy link
Contributor Author

pohly commented Aug 31, 2020

/assign @msau42

I noticed this bug when trying out the steps in the upcoming blog post kubernetes/website#23102

I'll work around the bug in the example, therefore we don't need to hurry with getting a bug fix release out.

pohly added a commit to pohly/pmem-CSI that referenced this pull request Aug 31, 2020
This works around
kubernetes-csi/external-provisioner#475 by not
creating any storage classes with immediate binding.
@@ -299,7 +299,7 @@ func (c *Controller) onTopologyChanges(added []*topology.Segment, removed []*top
continue
}
if !c.immediateBinding && sc.VolumeBindingMode != nil && *sc.VolumeBindingMode == storagev1.VolumeBindingImmediate {
return
continue
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a unit test for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can, but it's not perfect: we need a specific ordering of storage classes (first the one with immediate binding, then the one with late binding), and there's no guarantee that the informer will list them in any particular order. But in practice, the informer seems to use "first in, first out", so I was able to trigger the problem.

But worse, onTopologyChanges wasn't tested at all. Fixed...

A copy-and-pasted "return" instead of "continue" caused processing to
stop after encountering a storage class with immediate binding, which
had the effect that further CSIStorageCapacity objects for other
storage classes never got created.

This only happened when topology changed at runtime. When topology
was already known when the controller started, all expected objects
got created.
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 2, 2020
@msau42
Copy link
Collaborator

msau42 commented Sep 2, 2020

/lgtm
/approve

Can you cherry-pick this to release-2.0 branch?

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 2, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 2, 2020
@k8s-ci-robot k8s-ci-robot merged commit 28975c5 into kubernetes-csi:master Sep 2, 2020
@pohly
Copy link
Contributor Author

pohly commented Sep 3, 2020

Can you cherry-pick this to release-2.0 branch?

See PR #476

kbsonlong pushed a commit to kbsonlong/external-provisioner that referenced this pull request Dec 29, 2023
Update dependency go modules for k8s v1.28.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants