Always Start pvc-protection-controller and pv-protection-controller #61324

pospispa · 2018-03-18T17:33:10Z

What this PR does / why we need it:
After K8s 1.10 is upgraded to K8s 1.11 finalizer [kubernetes.io/pvc-protection] is added to PVCs
because StorageObjectInUseProtection feature will be GA in K8s 1.11.
However, when K8s 1.11 is downgraded to K8s 1.10 and the StorageObjectInUseProtection feature is disabled the finalizers remain in the PVCs and as pvc-protection-controller is not started in K8s 1.10 finalizers are not removed automatically from deleted PVCs and that's why deleted PVC are not removed from the system but remain in Terminating phase.
The same applies to pv-protection-controller and [kubernetes.io/pvc-protection] finalizer in PVs.

That's why pvc-protection-controller is always started because the pvc-protection-controller removes finalizers from PVCs automatically when a PVC is not in active use by a pod.
Also the pv-protection-controller is always started to remove finalizers from PVs automatically when a PV is not Bound to a PVC.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes N/A
This issue #60764 is for downgrade from K8s 1.10 to K8s 1.9.
This PR fixes the same problem but for downgrade from K8s 1.11 to K8s 1.10.

Special notes for your reviewer:

Release note:

NONE

pospispa · 2018-03-18T17:41:27Z

@liggitt @jsafrane @NickrenREN PTAL
@saad-ali I believe you have rights to re-assign the labels as needed. Please, do so.
/sig storage
/kind bug

pospispa · 2018-03-18T18:37:19Z

/test pull-kubernetes-e2e-gce

dims · 2018-03-19T00:48:33Z

/milestone v1.10

based on comment in #60764

liggitt · 2018-03-19T01:07:07Z

Until announced, 1.10 PRs should be opened against master. 1.10 branch is still fast-forwarding.

liggitt · 2018-03-19T01:08:15Z

I'm not sure this is necessary if the feature gate default is the same as it will be in 1.11

k8s-ci-robot · 2018-03-19T09:10:20Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

pospispa · 2018-03-19T09:16:05Z

Until announced, 1.10 PRs should be opened against master. 1.10 branch is still fast-forwarding.

Rebased to master.

I'm not sure this is necessary if the feature gate default is the same as it will be in 1.11

@liggitt OK, I agree that when the feature gate is the same this is not necessary. However, when the StorageObjectInUseProtection feature is enabled the finalizers are added to PVs/PVCs. In case admin disables the feature the finalizers will have to be removed manually from PVs/PVCs, but this PR introduces automatic removal of PV/PVC finalizers in case the StorageObjectInUseProtection feature is disabled.

pospispa · 2018-03-19T12:34:56Z

/retest

pospispa · 2018-04-20T07:47:35Z

@deads2k I believe I've addressed all your comments. Would you approve the PR? Should I squash the commits?

pospispa · 2018-04-20T08:03:18Z

We should also cherrypick this into 1.10

I don't know that I agree. Removing feature gate detection that we did all our testing with doesn't seem like a good idea.

@deads2k Let me, please, re-phrase which issue is this PR trying to fix:
StorageObjectInUseProtection feature is enabled by default in K8s 1.10+. Assume K8s cluster is used with this feature enabled, i.e. finalizers are added to all PVs and PVCs. In case the K8s cluster admin disables the StorageObjectInUseProtection feature and a user deletes a PVC that is not in active use by a pod then the PVC is not removed from the system because of the finalizer. Therefore, the user will have to remove the finalizer manually in order to have the PVC removed from the system. Note: deleted PVs won't be removed from the system also because of finalizers.

This PR is adding into K8s 1.11+ automatic removal of finalizers nevertheless the StorageObjectInUseProtection feature is enabled or disabled.

So do we need automatic removal of finalizers also in K8s 1.10?

Note: the automatic removal of finalizers was added into K8s 1.9.6 in PR #61370

deads2k · 2018-04-20T11:47:30Z

pkg/controller/volume/pvcprotection/pvc_protection_controller.go

 }

 // NewPVCProtectionController returns a new *{VCProtectionController.
 func NewPVCProtectionController(pvcInformer coreinformers.PersistentVolumeClaimInformer, podInformer coreinformers.PodInformer, cl clientset.Interface) *Controller {
 	e := &Controller{
 		client: cl,
 		queue:  workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "pvcprotection"),
+		storageObjectInUseProtectionEnabled: utilfeature.DefaultFeatureGate.Enabled(features.StorageObjectInUseProtection),


Don't take the feature gate dependency here, pass it in. Feature gates at the very edges, config everywhere else. Globals are evil.

@deads2k please, does it now look good from you point of view?

deads2k · 2018-04-20T16:49:11Z

cmd/kube-controller-manager/app/core.go

+		ctx.InformerFactory.Core().V1().PersistentVolumeClaims(),
+		ctx.InformerFactory.Core().V1().Pods(),
+		ctx.ClientBuilder.ClientOrDie("pvc-protection-controller"),
+		utilfeature.DefaultFeatureGate.Enabled(features.StorageObjectInUseProtection),


Yeah, this is ok for now. Eventually we'll want to push it back into the context/config, but this a fair point to start.

deads2k · 2018-04-20T16:50:01Z

/approve

deads2k · 2018-04-20T16:50:42Z

/approve

please squash before merge.

pospispa · 2018-04-20T17:07:44Z

@deads2k thank you for the approval. I've squashed the commits.

pospispa · 2018-04-20T17:34:57Z

/retest

After K8s 1.10 is upgraded to K8s 1.11 finalizer [kubernetes.io/pvc-protection] is added to PVCs because StorageObjectInUseProtection feature will be GA in K8s 1.11. However, when K8s 1.11 is downgraded to K8s 1.10 and the StorageObjectInUseProtection feature is disabled the finalizers remain in the PVCs and as pvc-protection-controller is not started in K8s 1.10 finalizers are not removed automatically from deleted PVCs and that's why deleted PVC are not removed from the system but remain in Terminating phase. The same applies to pv-protection-controller and [kubernetes.io/pvc-protection] finalizer in PVs. That's why pvc-protection-controller is always started because the pvc-protection-controller removes finalizers from PVCs automatically when a PVC is not in active use by a pod. Also the pv-protection-controller is always started to remove finalizers from PVs automatically when a PV is not Bound to a PVC. Related issue: kubernetes#60764

pospispa · 2018-04-20T18:52:42Z

/retest

deads2k · 2018-04-20T19:14:01Z

@deads2k thank you for the approval. I've squashed the commits.

I'll leave lgtm with original review @msau42

msau42 · 2018-04-20T20:10:31Z

/lgtm

Yes, I think we'll have to cherry pick this into 1.10, otherwise we'll have a gap in behavior between 1.9 and 1.11.

k8s-ci-robot · 2018-04-20T20:10:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, msau42, pospispa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-controller-manager/OWNERS~~ [deads2k]
~~pkg/controller/volume/OWNERS~~ [deads2k]
~~plugin/pkg/auth/OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2018-04-20T22:19:51Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-04-20T23:22:12Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-ci-robot · 2018-04-20T23:44:19Z

@pospispa: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-integration	`d3ddf7e`	link	`/test pull-kubernetes-integration`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-github-robot · 2018-04-21T00:23:17Z

Automatic merge from submit-queue (batch tested with PRs 61324, 62880, 62765). If you want to cherry-pick this change to another branch, please follow the instructions here.

StorageObjectInUseProtection feature is enabled by default in K8s 1.10+. Assume K8s cluster is used with this feature enabled, i.e. finalizers are added to all PVs and PVCs. In case the K8s cluster admin disables the StorageObjectInUseProtection feature and a user deletes a PVC that is not in active use by a pod then the PVC is not removed from the system because of the finalizer. Therefore, the user will have to remove the finalizer manually in order to have the PVC removed from the system. Note: deleted PVs won't be removed from the system also because of finalizers. That's why pvc-protection-controller is always started because the pvc-protection-controller removes finalizers from PVCs automatically when a PVC is not in active use by a pod. Also the pv-protection-controller is always started to remove finalizers from PVs automatically when a PV is not Bound to a PVC. Related issue: kubernetes#60764 Related PRs: kubernetes#61370 kubernetes#61324

…ction-feature-to-GA-2nd-attempt Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Bring StorageObjectInUseProtection feature to GA **What this PR does / why we need it**: It brings `StorageObjectInUseProtection` feature to GA. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes N/A **Special notes for your reviewer**: Related features: kubernetes/enhancements#498 and kubernetes/enhancements#499 Related PR: #61324 **Release note**: ```release-note StorageObjectInUseProtection feature is GA. ```

…ction-downgrade-issue-cherry-pick-into-K8s-1.10 Automatic merge from submit-queue. cherry-pick into K8s 1.10: Always Start pvc-protection-controller and pv-protection-controller **What this PR does / why we need it**: StorageObjectInUseProtection feature is enabled by default in K8s 1.10+. Assume K8s cluster is used with this feature enabled, i.e. finalizers are added to all PVs and PVCs. In case the K8s cluster admin disables the StorageObjectInUseProtection feature and a user deletes a PVC that is not in active use by a pod then the PVC is not removed from the system because of the finalizer. Therefore, the user will have to remove the finalizer manually in order to have the PVC removed from the system. Note: deleted PVs won't be removed from the system also because of finalizers. This problem was fixed in [K8s 1.9.6](https://github.com/kubernetes/kubernetes/releases/tag/v1.9.6) in PR #61370 This problem is also fixed in K8s 1.11+ in PR #61324 However, this problem is not fixed in K8s 1.10, that's why I've cherry-picked the PR #61324 and proposing to merge it into K8s 1.10. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes N/A Related issue: #60764 **Special notes for your reviewer**: **Release note**: ```release-note In case StorageObjectInUse feature is disabled and Persistent Volume (PV) or Persistent Volume Claim (PVC) contains a finalizer and the PV or PVC is deleted it is not automatically removed from the system. Now, it is automatically removed. ```

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 18, 2018

k8s-github-robot added the do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. label Mar 18, 2018

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 18, 2018

k8s-ci-robot requested review from enj, msau42 and spiffxp March 18, 2018 17:33

pospispa force-pushed the 60764-K8s-1.10-StorageObjectInUseProtection-downgrade-issue branch from d4bd21b to e3229aa Compare March 18, 2018 17:38

pospispa mentioned this pull request Mar 18, 2018

[job failed] 1.9-master upgrade|downgrade jobs #60764

Closed

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. kind/bug Categorizes issue or PR as related to a bug. labels Mar 18, 2018

k8s-ci-robot added this to the v1.10 milestone Mar 19, 2018

k8s-github-robot added the milestone/incomplete-labels label Mar 19, 2018

pospispa force-pushed the 60764-K8s-1.10-StorageObjectInUseProtection-downgrade-issue branch from e3229aa to 1c9fe2c Compare March 19, 2018 09:09

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 19, 2018

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 19, 2018

pospispa changed the base branch from release-1.10 to master March 19, 2018 12:34

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 19, 2018

jdumars removed this from the v1.10 milestone Mar 19, 2018

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 20, 2018

deads2k reviewed Apr 20, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 20, 2018

pospispa force-pushed the 60764-K8s-1.10-StorageObjectInUseProtection-downgrade-issue branch from 7e0d723 to 81c2747 Compare April 20, 2018 17:06

pospispa force-pushed the 60764-K8s-1.10-StorageObjectInUseProtection-downgrade-issue branch from 81c2747 to d3ddf7e Compare April 20, 2018 17:55

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2018

k8s-github-robot merged commit 663c6ed into kubernetes:master Apr 21, 2018

pospispa mentioned this pull request Apr 21, 2018

cherry-pick into K8s 1.10: Always Start pvc-protection-controller and pv-protection-controller #62938

Merged

liggitt mentioned this pull request Jan 22, 2021

Add KEP for job tracking in implementable status kubernetes/enhancements#2308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Always Start pvc-protection-controller and pv-protection-controller #61324

Always Start pvc-protection-controller and pv-protection-controller #61324

pospispa commented Mar 18, 2018

pospispa commented Mar 18, 2018

pospispa commented Mar 18, 2018

dims commented Mar 19, 2018

liggitt commented Mar 19, 2018

liggitt commented Mar 19, 2018

k8s-ci-robot commented Mar 19, 2018

pospispa commented Mar 19, 2018

pospispa commented Mar 19, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

deads2k Apr 20, 2018

pospispa Apr 20, 2018

deads2k Apr 20, 2018

deads2k commented Apr 20, 2018

deads2k commented Apr 20, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

deads2k commented Apr 20, 2018

msau42 commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

fejta-bot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018 •

edited

Loading

k8s-github-robot commented Apr 21, 2018

Always Start pvc-protection-controller and pv-protection-controller #61324

Always Start pvc-protection-controller and pv-protection-controller #61324

Conversation

pospispa commented Mar 18, 2018

pospispa commented Mar 18, 2018

pospispa commented Mar 18, 2018

dims commented Mar 19, 2018

liggitt commented Mar 19, 2018

liggitt commented Mar 19, 2018

k8s-ci-robot commented Mar 19, 2018

pospispa commented Mar 19, 2018

pospispa commented Mar 19, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

deads2k Apr 20, 2018

Choose a reason for hiding this comment

pospispa Apr 20, 2018

Choose a reason for hiding this comment

deads2k Apr 20, 2018

Choose a reason for hiding this comment

deads2k commented Apr 20, 2018

deads2k commented Apr 20, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

pospispa commented Apr 20, 2018

deads2k commented Apr 20, 2018

msau42 commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

fejta-bot commented Apr 20, 2018

k8s-github-robot commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018 • edited Loading

k8s-github-robot commented Apr 21, 2018

k8s-ci-robot commented Apr 20, 2018 •

edited

Loading