Enables resizing of block volumes. #81429

huffmanca · 2019-08-14T18:36:38Z

Enables resizing of block volumes.

What type of PR is this?
/kind bug

What this PR does / why we need it:
Fixes #79990

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Resolves an issue that prevented block volumes from being resized.

k8s-ci-robot · 2019-08-14T18:36:46Z

Hi @huffmanca. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

gnufied · 2019-08-14T18:59:45Z

/ok-to-test

gnufied · 2019-08-14T19:02:25Z

pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go

-	if !fsVolume {
-		klog.V(5).Infof("Block mode volume needn't to check file system resize request")
-		return
-	}
 	if processedVolumesForFSResize.Has(string(uniqueVolumeName)) {


Just to be on safe side, for in-tree drivers I think we should still skip calling fs resize on the node if volume is of type raw block.

The other thing is - we are going to need some e2e tests for this.

What if migration is enabled?

in-tree: no Pending condition on PVC if mode Block
CSI: always Pending condition on PVC
migrated CSI: ?

Our e2e test expects that Block volumes don't get the condition
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/testsuites/volume_expand.go#L190
I'm running the tests for aws migration.
So the test needs to know if the plugin is CSI or in-tree or migrated CSI which will be ugly.

That e2e is broken for CSI raw block volumes, no matter what we do here right? Plugin already knows if it is CSI or in-tree. I guess in-tree migrated CSI is tricky. But I think this ugliness will go away once we merge - container-storage-interface/spec#381

So our options are:

Keep in-tree drivers behaviour same as CSI, but since in CSI too we are moving towards skipping NodeExpandVolume for raw block devices, this seems like a step backward and will cause unnecessary churn.

Live with ugliness in e2e until we Add volume capability to volume expansion calls container-storage-interface/spec#381 is merged and fix the in-tree <-> csi drivers.

I am leaning towards #2. And yeah - I think we should not skip calling fs resize if driver is in-tree but migration is enabled for it.

I have opened #81611 to fix this problem. I think the condition checking there added very little value.

gnufied · 2019-08-14T19:03:08Z

/sig storage
/kind bug
/priority important-soon

gnufied · 2019-08-14T19:03:53Z

/assign

gnufied · 2019-08-20T17:50:51Z

pkg/volume/util/operationexecutor/operation_generator_test.go

+			}
+
+			e1, e2 := expandVolumeFunc.OperationFunc()
+			if e1 != nil {


This could be written as one line:

if e1 != nil || e2 != nil { t.Fatalf("unable to call volume expansion: %v, %v", e1, e2) }

bswartz · 2019-08-21T21:00:18Z

pkg/volume/util/operationexecutor/operation_generator.go

@@ -1649,7 +1596,7 @@ func (og *operationGenerator) GenerateExpandInUseVolumeFunc(
 				resizeOptions.DevicePath = volumeToMount.DevicePath
 				dmp, err := volumeAttacher.GetDeviceMountPath(volumeToMount.VolumeSpec)


This is definitely the wrong path for a raw block volume. It doesn't even exist.

should be fixed now.

bswartz · 2019-08-21T21:08:00Z

/assign

Perform a no-op when volume is of type raw block Fix bug with checking volume mounts for readonly

gnufied · 2019-08-22T19:52:02Z

/retest

gnufied · 2019-08-22T20:03:44Z

/assign @msau42

msau42 · 2019-08-22T20:12:19Z

pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go

 	if processedVolumesForFSResize.Has(string(uniqueVolumeName)) {
 		// File system resize operation is a global operation for volume,
 		// so we only need to check it once if more than one pod use it.
 		return
 	}
-	if mountedReadOnlyByPod(podVolume, pod) {
+	// We need to be consistent with what we check.


Is this a todo? I don't understand the comment

This is not a todo. I was just saying - the check we were doing here was overly complex. We should use same value that we use for deciding whether to format the disk and when calling resize during mount process. There is no need for online resizing to check containers etc. I will reword this though.

msau42 · 2019-08-22T20:30:35Z

pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go

 	if processedVolumesForFSResize.Has(string(uniqueVolumeName)) {
 		// File system resize operation is a global operation for volume,
 		// so we only need to check it once if more than one pod use it.
 		return
 	}
-	if mountedReadOnlyByPod(podVolume, pod) {
+	// We need to be consistent with what we check.
+	if volumeSpec.ReadOnly {


Should we pass readonly down to the plugin and let the plugin decide if resize should be skipped? Or actually let it fail? What does it mean to resize a read only volume?

yeah that is a good idea and ties into discussion we had this yesterday in CSI spec meeting. Currently NodeExpandVolumeRequest can't carry over that information and @bswartz wanted readonly volumes to be expandable on nodes too. In his case - all the plugin would do is, rescan iscsi lun.

I think, we should not be making that change right now in this PR.

I'm wondering if volumeSpec.ReadOnly is sufficient enough, or do we also need to be checking container readonly too?

The other question I have is more philosophical. If a user requests expansion on a read-only volume, should we actually ignore it and return success? We didn't actually expand the filesystem to what they requested.

For CSI - in the long run we want the decision to be taken by the plugin(SP) and not the CO. For in-tree implementations we currently support, doing resizing of readonly volumes with filesystem on it - could result in filesystem corruption.

I'm wondering if volumeSpec.ReadOnly is sufficient enough, or do we also need to be checking container readonly too?

The container readonly is handled by CRI and is an option for bind mount. I think checking volumeSpec.Readonly should be sufficient for our use case.

Shouldn't the resizing attempt fail if the volume is attached/mounted as read only?

In many cases - expanding the filesystem does not care how volume is mounted. Expanding(resize2fs/xfsgrow) the volume sometimes just takes device path (not the mount path).

So the PV.readOnly field is only used for determining if volumes should be attached readonly: #70505

Is that the only readOnly setting you want to skip here?

Yes - so attached as ReadOnly and mounted(NodePublished) as ReadOnly are two different things. Reading the linked issue, it seems like we are not setting PersistentVolume.spec.CSIPersistentVolumeSource.readOnly anywhere but once we do - this code should be updated to respect that and not resize a readonly attached volume.

Again this is with the caveat that - once we have option to pass ReadOnly to NodeExpandRequest - we will have to update the call to pass it as a flag and let SP decide, how best to handle it.

gnufied · 2019-08-22T22:13:56Z

/retest

Fix tests

bswartz · 2019-08-23T03:01:42Z

Okay this patch works for me. All flavors of resize are successful.

I haven't tried multi attached resize yet, I expect that to fail still, but this is good to merge IMO.

msau42 · 2019-08-23T16:53:25Z

/retest
/lgtm
/approve

k8s-ci-robot · 2019-08-23T16:54:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: huffmanca, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/volumemanager/OWNERS~~ [msau42]
~~pkg/volume/awsebs/OWNERS~~ [msau42]
~~pkg/volume/azure_dd/OWNERS~~ [msau42]
~~pkg/volume/cinder/OWNERS~~ [msau42]
~~pkg/volume/csi/OWNERS~~ [msau42]
~~pkg/volume/flexvolume/OWNERS~~ [msau42]
~~pkg/volume/gcepd/OWNERS~~ [msau42]
~~pkg/volume/rbd/OWNERS~~ [msau42]
~~pkg/volume/util/operationexecutor/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gnufied · 2019-08-23T19:22:17Z

/retest

k8s-ci-robot requested review from davidz627 and jsafrane August 14, 2019 18:37

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 14, 2019

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 14, 2019

gnufied reviewed Aug 14, 2019

View reviewed changes

k8s-ci-robot assigned gnufied Aug 14, 2019

wongma7 mentioned this pull request Aug 16, 2019

Exclude Block volumes from online resize test #81322

Closed

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 20, 2019

huffmanca force-pushed the resize_block_volume branch 2 times, most recently from 694a0b7 to 31f65fb Compare August 20, 2019 17:05

gnufied reviewed Aug 20, 2019

View reviewed changes

huffmanca force-pushed the resize_block_volume branch from 31f65fb to 3c41a07 Compare August 20, 2019 19:08

bswartz reviewed Aug 21, 2019

View reviewed changes

k8s-ci-robot assigned bswartz Aug 21, 2019

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 22, 2019

gnufied force-pushed the resize_block_volume branch from 5c6a793 to 9a42cb9 Compare August 22, 2019 19:38

Included resizing for CSI-based block volumes.

7a4cdf5

Perform a no-op when volume is of type raw block Fix bug with checking volume mounts for readonly

gnufied force-pushed the resize_block_volume branch from 9a42cb9 to 55be1b3 Compare August 22, 2019 19:46

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 22, 2019

gnufied force-pushed the resize_block_volume branch 2 times, most recently from 03ac9fd to d62d8e8 Compare August 22, 2019 20:03

k8s-ci-robot assigned msau42 Aug 22, 2019

msau42 reviewed Aug 22, 2019

View reviewed changes

Fix devicePath for raw block expansion

9dbe0b3

Fix tests

gnufied force-pushed the resize_block_volume branch from d62d8e8 to 9dbe0b3 Compare August 23, 2019 02:49

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 23, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 23, 2019

k8s-ci-robot merged commit f105fef into kubernetes:master Aug 24, 2019

k8s-ci-robot added this to the v1.16 milestone Aug 24, 2019

gnufied mentioned this pull request Aug 26, 2019

Update volume expansion e2e to create pods for block expansion - so as NodeExpandVolume can be called #80002

Closed

wongma7 mentioned this pull request Jan 6, 2020

Mark PVC for FS resize iff it's not Block kubernetes-csi/external-resizer#47

Closed

b0b0haha mentioned this pull request Dec 30, 2021

FlexVolume: should skip calling fs resize on the node if volume is of type raw block #107256

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enables resizing of block volumes. #81429

Enables resizing of block volumes. #81429

huffmanca commented Aug 14, 2019

k8s-ci-robot commented Aug 14, 2019

gnufied commented Aug 14, 2019

gnufied Aug 14, 2019

wongma7 Aug 15, 2019

wongma7 Aug 15, 2019

gnufied Aug 15, 2019

gnufied Aug 19, 2019

gnufied commented Aug 14, 2019

gnufied commented Aug 14, 2019

gnufied Aug 20, 2019

bswartz Aug 21, 2019

gnufied Aug 22, 2019

bswartz commented Aug 21, 2019

gnufied commented Aug 22, 2019

gnufied commented Aug 22, 2019

msau42 Aug 22, 2019

gnufied Aug 22, 2019

msau42 Aug 22, 2019

gnufied Aug 22, 2019

msau42 Aug 22, 2019

msau42 Aug 22, 2019

gnufied Aug 22, 2019

msau42 Aug 22, 2019

gnufied Aug 23, 2019 •

edited

Loading

msau42 Aug 23, 2019

gnufied Aug 23, 2019

gnufied commented Aug 22, 2019

bswartz commented Aug 23, 2019

msau42 commented Aug 23, 2019

k8s-ci-robot commented Aug 23, 2019

gnufied commented Aug 23, 2019

		@@ -1649,7 +1596,7 @@ func (og *operationGenerator) GenerateExpandInUseVolumeFunc(
		resizeOptions.DevicePath = volumeToMount.DevicePath
		dmp, err := volumeAttacher.GetDeviceMountPath(volumeToMount.VolumeSpec)

Enables resizing of block volumes. #81429

Enables resizing of block volumes. #81429

Conversation

huffmanca commented Aug 14, 2019

k8s-ci-robot commented Aug 14, 2019

gnufied commented Aug 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Aug 14, 2019

gnufied commented Aug 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bswartz commented Aug 21, 2019

gnufied commented Aug 22, 2019

gnufied commented Aug 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied Aug 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Aug 22, 2019

bswartz commented Aug 23, 2019

msau42 commented Aug 23, 2019

k8s-ci-robot commented Aug 23, 2019

gnufied commented Aug 23, 2019

gnufied Aug 23, 2019 •

edited

Loading