CSI volume test failing on newer Ubuntu 20.04 image

[AnishShah]

Cross-posting kubernetes/kubernetes#114221

What happened:

We have a Kubernetes cluster on Azure and AWS. Azure cluster is running on Ubuntu publisher: "Canonical" offer: "0001-com-ubuntu-server-focal" sku: "20_04-lts-gen2" version: "20.04.202211151"
and AWS cluster is running on Ubuntu ubuntu-minimal/images/hvm-ssd/ubuntu-focal-20.04-amd64-minimal-20221117

Azure cluster is running azuredisk-csi-driver and AWS cluster is running aws-ebs-csi-driver. The following test is failing for both the cluster consistently -

[Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works

and we notice the following error in Events -

FailedMount: MountVolume.MountDevice failed while expanding volume for volume "pvc-6e19b0fb-7491-4ec9-bc18-5308c2d8cd8a" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = Could not resize volume "vol-0ae95d7063d0afae1" ("/dev/sdc"):  resize of device /dev/sdc failed: exit status 1. resize2fs output: resize2fs 1.46.2 (28-Feb-2021)
resize2fs: Superblock checksum does not match superblock while trying to open /dev/sdc
Couldn't find valid filesystem superblock.

We created another with an older Ubuntu image (no other changes). Azure cluster is running publisher: "Canonical" offer: "0001-com-ubuntu-server-focal" sku: "20_04-lts-gen2" version: "20.04.202210180" and AWS cluster is running ubuntu-minimal/images/hvm-ssd/ubuntu-focal-20.04-amd64-minimal-20221018. The same test passed on this cluster.

This could be a Linux kernel issue?

What you expected to happen:

I expected the test to pass.

How to reproduce it:

Use the Ubuntu image mentioned above.

Anything else we need to know?:

Environment:

• CSI Driver version: v1.22.0
• Kubernetes version (use kubectl version): v1.25.4
• OS (e.g. from /etc/os-release): Ubuntu
• Kernel (e.g. uname -a): 5.15.0-1023-azure
• Install tools:
• Others:

[andyzhangx]

@AnishShah have you tried running resize2fs /dev/sdc manually on the node? and do you know whether it's the correct device name of the disk? On azure vm, we leverage udev rules to detect azure data disk, here is the udev rules script:

# cat /etc/udev/rules.d/66-azure-storage.rules
ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="Msft", ENV{ID_MODEL}=="Virtual_Disk", GOTO="azure_disk"
GOTO="azure_end"

LABEL="azure_disk"
# Root has a GUID of 0000 as the second value
# The resource/resource has GUID of 0001 as the second value
ATTRS{device_id}=="?00000000-0000-*", ENV{fabric_name}="root", GOTO="azure_names"
ATTRS{device_id}=="?00000000-0001-*", ENV{fabric_name}="resource", GOTO="azure_names"
ATTRS{device_id}=="?00000001-0001-*", ENV{fabric_name}="BEK", GOTO="azure_names"
# Wellknown SCSI controllers
ATTRS{device_id}=="{f8b3781a-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi0", GOTO="azure_datadisk"
ATTRS{device_id}=="{f8b3781b-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi1", GOTO="azure_datadisk"
ATTRS{device_id}=="{f8b3781c-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi2", GOTO="azure_datadisk"
ATTRS{device_id}=="{f8b3781d-1e82-4818-a1c3-63d806ec15bb}", ENV{fabric_scsi_controller}="scsi3", GOTO="azure_datadisk"
GOTO="azure_end"

# Retrieve LUN number for datadisks
LABEL="azure_datadisk"
ENV{DEVTYPE}=="partition", PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/../device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result", GOTO="azure_names"
PROGRAM="/bin/sh -c 'readlink /sys/class/block/%k/device|cut -d: -f4'", ENV{fabric_name}="$env{fabric_scsi_controller}/lun$result", GOTO="azure_names"
GOTO="azure_end"

# Create the symlinks
LABEL="azure_names"
ENV{DEVTYPE}=="disk", SYMLINK+="disk/azure/$env{fabric_name}"
ENV{DEVTYPE}=="partition", SYMLINK+="disk/azure/$env{fabric_name}-part%n"

LABEL="azure_end"