XFS: Superblock has unknown read-only compatible features (0x4) enabled #966
Comments
|
please provide steps to reproduce it. |
|
what is the kernel version? |
|
3.10.0-1062.18.1.el7.x86_64 |
|
steps to reproduce
|
|
i dont see any issue |
|
closing this as not a CSI issue, feel free to reopen if the issue persists |
|
The moment I do the following mount operation manually mkfs.xfs /dev/rbd0 -f meta-data=/dev/rbd0 isize=512 agcount=16, agsize=4096000 blks The mount operation is successful └─ceph--76d4bce9--fcf7--45c0--8483--9d17cf44ea36-osd--data--c7294b13--dadc--4524--8678--10697150f51f |
|
@Madhu-1 could you please reopen the iisue |
|
@chandr20 am not seeing this issue. but reopening again. if you can give exact steps to reproduce it am happy to do it again. |
|
steps to reproduce kernel (3.10.0-1062.18.1.el7.x86_64)
|
|
Having the same issue with Steps:
|
|
@iExalt please provide below details
|
|
At glance, rather than a kernel issue, my suspicion is that, this happens when we I could be wrong in this theory though. |
This theory can be validated by taking a look at the parent and cloned volume |
@humblec this is happening even without snapshot as mentioned in #966 (comment)
the same UUID is not possible, we always ensure the UUID generated is unique, even the mountpath is always unique isn't It as it is (PVC+volumeID)
the one mentioned #966 (comment) is not require the xfs_admin command |
|
@Madhu-1 I am referring @iExalt in your comment #966 (comment), the error is on |
|
Again, if this is because of |
|
@humblec I tried it again (and ran the commands on) another pvc, which was given /dev/rbd5. Sorry about the confusion. |
|
@Madhu-1 |
|
@humblec Will do tomorrow |
|
Just to add nothing has changed in v2.0.1 and 2.1.0 ceph-csi/pkg/rbd/nodeserver.go Lines 318 to 332 in be63187 ceph-csi/pkg/rbd/nodeserver.go Lines 392 to 406 in b38f2c5 |
|
I actually upgraded from v2.0.0 hoping that it would fix another error upon mounting a cloned PVC - interesting to know that nothing has changed though. |
|
@humblec Filesystem UUID is indeed the same, it makes sense that regenerating the UUID would fix the mount. Having said that, even though the PVC mounts successfully, it doesn't have my cloned data on it which concerns me. /dev/rbd1 is the original PVC How would I try mount option |
The problem happens when multiple PVCs with the same UUID are attached/mounted on a node. This can happen after creating a PVC from a snapshot, or cloning a PVC. make nouuid as the default mount option if the format type is xfs to avoid mounting issues. updates: #966 Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The problem happens when multiple PVCs with the same UUID are attached/mounted on a node. This can happen after creating a PVC from a snapshot, or cloning a PVC. make nouuid as the default mount option if the format type is xfs to avoid mounting issues. updates: #966 Signed-off-by: Madhu Rajanna <madhupr007@gmail.com> (cherry picked from commit 22a86c5)
The problem happens when multiple PVCs with the same UUID are attached/mounted on a node. This can happen after creating a PVC from a snapshot, or cloning a PVC. make nouuid as the default mount option if the format type is xfs to avoid mounting issues. updates: #966 Signed-off-by: Madhu Rajanna <madhupr007@gmail.com> (cherry picked from commit 22a86c5)
|
@chandr20 Thanks for the confirmation!!! Also appreciate for getting back with the result ! |
|
Our current e2e can not test this, as it requires a provisioner with CentOS-8 as base, running attachers on CentOS-7. Removing the |
|
While considering an approach to address this, I wonder if there is a reason why the provisioner does not do the full provisioning (and create the filesystem). Could you tell me why that is, @humblec or @Madhu-1? Would it not be cleaner to have a separation between provisioning and consumtion? Now there is a partial provisioning done, and the final steps are done in the NodeStageVolume procedure (creating a filesystem when it does not exist). |
Hmm -- would that be any better to have the controller map and mkfs the FS upon creation? The controller is a bottleneck right now as it is (since there is only 1 active) and it too could be a different kernel version than the worker nodes so it wouldn't address this issue, right? |
It would not address the issue. The |
Is there a specific part of the spec that you think ceph-csi is out of compliance with? |
the provisioner work is to just create the RBD images and it will be part of node plugin to make sure the image is available on a given path on the node where the PVC needs to be mounted. is it possible to check the mkfs supports reflink or not when formatting if supported use it else dont't use it? |
|
if we don't fix this issue before 3.0.0 we will hit it again. either we need to change our base image back to nautilus again or fix this issue in code. |
This would make the PVC unavailable in case it gets attached to a node that does not support reflink. It will be unpredictable and difficult for users to understand why a PVC can not be used on some nodes, but may work on others. We can run Shall we add an option to the StorageClass, like this: ---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: <cluster-id>
pool: rbd
reflink: disabled|enabled|autoCurrently reflink is only available for XFS, and disabling by default is the most suitable option:
|
Current versions of the mkfs.xfs binary enable reflink support by
default. This causes problems on systems where the kernel does not
support this feature. When the kernel the feature does not support, but
the filesystem has it enabled, the following error is logged in `dmesg`:
XFS: Superblock has unknown read-only compatible features (0x4) enabled
Introduce a check to see if mkfs.xfs supports the `-m reflink=` option.
In case it does, pass `-m reflink=0` while creating the filesystem.
The check is executed once during the first XFS filesystem creation. The
result of the check is cached until the nodeserver restarts.
Fixes: ceph#966
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Current versions of the mkfs.xfs binary enable reflink support by
default. This causes problems on systems where the kernel does not
support this feature. When the kernel the feature does not support, but
the filesystem has it enabled, the following error is logged in `dmesg`:
XFS: Superblock has unknown read-only compatible features (0x4) enabled
Introduce a check to see if mkfs.xfs supports the `-m reflink=` option.
In case it does, pass `-m reflink=0` while creating the filesystem.
The check is executed once during the first XFS filesystem creation. The
result of the check is cached until the nodeserver restarts.
Fixes: ceph#966
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Current versions of the mkfs.xfs binary enable reflink support by
default. This causes problems on systems where the kernel does not
support this feature. When the kernel the feature does not support, but
the filesystem has it enabled, the following error is logged in `dmesg`:
XFS: Superblock has unknown read-only compatible features (0x4) enabled
Introduce a check to see if mkfs.xfs supports the `-m reflink=` option.
In case it does, pass `-m reflink=0` while creating the filesystem.
The check is executed once during the first XFS filesystem creation. The
result of the check is cached until the nodeserver restarts.
Fixes: #966
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Describe the bug
A clear and concise description of what the bug is.
Environment details
ern.alert <1> Apr 21 09:28:18 xxxx-2-05-worker-0 kernel: XFS (rbd0): Superblock has unknown read-only compatible features (0x4) enabled.
kern.warning <4> Apr 21 09:28:18 xxxx-2-05-worker-0 kernel: XFS (rbd0): Attempted to mount read-only compatible filesystem read-write.
kern.warning <4> Apr 21 09:28:18 xxxx-2-05-worker-0 kernel: XFS (rbd0): Filesystem can only be safely mounted read only.
kern.warning <4> Apr 21 09:28:18 xxxx-2-05-worker-0 kernel: XFS (rbd0): SB validate failed with error -22.
daemon.info <30> Apr 21 09:28:18 xxxx-2-05-worker-0 kubelet: E0421 09:28:18.051150 5822 csi_attacher.go:329] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = mount failed: exit status 32
daemon.info <30> Apr 21 09:28:18 xxxx-2-05-worker-0 kubelet: Mounting command: mount
daemon.info <30> Apr 21 09:28:18 xxxx-2-05-worker-0 kubelet: Mounting arguments: -t xfs -o _netdev,defaults /dev/rbd0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a886c5d8-c6f7-4c5f-b9e9-e00acdb2b940/globalmount/0001-0009-rook-ceph-0000000000000001-60ac60d1-83b2-11ea-824a-8ae73a759cf9
daemon.info <30> Apr 21 09:28:18 xxxx-2-05-worker-0 kubelet: Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a886c5d8-c6f7-4c5f-b9e9-e00acdb2b940/globalmount/0001-0009-rook-ceph-0000000000000001-60ac60d1-83b2-11ea-824a-8ae73a759cf9: wrong fs type, bad option, bad superblock on /dev/rbd0, missing codepage or helper program, or other error.
daemon.info <30> Apr 21 09:28:18 xxxx-2-05-worker-0 kubelet: I0421 09:28:18.051177 5822 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Steps to reproduce
Steps to reproduce the behavior:
Actual results
Describe what happened
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
For example:
Any existing bug report which describe about the similar issue/behavior
The text was updated successfully, but these errors were encountered: