iSCSI mount fails starting from open-iscsi-2.1.10 release #11708
Comments
Have you opened an opensuse issue? Based on the information you've shared I don't see that there is anything to fix in k3s. The only involvement we have here is that the kubelet runs |
|
@brandond can you please re-open the issue? The bug is not caused by the open-iscsi package of openSUSE, but rather by changes done inside of open-iscsi upstream (sorry, I think I confused you by linking the package changelog instead of upstream's changelog). I can reproduce the same behaviour on a debian system:
On debian unstable I've also downgraded the open-iscsi from 2.1.10 to 2.1.8 and everything started working as expected. I think the real question is whether this is a bug of k3s or a bug of upstream kubernetes |
|
K3s is Kubernetes. We do not cary any patches to the Kubelet's volume mount code, and all the kubelet does is run |
|
Thanks for the clarification about the volume mount code. In the meantime I was trying to reproduce the issue with something different than k3s. I was able to reproduce that with RKE2. I'm going to open an issue against kubernetes/kubernetes. |
Environmental Info:
K3s Version: k3s version v1.31.5+k3s1 (56ec5dd), go version go1.22.10
However it happens also with older release of k3s like 1.29
Node(s) CPU architecture, OS, and Version:
Linux kubetest 6.4.0-150600.23.33-default #1 SMP PREEMPT_DYNAMIC Thu Jan 9 14:10:22 UTC 2025 (ba46628) x86_64 x86_64 x86_64 GNU/Linux
I've run into this issue after I upgraded the host node from openSUSE 15.5 to openSUSE 15.6, this changed the version of open-iscsi:
Cluster Configuration:
Single node cluster
Describe the bug:
After having upgraded my host from openSUSE Leap 15.5 to 15.6 I discovered I wasn't able to mount iSCSI volumes inside of my cluster.
The pod that uses the iSCSI volume remains in "ContainerCreating" phase forever. The journalctl shows only this message coming from the k3s service:
The node has successfully logged into the remote target:
However, no block device has been created:
The iSCSI target works fine when using
iscsadmmanually:As you can see, the
sdadevice was properly created.What is particularly strange is that everything is temporarily fixed by restarting the
iscsidservice, but it works only for mounting the target once. More details later.This what happens when the iSCSI target is not using authentication. The same problem happens when CHAP authentication is enabled.
I've run iscsid with debug messages turned on, but I didn't find any error. I've also changed the
k3sservice to start with--debug, however there was nothing relevant about iscsi being shown.Everything is working fine with openSUSE Leap 15.5. The version of open-iscsi between 15.5 and 15.6 received a small bump:
The package changelog can be found here. I got suspicious about some of the changes, I installed the open-iscsi-2.1.9 package on 15.6 and the bug disappeared.
Steps To Reproduce:
I've used VMs based on openSUSE-Leap-15.5-Minimal-VM.x86_64-Cloud.qcow2 and openSUSE-Leap-15.6-Minimal-VM.x86_64-Cloud.qcow2.
In both cases:
zypper in open-iscsi kernel-defaultzypper rm kernel-default-basecurl -sfL https://get.k3s.io | sh -Create the following pod:
The pod will be stuck in "ContainerCreating" forever.
Now perform the following operations:
systemctl restart iscsidThis time the pod will be created.
Now, delete the pod and recreate it. The pod creation will be stuck in "ContainerCreating" again.
Expected behavior:
Actual behavior:
Additional context / logs:
These are the messages obtained by running
journalctl -f | grep iscsi.Node fresh from boot, Pod scheduled:
The pod is in "ContainerCreating" status.
The pod is being deleted:
The
iscsidservice is restarted viasystemctl restart iscsid:The pod is created again:
The text was updated successfully, but these errors were encountered: