support host iscsi simultaneously with kubelet iscsi (pvc)

[travisghansen]

As it stands the host iscsid and the iscsid inside the kubelet container will conflict with each other (whichever is launched first wins). If iscsi is not used for PVCs but is used by the host generally all will be well. If the host does not use iscsi but the cluster is using iscsi PVCs all will be well. If you need both things get messy.

I think there are 2 use-cases where this is important:

1. Where the host needs to use iscsi outside the cluster for other purposes
2. Mixing csi drivers which use iscsid (ie: NetApp trident et al) with in cluster legacy iscsi workloads (csi drivers tend to leverage the host daemon/binaries)

I'm currently working on a csi driver and bumped into this situation when deploying it to a cluster which has a 'legacy' iscsi provisioner already installed. Both work independently but once deployed jointly in the same cluster things blow up.

I'm not entirely sure how to solve this. I think it can be solved with the following:

1. mount /var/lib/iscsi and /etc/iscsi into kubelet container
2. create an iscsiadm wrapper script (as noted in the blog entry below) which simply invokes the host iscsiadm in a chroot inside the container (apparently the client binary needs to match the version of the running daemon)

Given that step 2 requires generally a full host mount of the root (/) step 1 may not be required.

Interested in hearing other thoughts/feedback to hopefully find a solution.

• Add support for iscsi volume #144
• https://engineering.docker.com/2019/07/road-to-containing-iscsi/

[travisghansen]

Another issue is during rke updates new kubelet images have a new iqn. This can lead to some pretty weird scenarios I think.

[travisghansen]

So I've prototyped this by doing the following:

Adding the following script to /usr/local/sbin/iscsiadm on the host:

#!/bin/bash

# https://engineering.docker.com/2019/07/road-to-containing-iscsi/

if [[ ${HOST_ISCSI} -eq 1 ]];then
  chroot ${HOST_ISCSI_ROOT:=/host} /usr/bin/env -i PATH="/sbin:/bin:/usr/bin:/usr/sbin" iscsiadm "${@:1}"
else
  /usr/bin/env -i PATH="/sbin:/bin:/usr/bin:/usr/sbin" iscsiadm "${@:1}"
fi

And subsequently adding the following to cluster.yml:

    extra_env:
      - "HOST_ISCSI=1"
      - "HOST_ISCSI_ROOT=/host-iscsi"
    extra_binds:
      - "/usr/local/sbin/iscsiadm:/usr/local/sbin/iscsiadm"
      - "/:/host-iscsi"

I've confirmed that sessions are accessible from host/kubelet/etc and all things are shared as appropriate. Would be great to consider some proper integration with rke.

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

Not stale.

[immanuelfodor]

I was just about to set up iSCSI both on the host and for k8s PVCs, thanks for keeping this issue open as it kept me from lots of headaches :)

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[immanuelfodor]

unstale

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[immanuelfodor]

I've found some OpenEBS install notes that also need mounting ISCSI from the host under RKE: https://docs.openebs.io/docs/next/prerequisites.html#rancher

services:
  kubelet: 
    extra_binds: 
    - "/etc/iscsi:/etc/iscsi"
    - "/sbin/iscsiadm:/sbin/iscsiadm"
    - "/var/lib/iscsi:/var/lib/iscsi"
    - "/lib/modules"
    - "/var/openebs/local:/var/openebs/local"
    - "/usr/lib64/libcrypto.so.10:/usr/lib/libcrypto.so.10"
    - "/usr/lib64/libopeniscsiusr.so.0.2.0:/usr/lib/libopeniscsiusr.so.0.2.0"

I haven't try setting this up (at least yet) but I leave this here as it might become handy if running into any problems later.

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

unstale

[superseb]

Can you clarify the ask for this issue? Are you asking for having what you described as solution to be integrated as default with like one flag that sets all of that up? Or is it not solved fully yet?

[travisghansen]

I’m hoping for some ‘supported’ way of dealing with it generally. The only sane way to handle this is ‘everything host’ but that may be a breaking change for people (but probably a good breaking change).

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

unstale

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

unstale

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

Unstale

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

Unstale

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[travisghansen]

Unstale

[github-actions]

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.