Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFE] Support for an explicit and reasonable way for custom whereabouts.kubeconfig path (e.g. k3s) #465

Open
dannypv05261 opened this issue Apr 29, 2024 · 4 comments · May be fixed by #466
Assignees
Labels
enhancement New feature or request

Comments

@dannypv05261
Copy link

dannypv05261 commented Apr 29, 2024

Is your feature request related to a problem? Please describe.

When whereabouts.kubeconfig is not located in the default CNI folder /etc/cni/net.d/whereabouts.d in host (k3s e.g. /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig). There is no explicit and reasonable way to configure kubeconfig value in whereabouts.conf due to mixing up the container internal path and the host file path as a variable.

Below is the snapshot of code in install-cni.sh.
With changing to customize path with env variable CNI_CONF_DIR, I can make kubeconfig value ={custom_path}/whereabouts.d/whereabouts.kubeconfig in whereabouts.kubeconfig (WHEREABOUTS_KUBECONFIG_LITERAL), but no whereabouts.kubeconfig and whereabouts.conf will be in host eventually since changing on CNI_CONF_DIR also affect internal container path

CNI_CONF_DIR=${CNI_CONF_DIR:-"/host/etc/cni/net.d"}
WHEREABOUTS_RECONCILER_CRON=${WHEREABOUTS_RECONCILER_CRON:-30 4 * * *}

# Make a whereabouts.d directory (for our kubeconfig)

mkdir -p $CNI_CONF_DIR/whereabouts.d
WHEREABOUTS_KUBECONFIG=$CNI_CONF_DIR/whereabouts.d/whereabouts.kubeconfig
WHEREABOUTS_FLATFILE=$CNI_CONF_DIR/whereabouts.d/whereabouts.conf # Yuki~ Nikhil's note: imo we should remove "flatfile" from whereabouts vocabulary and call this "WHEREABOUTS_CONF_FILE" instead. Flatfile may be the format but it's confusing naming.
WHEREABOUTS_KUBECONFIG_LITERAL=$(echo "$WHEREABOUTS_KUBECONFIG" | sed -e s'|/host||')

.....

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_LITERAL}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}

The workaround I used below is to make container internal CNI folder path same as host folder path by charging the volumeMount path cni-net-dir and adding env variable CNI_CONF_DIR with same path as below, but it is not a good practice for users to change volumeMount path. The good practice should have the fixed internal path /host/etc/cni/net.d, users map their volume to the cni-net-dir and define custom settings in env variable.

      env:
        - name: CNI_CONF_DIR
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d"

        volumeMounts:
        - name: cni-net-dir
          # mountPath: /host/etc/cni/net.d
          mountPath: /var/lib/rancher/k3s/agent/etc/cni/net.d
        - name: cron-scheduler-configmap
          mountPath: /cron-schedule

    volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Another workaround is to create a link file from /etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig to the custom path. But it seems not a good idea to apply additional changes in host.

Describe the solution you'd like

In install-cni.sh, change variable from WHEREABOUTS_KUBECONFIG_LITERAL to the existing env variable WHEREABOUTS_KUBECONFIG_FILE_HOST

  touch $WHEREABOUTS_FLATFILE
  chmod ${KUBECONFIG_MODE:-600} $WHEREABOUTS_FLATFILE
  cat > $WHEREABOUTS_FLATFILE <<EOF
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "${WHEREABOUTS_KUBECONFIG_FILE_HOST}"
  },
  "reconciler_cron_expression": "${WHEREABOUTS_RECONCILER_CRON}"
}
EOF

I submitted a request for that in case it is useful. #466

In this way, if I need to use the custom path, the only changes in daemonset-install.yaml are in volumes and env. It aligns with the general practice of the deployment file. It will not affect people using the default path as well.

      env:
        - name: WHEREABOUTS_KUBECONFIG_FILE_HOST
          value: "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"

      volumes:
        - name: cnibin
          hostPath:
            #path: /opt/cni/bin
            path: /var/lib/rancher/k3s/data/current/bin
        - name: cni-net-dir
          hostPath:
            #path: /etc/cni/net.d
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d

Describe alternatives you've considered
Maybe mark the workaround in doc / README.md if this PR doesn't accept.

Additional context

@manuelbuil
Copy link
Collaborator

I recently updated the K3s docs and now there is a guide on how to deploy Multus + Whereabouts setting the correct path. Please have a look: https://docs.k3s.io/networking/multus-ipams. It'd be great if you could give it a try!

@koendelaat
Copy link

koendelaat commented Jul 16, 2024

@manuelbuil

I recently updated the K3s docs and now there is a guide on how to deploy Multus + Whereabouts setting the correct path. Please have a look: https://docs.k3s.io/networking/multus-ipams. It'd be great if you could give it a try!

I've tested your guide on how to deploy Multus + Whereabouts. However this doesn't work for me out of the box.
Modifications I needed to make it work:

  1. Create symlink for /etc/cni/net.d/whereabouts.d/ to /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/

I think this makes sense, given this line in config.go:

confdirs := []string{"/etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.conf", "/etc/cni/net.d/whereabouts.d/whereabouts.conf", "/host/etc/cni/net.d/whereabouts.d/whereabouts.conf"}

Without this modification, I get the following error: error adding container to network <network-name> : config file not found

And to my understanding, this is relative to the node/host, not from any pod.

@dannypv05261
Copy link
Author

dannypv05261 commented Jul 16, 2024

@manuelbuil Sorry for missing your message and thanks for the effort to k3s community. I just run your helm in a separate k3s cluster, since I don't want to touch my existing environment that works well with workaround. Unfortunately, the result as @koendelaat mentioned.

Back to my initial intention, I am not asking for a workable solution on K3S, but I am raising a request to enhance the configurable CNI path. Correct me if I am wrong, the path looks configurable with 'volumes' section in daemonset-install.yaml and the env variable CNI_CONF_DIR in install-cni.sh but it seems not the case now. It looks misleading and ispent me some time on figuring out the issue. If I am just overthinking and the project actually has no plan to support variable CNI_CONF_DIR, it is fine to remain unchanged.

@koendelaat
Copy link

I'm looking for a solution for K3s, so created the issue there: k3s-io/k3s#10545

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants