scripts (or k8s) presume to use first ethernet controller

[grahamwhaley]

I think, by default, the scripts will set up the k8s network hung off the first ether controller on the master node.
In my case, that is not correct - I have two ether cards, and my node pool is hung off the second one.

I'll see if I can figure out what needs to be told to use the second (or rather, a specific) network controller to talk to the nodes. If anybody wants to chip in with some input, please do :-)

[grahamwhaley]

To note. I'm running the setup scripts on a bare metal Clear Linux cluster btw, not in the vagrant VM setup.

[grahamwhaley]

From https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master
I probably need to:

--apiserver-advertise-address= argument to kubeadm init

as by default kubeadm picks the interface with the default gateway on it, and for me that is not the nic with the nodes on.
Do we need to add support to the script command line for this, or are we happy leaving it up to the user to modify to their needs?

[grahamwhaley]

I'm having trouble getting this to work yet.
I have a feeling maybe the ability to put this in the yaml init file was only added in 1.13.
See:
kubernetes/kubernetes#70761

I've tried adding in the option like:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
nodeRegistration:
  criSocket: /var/run/crio/crio.sock
  #localAPIEndpoint:
  #aPIEndpoint:
APIEndpoint:
  advertiseAddress: "192.168.xxx.yyy"
  bindPort: 6443
---

to the kubeadm.yaml with both the three different section name types (APIEndpoint and the two commented out), but no luck - I still end up with the IP address of my primary ether port.

I'm going to see if I can edit my ~/.kube/config and edit the cluster server IP address there and re-start the script part way through....

Any input/help appreciated here ;-)

[ganeshmaharaj]

@grahamwhaley yes, that is a thing that @mcastelino and i have observed and was something that we were discussing as well as that affects my vagrant setup too. we thought of settign it as an env variable that a user can set which the script can read and then add the necessary bits to kubeadm init.

[grahamwhaley]

grr, I tried to hack the IP into the kube config, but the cert is encoded to the IP address, so it nacked it....
sigh. So, looking for input on how to fix this in the kubeinit.yaml....

[krsna1729]

@grahamwhaley

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
nodeRegistration:
  criSocket: /var/run/crio/crio.sock
apiEndpoint:
  advertiseAddress: "192.52.100.11"
  bindPort: 6443
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  RuntimeClass: true
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: v1.12.0
controlPlaneEndpoint: "192.52.100.11:6443"
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServerExtraArgs:
  advertise-address: "192.52.100.11"
  feature-gates: RuntimeClass=true

results in kube config of

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <>
    server: https://192.52.100.11:6443
  name: kubernetes

But kubeadm join from node 2 did not work

[discovery] Trying to connect to API Server "192.52.100.11:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.52.100.11:6443"
[discovery] Failed to request cluster info, will try again: [Get https://192.52.100.11:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: net/http: TLS handshake timeout]
[discovery] Failed to request cluster info, will try again: [Get https://192.52.100.11:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: net/http: TLS handshake timeout]

[krsna1729]

@grahamwhaley i am testing in vagrant. Can you test the above in your setup? Just to make sure, the join failing is not specific to vagrant setup

[grahamwhaley]

@krsna1729 @mcastelino @ganeshmaharaj ...
thanks @krsna1729 - that worked for me!

 kubeadm join 192.168.42.1:6443 --token XXXXXXXX --discovery-token-ca-cert-hash sha256:YYYYYYYYYYYYYYYYYYYYYYY --cri-socket=/run/crio/crio.sock
[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

[discovery] Trying to connect to API Server "192.168.42.1:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.42.1:6443"
[discovery] Requesting info from "https://192.168.42.1:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.42.1:6443"
[discovery] Successfully established connection with API Server "192.168.42.1:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/run/crio/crio.sock" to the Node API object "sn-nuc001" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

and then once create_stack.sh had finished, as the user who ran the script:

$ kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
sn-nuc-ctl   Ready      master   7m48s   v1.12.3
sn-nuc001    NotReady   <none>   5m39s   v1.12.3

OK, so, we should discuss if those missing kernel modules on the node are an issue or if the node status of 'NotReady' is an issue, but things look connected across my second ether port!

[krsna1729]

@grahamwhaley /cc @mcastelino @ganeshmaharaj

[WARNING RequiredIPVSKernelModulesAvailable]

Currently list of modules pertaining to IPVS proxier loaded in clear (master and node) as per https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md.

$ cut -f1 -d " "  /proc/modules | grep -e ip_vs -e nf_conntrack
nf_conntrack_netlink
ip_vs_sh
ip_vs_wrr
ip_vs_rr
ip_vs
nf_conntrack

Seems like have we all. NOTE: Our kernel is 4.19.5-297.kvm and the check is looking for nf_conntrack_ipv4, which the readme says to use nf_conntrack instead for 4.19 and above.

This bug has been fixed in 1.13
https://sourcegraph.com/github.com/kubernetes/kubernetes@release-1.13/-/blob/pkg/util/ipvs/kernelcheck_linux.go#L47:17

vs

https://sourcegraph.com/github.com/kubernetes/kubernetes@release-1.12/-/blob/pkg/util/ipvs/kernelcheck_linux.go#L79:31

IPTables will be used as fallback in 1.12. So we could potentially ignore the warning for now.

NotReady

It is generally because of networking. Can you paste the output of

kubectl get po --all-namespaces -owide

Check if the canal pod on the second node is running, paste the pod describe output too if possible and there is a config file under /etc/cni/net.d/.

[grahamwhaley]

Hi @krsna1729 - I think the NotReady was transient, and all my nodes are now up and ready. I can post the pod info if you still want, but I don't think it's a real problem. If I see it again and it 'sticks', then I'll re-post.

[krsna1729]

@grahamwhaley ah yes the network pod runs as a daemonset pod which might take time to spin up on the new node added. Suspected it was transient, asked because of 5+ min in the get node output.

Can I close this?

[grahamwhaley]

Thanks @krsna1729 Yeah, I think the time anomaly is because sn-nuc001 was my first test node, and I have a feeling it did not shutdown/clean/delete cleanly before I ran up the 'full cluster' - so, the master thought it had still been online.
Sure, let me close this. I was expecting to maybe run into some issues wrt accessing prometheus etc. via the first ether nic when the nodes/setup are on the second nic - but I'll open a new Issue if I run into anything.