Incorrect value set to environment variable KUBERNETES_SERVICE_HOST for secondary Tenant Partition (TP)

[Hong-Chang]

arktos/kubernetes will set environment variable KUBERNETES_SERVICE_HOST for kubernetes client to connect the API server. It seems hardcoded the same service address for all TPs which is wrong. Every TP is using different kubernetes service address.

What happened:

1. Deploy tp1, tp2, rp1, rp2 using Mizar cni
2. Mizar operator is working correctly in tp1 but failed in tp2
3. The error log from failed mizar operator: (In corresponding rp, mizar operator are using kubernetes client to execute command. But the kubernetes client failed to connect API server)
   2021-12-28T17:44:16.720831173Z stderr F [2021-12-28 17:44:16,720] urllib3.connectionpo [WARNING ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5540c52f10>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/default/configmaps
4. Kubernetes client are using "10.0.0.1" to connect to server. Either client from tp1 or from tp2 are using same "10.0.0.1". This is incorrect. Kubernetes client from tp2 shall using "10.1.0.1".
5. In different TP, using "cluster/kubectl.sh get services kubernetes", we will see tp1's cluster ip is "10.0.0.1" and tp2's cluster ip is "10.1.0.1". That's the reason we know kubernetes client from tp2 shall using "10.1.0.1".
6. Kubernetes client get API server address from environment variable "KUBERNETES_SERVICE_HOST".

So it seems arktos has an issue when setting environment variable "KUBERNETES_SERVICE_HOST". It's hardcoded to default one.

What you expected to happen:
Each tp shall set environment variable "KUBERNETES_SERVICE_HOST" according to their different kubernetes services ip.

How to reproduce it (as minimally and precisely as possible):
See "What Happened" section.

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):
  Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.9.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"2021-12-27T19:08:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"", Minor:"", GitVersion:"v0.9.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"2021-12-27T19:08:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  AWS

• OS (e.g: cat /etc/os-release):
  NAME="Ubuntu"
  VERSION="20.04.3 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.3 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• Kernel (e.g. uname -a):
  Linux ip-172-31-26-226 5.11.0-1022-aws endpoint resource should support tenant  #23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Network plugin and version (if this is a network-related bug):

• Others:

[Sindica]

This is not an issue in arktos. KUBERNETES_SERVICE_HOST cannot be set in RP as it connects to multiple TPs.

[Sindica]

Resolving as PR 1257 fixed this issue.