Adding kube-proxy-replacement support in cilium

[MrFreezeex]

Signed-off-by: Arthur Outhenin-Chalandre arthur@cri.epita.fr

What type of PR is this?
/kind feature

What this PR does / why we need it:
Add kube-proxy-replacement support for a kube proxy free deployment with cilium https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/

It still needs #5554 to fix various kube_proxy_remove bugs...

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add new variable cilium_kube_proxy_replacement

[k8s-ci-robot]

Hi @MrFreezeex. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[floryut]

/ok-to-test

[MrFreezeex]

I have tested this PR with #5554 in strict mode and I have a working cluster without kube-proxy 🎉.

[MatthiasLohr]

Is there anything one can help to speed up things here?

[floryut]

Is there anything one can help to speed up things here?

as #5554 is now merged, could you rebase and we can see if the test succeed ?

[MrFreezeex]

I rebased !

Side note during my test I had issues on ubuntu 18 because of the old kernel version. If this fails again It's probably because of that, but let's see how It goes...

[MrFreezeex]

The test did failed... Probably due to the same issue I had. I changed the base distro to debian10 so It should works in the CI!

[MrFreezeex]

The CI job for cilium in strict mode has succeeded https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/638214670 ! This PR should be ready to be reviewed :).

[mmack]

Would like to see this merged... Can I help somehow? Test anything?

[floryut]

Would like to see this merged... Can I help somehow? Test anything?

There was a CI issue and the PR drop out of sight I guess.. But I restarted jobs this afternoon and was ready to lgtm :)
I'll take a look tomorrow and will try to put this in master

[floryut]

Well seems fine to me, agreed with the manual part as Cilium job is manual.
/lgtm

All yours @Miouge1

[Miouge1]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miouge1, MrFreezeex

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Miouge1]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mmack]

ok guys, i've tested this and i think we need to set the k8s api host and port at the cilium agent's:

        - name: KUBERNETES_SERVICE_HOST
          valueFrom:
            configMapKeyRef:
              key: k8sServiceHost
              name: cilium-config
              optional: true
        - name: KUBERNETES_SERVICE_PORT
          valueFrom:
            configMapKeyRef:
              key: k8sServicePort
              name: cilium-config
              optional: true

and in the vars:

k8sServiceHost: real_node_ip or apiserver LB host
k8sServicePort: 443

because the "normal" k8s api service is not reachable without a kube-proxy beforehand....
Also see this docs: https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/

[MrFreezeex]

ok guys, i've tested this and i think we need to set the k8s api host and port at the cilium agent's:

        - name: KUBERNETES_SERVICE_HOST
          valueFrom:
            configMapKeyRef:
              key: k8sServiceHost
              name: cilium-config
              optional: true
        - name: KUBERNETES_SERVICE_PORT
          valueFrom:
            configMapKeyRef:
              key: k8sServicePort
              name: cilium-config
              optional: true

and in the vars:

k8sServiceHost: real_node_ip or apiserver LB host
k8sServicePort: 443

because the "normal" k8s api service is not reachable without a kube-proxy beforehand....
Also see this docs: https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/

I also saw this in the docs but I didn't encouter any problem testing in strict mode without these variables. Also the test case added in this PR pass with an healthy cluster. Do you have an example of a setup in which a kube proxy less cilium deployment is not working ?

[mmack]

I guess it's not a problem with a running cluster b/c kube-proxy created the svc already.

But spawn a completely new cluster, then it won't work.

[MrFreezeex]

I guess it's not a problem with a running cluster b/c kube-proxy created the svc already.

But spawn a completely new cluster, then it won't work.

I did it with a new cluster though and the test is already doing that... What is your test setup ?

[mmack]

bare metal, ubuntu 20. wireguard tunnel between nodes, therefore ip = $internal_ip and ansible_host: $externalip.

loadbalancer_apiserver is set to a ip that balances to 2 etcd nodes which run the apiserver.

It worked all with kube-proxy and cilium.

[MrFreezeex]

bare metal, ubuntu 20. wireguard tunnel between nodes, therefore ip = $internal_ip and ansible_host: $externalip.

loadbalancer_apiserver is set to a ip that balances to 2 etcd nodes which run the apiserver.

It worked all with kube-proxy and cilium.

Trying to replicate and fix in #6473.