cannot add server target for server XXXXXXX: public ipv4 is required (missing_ipv4)

[TimoGoetze]

Great Tool. Thanks. I managed to setup a test cluster easily. Using a cloud project with private /16 network, one deployment vm with NAT on this host and the rest only private network. So what I want is to always use Hetzner Load Balancers for all my work load.

When I tried to setup a loadbalancer, everything works (like external-dns), as long I precreate one and provide the external IP and name of the clusterLB manually as annotation (LB-11). But when I want to totaly automate my setup I struggle on the following error and the Loadbalance state always pending. ..and no certs.. and no dns A record.. - BUT a Hetzner Loadbalancer was setup and is working - so I can reach my pods via the external IP in the Hetzner UI... - But no IP is given back to the cluster so that the rest can setup automatically.

Events:
  Type     Reason                  Age                From                Message
  ----     ------                  ----               ----                -------
  Warning  SyncLoadBalancerFailed  30s                service-controller  Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: target kubernetestest-master1: cannot add server target for server 55065427: public ipv4 is required (missing_ipv4)
  Warning  SyncLoadBalancerFailed  23s                service-controller  Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: target kubernetestest-pool-small-static-worker4: cannot add server target for server 55065456: public ipv4 is required (missing_ipv4)
  Normal   EnsuringLoadBalancer    13s (x3 over 36s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  10s                service-controller  Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: target kubernetestest-pool-small-static-worker1: cannot add server target for server 55065455: public ipv4 is required (missing_ipv4)

I read about the extra_args issue and I tried adding it to the cluster config. But it seems to be wrong or not the right fix:

1. try

kubelet_args:
  - cloud-provider=external

2. try:

kubelet:
      extra_env:
        - KUBELET_EXTRA_ARGS=--cloud-provider=external

I am still in the learning phase on this hetzner manager things. Is there a bug or am I doing something totally wrong?

Thanks,

cheers.
Timo

[vitobotta]

Hi, cloud-provider=external is already configured by hetzner-k3s automatically otherwise the CCM wouldn't work.

How are you creating the load balancer? With an ingress controller? Can you share the manifest with all the annotations?

[TimoGoetze]

After testing with exact configuration of create cluster example - only network to 10.201.0.0/16 due to private network range the picture is a bit different.

My special usecase is to deploy one application multiple times.
Initially I did not annotate an initial "general" loadbalancer. Instead I built a helm chart that creates own namespace, apache and service with annotated loadbalancer, so, that for each deploy an own hetzner-cloud-loadbalancer is created. This is wanted behaviour, because I want to see the traffic for each deployment and some might have special external firewall rules for users.
So my "no external ipV4 given back from hetzner" was for this setup.

Now, I used annotated Loadbalancer as written in create-cluster example which leads to an initial load balancer for the cluster. It seems, after the first one is created, for all my subsequent deployments private loadbalancers come up with external ip in cluster shown. each in an own namespace. BUT now..... external DNS is routing everything to the first loadbalancer- not to the ones created by the helm chart. so all A and AAA records have the wrong IP now.

For now, I simplified my helm chart and only use the first load balancer - this works as expected but does not have own IPs for each deployments Loadbalancer and no traffic measuement per deployment.

It will take some time to dig deeper in it, which I am short off right now.-. I might post a follow up with helm chart example. cheers and thanks for fast reply.

[vitobotta]

If I got it right, why do you need one load balancer for each application? You could use a single load balancer with an ingress controller and just create an ingress resource for each app.

By the way, I’m turning this into a discussion because it doesn’t seem to be an actual tool issue. Maybe others can add their thoughts too.