expose apiserver_port

[bcg62]

expose apiserver_port so users can set it back to 443 if they chose to not use 6443

I've booted this to validate it works

[dghubble]

Typhoon v1.10.5 switched kube-apiserver (along with load balancers, firewall rules, and security groups) from using 443 to 6443 across bare-metal, AWS, and DigitalOcean. The terraform-render-bootkube module only has an "unofficial, temporary, may be removed without notice" apiserver_port variable since Google Cloud TCP proxies still whitelist only certain ports (and don't include the Kubernetes default 6443). Otherwise, you shouldn't have trouble using 6443 now.

Once there is a solution on Google Cloud, the terraform-render-bootkube apiserver_port will be dropped. Typhoon clusters will all use 6443. kube-apiserver manifests can drop privileges that were only for low port binding. There's no intention of supporting a configuration option here.

Also, this move wasn't taken lightly or just to cause trouble. It helped enable the load balancer consolidations in v1.10.5 that reduced costs of running clusters. Alignment with upstream and consistency across DO and bare-metal were nice-to-have's.

• Change apiserver port from 443 to 6443 (except GCP) #248
• Change apiserver port from 443 to 6443 terraform-render-bootstrap#67

[bcg62]

it makes sense when you have a native LB service available to you but this its semi non existent in bare metal at the moment unless you use something like metallb which has its own issues with calico.

[dghubble]

Hm, switching bare-metal clusters was without difficulty for me. In the simple case, Typhoon only asks that there be some record resolving to controllers so no changes are needed.

When using your own software or hardware load balancer, the configuration change depends what you're using. But ordinary load balancing software is perfectly fine for balancing across some backends (nginx, haproxy, etc) - there's nothing special or Kubernetes-centric about the problem. Just N backend servers with a TCP service on 6443.

If you still require 443, you can keep this fork around. There's some time before https://github.com/poseidon/terraform-render-bootkube/blob/master/variables.tf#L108 is removed.

[bcg62]

Right, its not the functionality of the change, it's the practicality in this use case.

Cloud providers have a tight integration with LB services which typhoon can configure directly. With metal there's now a dependency to use an external LB service managed outside of typhoon which needs to be maintained in sidestep.

This very well may be worth the extra work for the added benefit of load balancing if you have LB infrastructure available to you, but not everyone does.

Additionally changing from 443 to use 6443 directly without an LB in-between can be cumbersome in physical networks with various layers of firewalls and access restrictions.

[dghubble]

With metal there's now a dependency to use an external LB service managed outside of typhoon which needs to be maintained in sidestep.

Changing the port did not introduce any new requirement to load balance apiservers.

• Those who just use the bare-bones DNS approach in the tutorial change nothing.
• Those who have been using their own custom load balancing, continue to use that custom load balancing. For new clusters beyond v1.10.5, configure the balanced port to be 6443 instead of 443 and adjust any firewall rules. Its true there's an assumption here - that those opting to load balance with a custom approach are able to configure that.

[bcg62]

maybe workaround is a better phrase than dependency.

the option to control what port is used is being removed, so going forward it must be 6443.

6443 isn't a standard port (which is understandable as its not meant to be public). Along these same lines 6443 wont be open in most physical/corporate firewalls and justification to open it could be challenging.

I do use the bare-bones DNS approach, but can't access the new port due to the above reasons.

So the last option is to use a LB or NAT to translate the ports.

On the other hand I do totally agree with the drive to drop privileges where applicable, in an ideal world it would be nice to have the flexibility for either configuration.

[dghubble]

Ah ok bare-bones DNS and your company's network prevents 6443 traffic.

In your shoes, I'd try to avoid introducing new load balancing infrastructure to workaround what is ultimately a policy / people problem. Perhaps, try to impress upon security/networking folks that kubernetes#34719 made the decision a while ago (~Oct 2016). Its by no means required, but certainly seems to be the expectation today. You might also mention that your network doesn't block flannel / Calico traffic (depending on which you use) and a number of other Kubernetes centric, but not necessarily standardized ports.

I totally sympathize with the slow moving corporations plights (got plenty of that sorta thing at work too) but I think if there are still those internal blockers, its best you use the patch you've proposed here and carry it until the port changes are permitted. I want to draw the line somewhere and I think merging the option would discourage switching and drag this out further. I do apologize for the change as I try to avoid them.

In the v1.10.5 release notes, I should have also that 6443 had to be permitted:

Users who exposed kube-apiserver on a WAN via their router/load-balancer will need to adjust its configuration (e.g. DNAT 6443). Most apiservers are on a LAN (internal, VPN-only, etc) so if you didn't specially configure network gear for 443, no change is needed. (possible action required