Consider adopting kube-aws

[linki]

This projects gets bigger and we're solving problems that are already solved by similar projects.

We could have a lot of synergistic effects by adopting and contributing to https://github.com/coreos/kube-aws.

One of the main issues we had before has been fixed in the v0.9.0 release of kube-aws, so it may makes sense to reconsider it:

• Discrete (and HA) etcd cluster

Similar features of kube-aws and this tool

• based on AWS / Cloud Formation / CoreOS
• supports cluster upgrades and node draining
• support for multi-zone worker nodes
• supports e2e tests
• allows to specify AMI for specialized cluster nodes

Desired by us and already in or planned for kube-aws

• initial spot fleet support
• initial node pools support
• self-hosted kubernetes deployment
• dedicated subnets for controller nodes
• secured via client certs/tls (worker->master->etcd)
• private node IPs
• correctly tainted nodes allows scheduling on masters
• can be used as a library
• yaml based definition
• golang

Things to check and could be a blocker to adopt kube-aws

• security based on client certs must be manageable

please feel free to comment and iterate on the points above

[linki]

a nice overview of the production readiness of their setup can be found here: kubernetes-retired/kube-aws#9

[linki]

the goal should be to entirely adopt it and work in close cooperation with the people building it currently.

[Raffo]

I agree that this can be a good approach and definitely better than create our own thing (i.e. admiral, the python script). Also, we already have contacts with CoreOS and would be easy/nice to contribute.
I would just try to consider and keep in mind what is the priority now: HackWeek is 5 weeks away, which means we have essentially 3-4 weeks of work left. kube-aws must definitely be kept into account for the mid and long term strategy but I would justify "hacky" solutions that brings the system "live".

[linki]

kube-aws now has experimental support for spot fleet kubernetes-retired/kube-aws#113

[linki]

kube-aws now also has experimental support for node pools
kubernetes-retired/kube-aws#46 (comment)

[mumoshu]

Hi @linki, thanks for sharing this issue.
Also; @Raffo, thanks for helping me in cluster-autoscaler development 🙇

I'm a kube-aws maintainer.

secured via client certs/tls (worker->master->etcd)

I don't remember exactly but I believe it is supported since several versions ago.

dedicated subnets for controller nodes
private node IPs
correctly tainted nodes allows scheduling on masters

I believe they're supported since kube-aws v0.9.4.

can be used as a library

👍
Btw what is your use-case for this?

It is just an idea at this stage but personally, I'd like to merge kube-aws into, say, kops.
https://groups.google.com/forum/#!topic/kubernetes-sig-aws/L1E2F2Easbw
I believe we need this feature to make kube-aws start interacting with kops in the first place.

golang

Instead of bash scripts? 😄 If so; I agree.
One thing I'd like to sort out is how to keep kube-aws easily customizable after translating all the bash scripts into go binaries.
Do we need to force users rebuild binaries/container images?

security based on client certs must be manageable

I'm interested to this. Let me guess; do you mean to revoke/rotate certs and keys without incurring downtime?

[mikkeloscar]

@mumoshu Thanks a lot for your input. I'm sorry you didn't get a response until
now.

This issue is quite old, and since it was created we have decided to implement
our own cluster provisioning tool which basically watches for changes to this
repository and applies the CloudFormation stack and Kubernetes manifests to
all relevant clusters.

The tool is currently closed source and not in a state were it could be so useful
for others (we are working towards this). We think our cluster configuration is
more useful, which is why we have this opensource here.

Our solution is very much inspired by kube-aws, but we choose not to use it
directly because kube-aws targets a broader audience, and we wanted to strip
out all the features we don't need to keep it simple.

I will close this issue for now, but ofc. we are still interested in advances
to kube-aws and other provisioning tools.