kube-aws: Node pools

[mumoshu]

I'd like to add GKE's "Node Pools" like feature on top of kube-aws so that, without maintaining multiple etcd clusters and kubernetes controller planes, mainly for cost optimization purpose:

(1) I can differentiate

• instance type
• storage size
• instance profile
• security group
• or vice versa..

for each node pool

(2) Make a node pool a spot fleet

Do you think it would be appropriate to implement it in kube-aws or in another repo?

cc @colhom

[colhom]

@mumoshu to clarify- are you talking about having multiple specifications for worker nodes?

workers:
  - instanceType: m3.medium
    count: 4
    labels:
       ....
  - instanceType: t2.small
    count: 20
    labels:
       ....

[mumoshu]

@colhom Yes. It would also be good to have subnets configurable per pool like

workers:
  - instanceType: m3.medium
    count: 4
    labels:
      ...
    subnets:
    - availabilityZone: ap-northeast-1a
       instanceCIDR: "10.0.0.0/24"
  - instanceType: m3.medium
    count: 4
    labels:
      ...
    subnets:
    - availabilityZone: ap-northeast-1b
       instanceCIDR: "10.0.1.0/24"

so that we can also provide a way to solve #668

[FrederikNJS]

I'm starting to migrate our docker containers to kubernetes, and would like most of our services to run on fairly small machines (m3.medium, 3.75 GB Memory), but a few of our containers require lots of memory (16 GB each).

It would be fantastic if it was possible to run a cluster with mixed instance types, such that the lightweight containers could live on the small instances, and the big containers could live on the big instances.

I know that it's possible to make the containers prefer specific nodes using the labels, I just need to be able to run the bigger instances as part of the cluster.

[mumoshu]

I'm thinking of how I could build a POC of this.

I'd like to have something consistent with today's kube-aws regarding UI/UX.
Also, we should take into account that we may bring up several node-pools.
Thus the approach discussed in #667 (comment), which can bloat single cluster.yaml and cfn template, should be revised.

For the time being, I've come up with something like:

kube-aws node-pools init --pool-name my-pool
#=> generates node-pools/my-pool/node-pool.yaml like `kube-aws init`

# in the node-pool.yaml, you'd be able to customize instance type, EBS volume size, spot price, etc
# also I guess we don't need to expose every customization point via command-line flags at the beginning

kube-aws node-pools render --pool-name my-pool
#=> generates node-pools/my-pool/stack-template.json like `kube-aws render` or `kube-aws render stack`

kube-aws node-pools up --pool-name my-pool
#=> creates cloud formation stack named "<cluster name>-my-pool" like `kube-aws up`

kube-aws node-pools up --pool-name my-pool --export
#=> writes a cloudformation stack template into a local file like `kube-aws up --export`

kube-aws node-pools update --pool-name my-pool
#=> rolling-update nodes managed by the ASG/Launch Configuration for the node pool like `kube-aws update` to reflect changes in `node-pool.yaml`

...which is heavily inspired by the node-pools feature and its related gcloud commands for GKE.
Not very creative but after researching GKE and gcloud, I got to believe it is well designed so I'm trying to borrow as much from it.

Any thoughts, suggestions, etc?

[mumoshu]

Closing, as I'd like to discuss further in the new repo 💨