Multiple worker types

[odarriba]

Feature Request

Feature

There are some situations in which the complete cluster has no sense with the same instance types, as different services requires different specs.

It would be awesome to be able to define different "worker pools" (as GKE defines them) with different instance types.

Tradeoffs

The pros are the ability to create workers with different specifications, so a better usage of the machines can be made.

The cons, probably the difficulty to match the specs when refreshing state of the cloud infrastructure and, probable, that it is a feature not everyone is going to use. But it would be nice to have that.

(And thanks to all contributors for this awesome project ❤️ )

[dghubble]

The google-cloud module was broken into controllers and workers early on, in anticipation of matching GKE's worker pool functionality in future. Its plausible you could use it already. There is no documentation or testing of this ability since it hasn't formally been made a feature.

[dghubble]

GCP worker pools will be available as an advanced feature in the next release, with #148

[odarriba]

hey there!

Thanks for the update!

I was thinking originally in a more general solution (i'm currently using DigitalOcean as provider). Something like being able that you want 3 workers of 4Gb and 4 of high CPU), maybe using tags or something 🤔

However, thanks a lot for the update and the continuous effort on this!

[dghubble]

You can effectively do that with the worker pools feature. However, I'd only like to add it on AWS and Google Cloud since they have a notion of managed instance groups (and ASGs on AWS) that are a good fit for thinking about groups of homogeneous nodes.

Perhaps after AWS and GCP worker pools are mature. But for now, I think you'd want to carry a few patches that change the DO workers to be what you want.

[dghubble]

I don't know how you're envisioning tags being used to enable this.

[odarriba]

Well, to be honest, I don't know too much about Terraform internals as to do this by my own (but I'll try, for sure)

My point about the tags is that the most difficult thing (from my point of view, there are possibly plenty of other things harder in this) is to know which instances are from each group (if there isn't a provider-solution like worker pools on GCP).

So my idea is that maybe using a pool name and adding it as a tag to the instance, we can know how many instances are there of each pool and update them as needed.

Again, I'm not keen on Terraform internals and how Typhoon are made, that was my main point of asking here if that would be possible.

Regards!

[dghubble]

The next release will support "worker pools" on AWS and GCP for attaching groups of workers with different properties to an existing cluster. AWS and GCP have a way to represent a homogeneous group of machines. https://typhoon.psdn.io/advanced/worker-pools/

• For Digital Ocean, we could define groups of workers entirely in a Typhoon module, but its extra work to support and a little messy (remote_exec's). I'd rather just wait for Digital Ocean to provide the equivalent group management feature, bearing in mind "no is temporary, yes is forever". There are pros/cons to each provider, and Digital Ocean missing some of the niceties of larger providers is a tradeoff.
• For bare-metal, machines are assumed to be heterogeneous already so this is already supported.

Thanks for the point about labels. I think it'd be a relatively minor detail for each group of workers to have a different label to allow users to identify them properly. Definitely not the sticking point.

[dghubble]

I've put Digital Ocean worker pools further down on the roadmap as a "maybe" to re-evaluate. https://github.com/poseidon/typhoon/wiki/Roadmap

[odarriba]

Cool! Thank you! El 4 mar. 2018 0:05, "Dalton Hubble" <notifications@github.com> escribió:

…

I've put Digital Ocean worker pools further down on the roadmap as a "maybe" to re-evaluate. https://github.com/poseidon/typhoon/wiki/Roadmap — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#127 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAiYv5DO3KaPKPxQosjRU14hHD49MBf6ks5tayGxgaJpZM4SGv3N> .