You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cylc has load balancing logic (based on psutils) which we use during suite server selection.
Cylc will soon have the concept of platform groups which provide a neat configurable way to assemble a bunch of platforms behind an abstraction.
Put these two things together and it should be pretty simple to take a bunch of linux boxes and get Cylc to assemble them into a crude cluster with load-balancing.
This isn't the most important functionality in the world, but, since the required functionality is all implemented anyway it's mostly a configuration problem. Giving this issue the "small" flag, as it will take much less time to implement than it will to discuss.
Putting up an issue after conversations with @dpmatthews.
Example
Say we have a bunch of hosts named my_host_1, my_host_2 ... my_host_N and we want a system to pick a host based on load average, excluding any hosts with low RAM.
With rose host-select
This functionality is currently present in the soon-to-be-depreated rose host-select command:
[rose-host-select]
group{my_host_group}=my_host_1, my_host_2, ... my_host_N
# memory must be greater than 1GB to be considered for selection
thresholds{my_host_group}=thresholds{linux}=mem:1000
# pick the server based on server load average (persumably 15min average?)
method{my_host_group}=load
Use the rose host-select command for the host configuration to use this in a Cylc7 workflow.
First we need to define one platform per-host like so:
[platforms]
[[my_host_*]]
Then define the platform group like so:
[platforms]
[[my_platform_group]]
platforms = my_host_1, my_host2, .. my_host_N # perhaps we should support globs here?
So far so good, we now have a system which will pick one of our hosts at random.
Next we configure load balancing using the same "ranking" system as we use for suite server selection:
[platforms]
[[my_platform_group]]
platforms = my_host_1, my_host2, .. my_host_N # perhaps we should support globs here?ranking = """ # memory must be greater than 1GB to be considered for selection virtual_memory().available > 1000 # pick the server based on 15min server load average getloadavg()[2]"""
Just one problem, this ranking has to be "evaluated" on a host, but platform groups are configured in terms of platforms not hosts. In this case it's kinda simple since we only have one host-per-platform, however, this is not guaranteed.
Problem/Solution
How to handle this misalignment between hosts and platforms, options:
Don't handle it at all.
Expect admins to configure hosts sensibly.
From each platform in the group pick one host for selection using the configured method (random by default).
If a platform in a group has more than one host it will still work, but the results might not make sense.
Require that platforms in a group have exactly one host.
Extra validation step each time we load in the global config.
If you "break" the global config all running suites may shut down due to being unable to reload the global config.
Let's just not do this in the first place.
Users/sites have until we formally retire rose host-select to change their working practices or implement their own infrastructure.
Other?
[Personal preference for (1)]
The text was updated successfully, but these errors were encountered:
psutils
) which we use during suite server selection.platform groups
which provide a neat configurable way to assemble a bunch of platforms behind an abstraction.Put these two things together and it should be pretty simple to take a bunch of linux boxes and get Cylc to assemble them into a crude cluster with load-balancing.
This isn't the most important functionality in the world, but, since the required functionality is all implemented anyway it's mostly a configuration problem. Giving this issue the "small" flag, as it will take much less time to implement than it will to discuss.
Putting up an issue after conversations with @dpmatthews.
Example
Say we have a bunch of hosts named
my_host_1
,my_host_2
...my_host_N
and we want a system to pick a host based on load average, excluding any hosts with low RAM.With
rose host-select
This functionality is currently present in the soon-to-be-depreated
rose host-select
command:Use the
rose host-select
command for thehost
configuration to use this in a Cylc7 workflow.With Cylc
[platforms]
First we need to define one platform per-host like so:
[platforms] [[my_host_*]]
Then define the platform group like so:
So far so good, we now have a system which will pick one of our hosts at random.
Next we configure load balancing using the same "ranking" system as we use for suite server selection:
Just one problem, this ranking has to be "evaluated" on a host, but
platform groups
are configured in terms of platforms not hosts. In this case it's kinda simple since we only have one host-per-platform, however, this is not guaranteed.Problem/Solution
How to handle this misalignment between hosts and platforms, options:
rose host-select
to change their working practices or implement their own infrastructure.[Personal preference for (1)]
The text was updated successfully, but these errors were encountered: