Should we abandon the current 1 demand == 1 CPU constraint?

[maxfischer2781]

This is triggered by #209, where I started writing it. But it's not really a "we shouldn't do #209!", it's more of a "what does #209 tell us?".

---

I'm not sure the way we translate demand to "CPU Cores" right now is the correct approach.

As a reminder, COBalD doesn't care whether a Drone has 1, 2, 4, 8, … CPU Cores, just as it doesn't care whether a Drone has 1, 2, 3, … GB RAM/Core. COBalD doesn't care that a Drone has CPU Cores or RAM.
In fact, COBalD doesn't even care about Drones. In fact, COBalD doesn't even care about demand beyond it being a number.

So equating "1 demand" to "1 CPU Core" to "1 Drone / n CPU Cores" and trying to make things fit to that equality is a non sequitur. "1 demand" can mean whatever we want it to, and that is kind of the point.
If we get stuck in goose chases because we enforce this equality, perhaps we should just abandon it?

---

As I see it, Option 1 is that we stick to demand ~ Cores and Option 2 is that we don't.

For Option 1, if we want COBalD to increase/decrease in "integer Drones" only we must set the Controller to do so. E.g. for the LinearController that means interval * rate == n CPU/Drone.
If we stick with 8 CPU Drones, that's 1s * 8, 8s * 1, 16s * 1, 16s * 0.5 or whatever. Standardizing that to multiples of 8 helps with rounding errors but isn't strictly needed (as #209 shows, actually).

For Option 2, we can choose whatever demand actually means. E.g. we can say that "1 demand" means "1 Drone" and things are pretty straightforward. Or we can say that "1 demand" means "1 CPU x 1 GB RAM" and numbers are huge but comparable.
Especially if we do abandon granularity, I think that would actually be helpful. E.g. for the outstanding idea of probing empty pools, we just care that there is "more than 0" (every now and then) but the details of what that means are more of a backend decision.

[mschnepf]

I think one problem with demand is equal to the number of drones is that the quota limits the number of drones. Usually, the quota is defined in the number of CPU-cores. Therefore, the limiter from cobald can't be used to ensure that quota for drones with different numbers of CPU-cores.

[mschnepf]

I mean your first definition in first place. This limit, how many CPU cores should be provided is set by the user. That limit ensures that the provider does not get too many requests. Furthermore, that is a simple way to ensure a constant usage of resources for a longer time instead of using the hole grant in a short time. I think there are better ways to handle the usage over time.

[eileen-kuehn]

A question that comes to my mind: Is the demand really tied that strictly to user's limits? Do we mix up different conceptual things here? I feel I still have to think about it.

Update: I think the proper feedback-based functioning of supply and demand is independent from user limits. First, we need to ensure that those are working correctly; that demand does not have artificial values that do not have to do anything with reality e.g. demand >> supply. If this is as expected, that is drones are requested reasonably, we can further add limits on top capping demands and therefore limiting the supply. But the limits are in no case a proper solution to get the scaling of supply and demand working – this is a tool for the user to maybe spread quotas over a range of time, as mentioned by @mschnepf above. If we need to do this, it means we don't have a proper feedback loop in place.

[maxfischer2781]

Hm, I think you are mentioning two separate issues that we should handle separately.

• Having a straightforward handle for the cases where we do care about "Number of cores" primarily. These are currently still the vast majority in practice (*grudgingly hides GridKa hat*). @giffels also reminded me that we have composite pools of 1 & 8 core drones combined.
• Actually translating "CPU limits" that are in other systems to "COBalD limits". These I think should not be a motivation to stick to this. Instead we should use the proper means of COBalD to handle this – namely using a less naive Controller (I think that's also what @eileen-kuehn is saying.)

Any thoughts on this separation?

[giffels]

Sorry, for joining late. Generally, I like the approach of defining a drone as smallest resource managed in COBalD.

First I would like to provide a little bit more insight, why we used CPU cores in first place. In High Energy Physics everything is defined per Core. We have 2GB/Core memory and 15 GB/Core disk space, etc. So, we defined initially a CPU Core as smallest resource managed in COBalD also having the management of composite pools in mind.

Concerning the granularity I cannot remember why we actually introduced it. I checked the commit that introduced it to TARDIS, but the reason was not obvious. I can only imagine that we introduced it to make COBalD aware that increasing the demand by one does not mean one additional Drone. This probably needs further tests.

As @maxfischer2781 already mentioned the only issue that comes to my mind are the composite pools, where we manage for example single core and multi-core drones. I think this would probably not work, when defining a drone as smallest resource.

I think "CPU limits" is not a show stopper for using a drone as smallest resource. TARDIS knows how much cores a drone provides and I think we can simply translate CPU limits internally.

Last but not least, I think we should go this way, provide a prototype implementation and use our simulation/mocking frameworks to perform detailed tests of the new approach. In particular, in the direction of handling composite pools with the new approach.

[mschnepf]

I agree that it should be possible to implement "CPU limits" into TARDIS.

I'm not sure the composite pools with different types of drones are much of a problem. Large drones and small drones are requested at the same rate. This results in more resources, such as CPU cores, being allocated by the large drones than by the small drones. When the demand is equal to the number of CPU cores, there is an equal distribution of CPU cores between small and large drones. Additionally, we usually want to have larger drones to allow the overlay batch system more scheduling options, which hopefully increases the overall utilization of all drones. If the mix of drones provided is not well utilized, the feedback system should change the composition. However, this should be verified by simulation.