Add proposal for GetPreferredAllocation() to TopologyManager KEP

[klueska]

This proposal adds an API to allow a device plugin to forward a "preferred allocation" to the devicemanager so it can incorporate this information into its allocation decisions. It leaves the devicemanager in charge of making the final allocation, but gives the plugin the chance to help influence it more directly.

Using this new API call, the devicemanager will call out to a plugin at pod admission time, asking it for a preferred device allocation of a given size from a list of available devices. One call will be made per-container for each pod.

The list of available devices passed to the GetPreferredAllocation() call do not necessarily match the full list of available devices on the system. Instead, the devicemanager treats the GetPreferredAllocation() call as a "last-level" filter on the set of devices it has to choose from after taking all TopologyHint information into consideration. As such, the list of available devices passed to this call will already be pre-filtered by the topology constraints encoded in the TopologyHint.

As such, the preferred allocation is not guaranteed to be the allocation ultimately performed by the devicemanager. It is only designed to help the devicemanager make a more informed allocation decision when possible.

When deciding on a preferred allocation, a device plugin will likely take internal topology-constraints into consideration, that the devicemanager is unaware of. A good example of this is the case of allocating pairs of NVIDIA GPUs that always include an NVLINK.

On an 8 GPU machine, with a request for 2 GPUs, the best connected pairs by NVLINK might be:

{{0,3}, {1,2}, {4,7}, {5,6}}

Using GetPreferredAllocation() the NVIDIA device plugin is able to forward one of these preferred allocations to the device manager if the appropriate set of decvices are still available. Without this extra bit of information, the devicemanager would end up picking GPUs at random from the list of GPUs available after filerting by TopologyHint. This API, therefore allows it to ultimately perform a much better allocationt , with very minimal cost.

If a plugin does not implement this new GetPreferredAllocation() method, then we should simply follow the strategy that exists today with no change (i.e. allocate devices directly from the available devices list).

However, if GetPreferredAllocation() is implemented, then the preferred allocation should be chosen over simply pulling devices at random from the available devices list.

There are 4 cases to consider:

1. TopologyManager disabled, GetPreferredAllocation() not implemented
2. TopologyManager enabled, GetPreferredAllocation() not implemented
3. TopologyManager disabled, GetPreferredAllocation() implemented
4. TopologyManager enabled, GetPreferredAllocation() implemented

With the TopologyManager disabled and GetPreferredAllocation() unimplemented, the existing strategy is to simply pull devices from the front of the available devices list -- this should go unchanged.

With the TopologyManager enabled and GetPreferredAllocation() unimplemented, the existing strategy is to pull devices from the available devices list, such that they have the desired NUMA affinity -- this should also go unchanged.

With the TopologyManager disabled and GetPreferredAllocation() implemented, the new strategy should be to prefer allocations from the list returned by GetPreferredAllocation() if possible, and fall back to pulling devices from the front of the available devices list if not.

With the TopologyManager enabled and GetPreferredAllocation() implemented, the new strategy should be to prefer allocations from the list returned by GetPreferredAllocation() such that they have the desired NUMA affinity presented by the TopologyManager.

If that is not possible, fall back to pulling devices at random from the available devices list, such that they have the desired NUMA affinity. In this way, we will always follow a best-effort policy for honoring preferred allocations specified by this interface. We will NOT fail pod admission due to it.

[k8s-ci-robot]

Hi @klueska. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ConnorDoyle]

/ok-to-test

[klueska]

One way to look at this proposal, is to think of it as a way of generating intra-device topology-aware allocation preferences from each plugin without having to expose any device specific topology information (e.g. NVLINK topologies) to the kubelet.

In this way, the TopologyManager can be restricted to only deal with common node-level topology constraints (e.g. NUMA node, PCIe bus, etc.), while still having a way of incorporating device-specific topology constraints into its allocation decisions.

[derekwaynecarr]

/assign

[derekwaynecarr]

this seems like a simple way to make the kubelet indirectly aware of a device specific affinity preference. just a couple questions to make sure we have a common understanding. in particular, what is the behavior if a device plugin on a node does not yet implement the new method? will we fallback gracefully?

[derekwaynecarr]

do we think this should be a v1beta2 version of the api? if a plugin did not implement the new method, will we fall back on old behavior?

[klueska]

If a plugin does not implement the new method, then there should be no change from existing behaviour. As such, I don't think this requires an API bump.

[klueska]

in particular, what is the behavior if a device plugin on a node does not yet implement the new method? will we fallback gracefully?

If a plugin does not implement the new method, then we should simply follow the strategy that exists today (i.e. allocate devices directly from the available devices list). If however, GetPreferredAllocations() is implemented, then one of the preferred allocations should be chosen over simply pulling devices at random from the available devices list.

There are 4 cases to consider:

1. TopologyManager disabled, GetPreferredAllocations() not implemented
2. TopologyManager enabled, GetPreferredAllocations() not implemented
3. TopologyManager disabled, GetPreferredAllocations() implemented
4. TopologyManager enabled, GetPreferredAllocations() implemented

With the TopologyManager disabled and GetPreferredAllocations() unimplemented, the existing strategy is to simply pull devices from the front of the available devices list -- this should go unchanged.

With the TopologyManager enabled and GetPreferredAllocations() unimplemented, the strategy is to pull devices from the available devices list, such that they have the desired NUMA affinity -- this should also go unchanged.

With the TopologyManager disabled and GetPreferredAllocations() implemented, the new strategy should be to prefer allocations from the list returned by GetPreferredAllocations() if possible, and fall back to pulling devices from the front of the available devices list if not.

With the TopologyManager enabled and GetPreferredAllocations() implemented, the new strategy should be to prefer allocations from the list returned by GetPreferredAllocations() such that they have the desired NUMA affinity. If that is not possible, and we are in strict mode, then fail pod admission. If that is not possible and we are in preferred mode, then fall back to pulling devices at random from the available devices list, such that they have the desired NUMA affinity.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[derekwaynecarr]

@klueska thanks for answering the question, do you see any further updates you would like to make for this?

[klueska]

There was an offline comment from @ipuustin a few months (!) back that I'm just getting around to adding here:

Regarding GetPreferredAllocations API: I think that in principle, if GetPreferredAllocations() had this type, it might be "future-proof" enough to implement this scheme:

rpc GetPreferredAllocations(PreferredAllocationsRequest) returns (PreferredAllocationsResponse) {}

message PreferredAllocationsForDomain {
ContainerAllocateRequest preferred_allocations = 1;
string resource_domain = 2;
int32 cost = 3;
}

message PreferredAllocationsResponse {
repeated PreferredAllcationsForDomain preferred_allocations_for_domain = 1;
}

The "cost" and "resource_domain" fields could be unspecified or const at this point. Even just having the one layer of indirection between the PreferredAllocationsResponse and ContainerAllocateRequest would help, so that the extra fields could be added later.

[klueska]

I think adding a level of indirection here is a reasonable thing to do. That way we can easily extend the repeatable element over time with more information than just the set of devices.

[klueska]

Updated KEP proposal based on feedback.

[klueska]

@kad had concerns that plugins might "misuse" this new API call to try and game the system for the preferred allocations they decide to inform the kubelet about.

Given that this API call is only called optionally, and is designed to be a "last-level-filter" rather than a definitive call to perform an allocation, I am not too worried about this actually happening in practice.

If we come up with a future design of the TopologyManager that wouldn't benefit from making this call, then we could simply stop calling it without consequence.

As it stands today, adding this call can give us a large benefit for little cost, and I think it's worth moving forward with it for this reason.

[derekwaynecarr]

@klueska @ipuustin thanks for collaborating on this.

i agree this is a nice incremental low-risk change that can make a big impact.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, klueska

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-node/OWNERS [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kad]

I provided that feedback verbally, in several discussions, but will write it also in here: the approach of "last level filter" even were is not doing actual allocation, but is able to influence decision that allocator in device manager will have only one choice - obey whatever plugin is said as result of filter. Gamification of allocations by 3rd party plugins can't be prevented in that way.

Simply not calling over time this method is not really a solution of deprecation either: it will be plugins in wild that are dependant on that call, and breaking them will be considered as bad experience. Removing methods from gRPC and versions skews are also not a cheap things.