core: plumbing to support numa aware scheduling #18681
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shoenig
force-pushed
the
numa-support-ce
branch
from
36a4516
to
290960a
Compare
shoenig
force-pushed
the
numa-support-ce
branch
from
ae294ab
to
613df11
Compare
tgross
reviewed
Oct 17, 2023
Comment on lines
+3123
to
+3125
| // Compatibility will translate the LegacyNodeCpuResources into NodeProcessor | ||
| // Resources, or the other way around as needed. | ||
| func (n *NodeResources) Compatibility() { |
There was a problem hiding this comment.
When servers are upgraded before clients, the old clients will be transmitting the old fingerprints but that only happens if they have a new fingerprint to send (not on heartbeat). And the scheduler won't typically call Copy or Merge, so if we upgrade and then dequeue an eval that reads an old node before we read that new fingerprint then the node won't have these compatibility updates.
So maybe we could call this from FSM restore, so that we immediately fix-up anything that's in Raft. I think that would also let us move this into a Canonicalize method on Node as well.
There was a problem hiding this comment.
Oh nice catch; added a call to Compatibility() inside Node.Canonicalize() in e79311c
Handle the case where an upgraded server dequeus an evaluation before a client triggers a new fingerprint - which would be needed to cause the compatibility fix to run. By running the compat fix on restore the server will immediately have the compatible pseudo topology to use.
nvanthao
pushed a commit
to nvanthao/nomad
that referenced
this pull request
Mar 1, 2024
* core: plumbing to support numa aware scheduling * core: apply node resources compatibility upon fsm rstore Handle the case where an upgraded server dequeus an evaluation before a client triggers a new fingerprint - which would be needed to cause the compatibility fix to run. By running the compat fix on restore the server will immediately have the compatible pseudo topology to use. * lint: learn how to spell pseudo
nvanthao
pushed a commit
to nvanthao/nomad
that referenced
this pull request
Mar 1, 2024
* core: plumbing to support numa aware scheduling * core: apply node resources compatibility upon fsm rstore Handle the case where an upgraded server dequeus an evaluation before a client triggers a new fingerprint - which would be needed to cause the compatibility fix to run. By running the compat fix on restore the server will immediately have the compatible pseudo topology to use. * lint: learn how to spell pseudo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the bulk of implementation for NUMA-aware scheduling, a feature coming for Nomad Enterprise 1.7. See NMD-187 (internal) for details.
The CE specific bits are in
nomad/job_endpoint_hook_numa_ce.goandscheduler/numa_ce.go.Note: e2e, documentation, changelog, etc. will be in followup PRs.
Overview of changes
task.resources.numa.affinity = <string>NodeResourcesCpuResourcesis removed except for client<->server upgrade compatibilityresources.numain CEresources.numamockpackage now providesMockBasicTopology()to be used instead of manually setting cpu resources in testscoreSelector.Select()(for CE here) for choosing cpu cores in the same way as they were previouslyRankNode