Update enqueue action, import overcommit plugin to limit pending jobs from inqueue. #1298

jiangkaihua · 2021-02-07T07:26:52Z

No description provided.

k82cn · 2021-02-07T12:52:54Z

pkg/scheduler/framework/session_plugins.go

@@ -655,7 +663,7 @@ func (ssn *Session) NodeOrderMapFn(task *api.TaskInfo, node *api.NodeInfo) (map[
 	return nodeScoreMap, priorityScore, nil
 }

-// NodeOrderReduceFn invoke node order function of the plugins
+// NodeOrderReduceFn invoke nodeOrder function of the plugins


seperate those typo to different PRs.

already deleted these.

huone1 · 2021-02-08T07:01:17Z

pkg/scheduler/plugins/overcommit/overcommit.go

+
+func (op *overcommitPlugin) OnSessionOpen(ssn *framework.Session) {
+	overcommitStartTime := time.Now().UnixNano()
+	op.pluginArguments.GetFloat64(&op.overCommitFactor, overCommitFactor)


add overCommitFactor illegal check

Added overcommit-factor validity check: if input < 1, print warning logs & use default value instead.

what happen if overcommit-factor is an invalid number?

what happen if overcommit-factor is an invalid number?

GetFloat64() do not return err, only print warning logs, so here initializes overcommit-factor with an illegal value -1. If its value is not updated after GetFloat64(), it will be processed the sam as condition 'input < 1'.

volcano/pkg/scheduler/framework/arguments.go

Line 51 in c0fe287

func (a Arguments) GetFloat64(ptr *float64, key string) {

Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

…ion. Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

william-wang · 2021-02-08T14:07:21Z

pkg/scheduler/plugins/overcommit/overcommit.go

+	}
+	klog.V(4).Infof("overcommit plugin starts, overCommitFactor: %f", op.overCommitFactor)
+	defer klog.V(4).Infof("overcommit plugin finishes, execution time: %dns",
+		time.Now().UnixNano()-overcommitStartTime)


Please remove time calculation in product, it only used for debug or analysis.

time calculation is already removed.

william-wang · 2021-02-08T14:17:01Z

pkg/scheduler/plugins/overcommit/overcommit.go

+			op.inqueueResource.Add(jobMinReq)
+			return true
+		}
+		klog.V(4).Infof("idle resource in cluster is overused, ignore job <%s/%s>",


change log from "idle resource in cluster is overused" to "resource in cluster is overused, not allow job <%s/%s> be inqueue"

modified this log.

Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

william-wang

/lgtm
/approve

Thor-wl · 2021-02-09T01:23:10Z

/lgtm
/approve

volcano-sh-bot · 2021-02-09T01:23:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiangkaihua, Thor-wl, william-wang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Thor-wl,william-wang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/scheduler/framework/session_plugins.go

k82cn · 2021-02-09T02:30:34Z

pkg/scheduler/plugins/overcommit/overcommit.go

+			" using default value: %f", op.overCommitFactor, defaultOverCommitFactor)
+		op.overCommitFactor = defaultOverCommitFactor
+	}
+	klog.V(4).Infof("Enter overcommit plugin ...")


move to the first line of this func.

k82cn · 2021-02-09T02:30:51Z

pkg/scheduler/plugins/overcommit/overcommit.go

+func (op *overcommitPlugin) OnSessionOpen(ssn *framework.Session) {
+	op.pluginArguments.GetFloat64(&op.overCommitFactor, overCommitFactor)
+	if op.overCommitFactor < 1.0 {
+		klog.Warningf("invalid input %f for overcommit-factor, reason: overcommit-factor cannot be less than 1,"+


s/invalid/Invalid

k82cn · 2021-02-09T02:35:25Z

pkg/scheduler/plugins/overcommit/overcommit.go

+	inqueue := api.EmptyResource()
+	for _, job := range ssn.Jobs {
+		if job.PodGroup.Status.Phase == scheduling.PodGroupInqueue {
+			inqueue.Add(api.NewResource(*job.PodGroup.Spec.MinResources))


MinResources maybe nil

k82cn · 2021-02-09T02:35:42Z

pkg/scheduler/plugins/overcommit/overcommit.go

+			inqueue.Add(api.NewResource(*job.PodGroup.Spec.MinResources))
+		}
+	}
+	op.inqueueResource = inqueue.Clone()


why clone it?

k82cn · 2021-02-09T02:37:26Z

pkg/scheduler/plugins/overcommit/overcommit.go

+		inqueue := api.EmptyResource()
+		inqueue.Add(op.inqueueResource)
+		if job.PodGroup.Spec.MinResources == nil {
+			klog.V(4).Infof("job <%s/%s> is bestEffort, allow it be inqueue", job.Namespace, job.Name)


k82cn · 2021-02-09T02:37:32Z

pkg/scheduler/plugins/overcommit/overcommit.go

+		//TODO: if allow 1 more job to be inqueue beyond overcommit-factor, large job may be inqueue and create pods
+		jobMinReq := api.NewResource(*job.PodGroup.Spec.MinResources)
+		if inqueue.Add(jobMinReq).LessEqual(idle) {
+			klog.V(4).Infof("sufficient resources, allow job <%s/%s> be inqueue", job.Namespace, job.Name)


k82cn · 2021-02-09T02:37:41Z

pkg/scheduler/plugins/overcommit/overcommit.go

+			op.inqueueResource.Add(jobMinReq)
+			return true
+		}
+		klog.V(4).Infof("resource in cluster is overused, not allow job <%s/%s> be inqueue",


volcano-sh-bot requested review from Thor-wl and william-wang February 7, 2021 07:26

volcano-sh-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 7, 2021

jiangkaihua mentioned this pull request Feb 7, 2021

[fix #1265] Update idle resources in enqueue action. #1278

Closed

jiangkaihua force-pushed the overcommit branch 8 times, most recently from d5a0744 to e721964 Compare February 7, 2021 12:43

k82cn reviewed Feb 7, 2021

View reviewed changes

jiangkaihua force-pushed the overcommit branch 3 times, most recently from 8bba1de to 5ff6c97 Compare February 8, 2021 06:55

huone1 reviewed Feb 8, 2021

View reviewed changes

jiangkaihua force-pushed the overcommit branch 2 times, most recently from 0c1a593 to b08bcc3 Compare February 8, 2021 09:52

Update enqueue action.

e30d844

Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

jiangkaihua force-pushed the overcommit branch from b08bcc3 to fce367d Compare February 8, 2021 10:00

Add overcommit plugin to limit pending jobs going through enqueue act…

7014b84

…ion. Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

jiangkaihua force-pushed the overcommit branch from fce367d to 7014b84 Compare February 8, 2021 11:57

william-wang reviewed Feb 8, 2021

View reviewed changes

jiangkaihua force-pushed the overcommit branch from 382814a to 07da59b Compare February 8, 2021 14:21

Update log for overcommit plugin.

763fdbf

Signed-off-by: jiangkaihua <jiangkaihua1@huawei.com>

jiangkaihua force-pushed the overcommit branch from 07da59b to 763fdbf Compare February 8, 2021 14:35

william-wang approved these changes Feb 8, 2021

View reviewed changes

volcano-sh-bot assigned william-wang Feb 8, 2021

volcano-sh-bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 8, 2021

volcano-sh-bot assigned Thor-wl Feb 9, 2021

Thor-wl merged commit 844ccef into volcano-sh:master Feb 9, 2021

k82cn reviewed Feb 9, 2021

View reviewed changes

pkg/scheduler/framework/session_plugins.go Show resolved Hide resolved

k82cn reviewed Feb 9, 2021

View reviewed changes

jiangkaihua deleted the overcommit branch February 20, 2021 02:20

jiangkaihua mentioned this pull request Sep 20, 2022

Add jiangkaihua as reviewer. #2497

Merged

jiangkaihua mentioned this pull request Dec 24, 2022

Request to be a member of Volcano community. volcano-sh/community#35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update enqueue action, import overcommit plugin to limit pending jobs from inqueue. #1298

Update enqueue action, import overcommit plugin to limit pending jobs from inqueue. #1298

jiangkaihua commented Feb 7, 2021

k82cn Feb 7, 2021

jiangkaihua Feb 8, 2021

huone1 Feb 8, 2021

jiangkaihua Feb 8, 2021

k82cn Feb 9, 2021

jiangkaihua Feb 9, 2021

william-wang Feb 8, 2021

jiangkaihua Feb 8, 2021

william-wang Feb 8, 2021

jiangkaihua Feb 8, 2021

william-wang left a comment

Thor-wl commented Feb 9, 2021

volcano-sh-bot commented Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

k82cn Feb 9, 2021

Update enqueue action, import overcommit plugin to limit pending jobs from inqueue. #1298

Update enqueue action, import overcommit plugin to limit pending jobs from inqueue. #1298

Conversation

jiangkaihua commented Feb 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

william-wang left a comment

Choose a reason for hiding this comment

Thor-wl commented Feb 9, 2021

volcano-sh-bot commented Feb 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment