Retry provisioning request #1351

mimowo · 2023-11-20T12:29:59Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Part of #1353

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Support for retry of provisioning request.

When `ProvisioningACC` is enabled, and there are existing ProvisioningRequests, they are going to be recreated.
This may cause job evictions for some long-running jobs which were using the ProvisioningRequests.

netlify · 2023-11-20T12:30:15Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`dcde2d8`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/655cf43b5fe005000878677f

mimowo · 2023-11-20T15:11:40Z

/kind feature
/kind api-change

mimowo · 2023-11-20T15:11:51Z

/hold
For KEP update

mimowo · 2023-11-21T12:20:28Z

/hold cancel
We have the KEP update PR: #1355

alculquicondor · 2023-11-21T16:57:12Z

pkg/controller/admissionchecks/provisioning/controller.go

+			if matches(req, wl.Name, checkName) {
+				prc, err := c.helper.ProvReqConfigForAdmissionCheck(ctx, checkName)
+				if err == nil && c.reqIsNeeded(ctx, wl, prc) && requestHasParamaters(req, prc) {
+					if currPr, exists := checkNameToPr[checkName]; !exists || getAttempt(ctx, currPr, wl.Name, checkName) < getAttempt(ctx, req, wl.Name, checkName) {


There should only be one ProvReq that is not Failed at a time. Furthermore, the most recent one is the one that we should care about. There shouldn't be a need to parse the number of attempts from the name.

OTOH, the counter as part of the name has a nice implication: you can't create two ProvReq with the same name, so we don't have to worry about having more than one active ProvReq when the informer cache is slightly out of date.

Also I would add two arguments:

we need to keep the number of attempts somewhere, we could use annotation, but this is API change and the intention is to backport to 0.5

it is possible that the request failed and the cloud provider needs some time to clean up nodes / resources corresponding to the old one. Starting new avoids the problem

alculquicondor

/approve

alculquicondor · 2023-11-21T16:58:12Z

pkg/controller/admissionchecks/provisioning/controller.go

+	return reconcile.Result{}, nil
+}
+
+func (c *Controller) getCheckNameToPr(ctx context.Context, wl *kueue.Workload, relevantChecks []string, ownedPrs []autoscaling.ProvisioningRequest) map[string]*autoscaling.ProvisioningRequest {


Suggested change

func (c *Controller) getCheckNameToPr(ctx context.Context, wl *kueue.Workload, relevantChecks []string, ownedPrs []autoscaling.ProvisioningRequest) map[string]*autoscaling.ProvisioningRequest {

func (c *Controller) activeOrLastPRNameForChecks(ctx context.Context, wl *kueue.Workload, relevantChecks []string, ownedPrs []autoscaling.ProvisioningRequest) map[string]*autoscaling.ProvisioningRequest {

Yes, this is more specific, rename done, but I use activeOrLastPRForChecks, to avoid a Get inside syncCheckStates

pkg/controller/admissionchecks/provisioning/controller.go

alculquicondor · 2023-11-21T17:19:33Z

test/integration/controller/admissionchecks/provisioning/provisioning_test.go

@@ -40,7 +42,7 @@ const (
 	customResourceTwo = "example.org/res2"
 )

-var _ = ginkgo.Describe("Provisioning", func() {
+var _ = ginkgo.Describe("Provisioning", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {


why this change?

I modify the vars, and different tests use different value, so I don't want the tests to run in parallel, but maybe this is not needed?

test/integration/controller/admissionchecks/provisioning/provisioning_test.go

pkg/controller/admissionchecks/provisioning/controller_test.go

alculquicondor · 2023-11-21T17:22:39Z

test/integration/controller/admissionchecks/provisioning/provisioning_test.go

+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Checking the admission check is pending", func() {


We don't need to check this. We only need to check that a new ProvReq appears and the check is still Pending by then.

Well. I have added the check, because this was a source of a bug in my initial implementation. I would transition to Rejected, and then back to Failed when the replacement provisioning request is added. So, I wanted to test that there is no flip to Rejected

test/integration/controller/admissionchecks/provisioning/provisioning_test.go

k8s-ci-robot · 2023-11-21T17:24:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alculquicondor · 2023-11-21T17:24:32Z

/remove-kind api-change

alculquicondor

/lgtm

k8s-ci-robot · 2023-11-21T18:05:10Z

LGTM label has been added.

Git tree hash: eaa1ae065806872292fd04bc9ba2ee7cbf9ce652

alculquicondor · 2023-11-21T18:05:22Z

/hold
actually, please squash

alculquicondor · 2023-11-21T18:31:00Z

/hold cancel
/lgtm

k8s-ci-robot · 2023-11-21T18:31:09Z

LGTM label has been added.

Git tree hash: ef4a44ddb8782acf7cf00a21007f3391bc91e011

tenzen-y · 2023-11-21T19:44:25Z

pkg/controller/admissionchecks/provisioning/controller.go

+func GetProvisioningRequestName(workloadName, checkName string, attempt int32) string {
+	fullName := fmt.Sprintf("%s-%s-%d", workloadName, checkName, int(attempt))
 	return limitObjectName(fullName)
 }


@mimowo @alculquicondor What happens if admins upgrade kueue v0.4 to the latest version?
I guess that ProvisioningRequests (PR) using the old name form will be recreated, and I'm wondering if this name form change will affect the PRs if CA is provisioning nodes triggered by the PRs.

Regarding PRs with CapacityAvailable, Provisioned, or Failed, I don't think that the above concerns exist.

It will be considered attempt 1.

It will be considered attempt 1.

You mean that PRs with old name forms wouldn't recreated, right? If so, that makes sense.
Thanks!

Actually, the old PRs need to be recreated as they would not pass the matches function.

It is easy to match old PRs (without the suffix) as attempt "1", but one thing to consider is that provisioning request wl-ac-1, would be matching to both ac and ac-1.

I think this is ok to leave as is, because there is no support for provisioning requests in Kueue 0.4, it starts in 0.5.0 and the ProvisioningACC is Alpha. See feature-stages.

@mwielgus @alculquicondor @trasc WDYT?

Added a release note.

It is always good to have additional safety and not loosing any in-flight requests, but additional work to perfect upgrade story for just released non-default, alpha-like feature seem to have relatively little impact and return on investment.

It is always good to have additional safety and not loosing any in-flight requests, but additional work to perfect upgrade story for just released non-default, alpha-like feature seem to have relatively little impact and return on investment.

That makes sense. I'm ok with leaving on recreation for ProvisioningRequest since the ProvisioningACC is the Alpha stage.

I believe that users don't use the featureGate yet in production.

@mimowo Thanks for updating the release note. The notification would be helpful :)

…ts (#1595) * Support retry for provisioning requests * Fix creation of the second provisioning request

alculquicondor · 2024-02-13T20:11:59Z

/release-note-edit

Support for retry of provisioning request.

When `ProvisioningACC` is enabled, and there are existing ProvisioningRequests, they are going to be recreated.
This may cause job evictions for some long-running jobs which were using the ProvisioningRequests.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 20, 2023

k8s-ci-robot requested review from tenzen-y and trasc November 20, 2023 12:30

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 20, 2023

mimowo force-pushed the retry-provisioning-request branch 8 times, most recently from 70b345a to 90dae37 Compare November 20, 2023 14:34

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Nov 20, 2023

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2023

mimowo force-pushed the retry-provisioning-request branch 2 times, most recently from c602a0a to b3a3eff Compare November 21, 2023 12:12

mimowo changed the title ~~WIP: Retry provisioning request~~ Retry provisioning request Nov 21, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 21, 2023

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 21, 2023

mimowo force-pushed the retry-provisioning-request branch 4 times, most recently from 8c033e5 to d25d221 Compare November 21, 2023 16:26

alculquicondor reviewed Nov 21, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 21, 2023

k8s-ci-robot removed the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Nov 21, 2023

alculquicondor reviewed Nov 21, 2023

View reviewed changes

k8s-ci-robot assigned alculquicondor Nov 21, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 21, 2023

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 21, 2023

Support retry for provisioning requests

dcde2d8

mimowo force-pushed the retry-provisioning-request branch from 4013407 to dcde2d8 Compare November 21, 2023 18:17

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 21, 2023

k8s-ci-robot requested a review from alculquicondor November 21, 2023 18:17

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Nov 21, 2023

k8s-ci-robot merged commit 92079c0 into kubernetes-sigs:main Nov 21, 2023
14 checks passed

k8s-ci-robot added this to the v0.6 milestone Nov 21, 2023

tenzen-y reviewed Nov 21, 2023

View reviewed changes

mimowo mentioned this pull request Nov 23, 2023

Support for configurable ProvisioningRequest retries #1353

Open

3 tasks

mimowo deleted the retry-provisioning-request branch November 29, 2023 14:57

alculquicondor mentioned this pull request Jan 8, 2024

Flaky Provisioning Request integration test #1558

Closed

This was referenced Jan 16, 2024

Fix syncing of the second provisioning request #1585

Merged

Automated cherry pick of #1351: Support retry for provisioning requests #1595

Merged

k8s-ci-robot pushed a commit that referenced this pull request Jan 17, 2024

Automated cherry pick of #1351: Support retry for provisioning reques…

574c3e6

…ts (#1595) * Support retry for provisioning requests * Fix creation of the second provisioning request

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry provisioning request #1351

Retry provisioning request #1351

mimowo commented Nov 20, 2023 •

edited by k8s-ci-robot

Loading

netlify bot commented Nov 20, 2023 •

edited

Loading

mimowo commented Nov 20, 2023

mimowo commented Nov 20, 2023

mimowo commented Nov 21, 2023

alculquicondor Nov 21, 2023

trasc Nov 21, 2023

alculquicondor Nov 21, 2023

mimowo Nov 21, 2023

alculquicondor left a comment

alculquicondor Nov 21, 2023

mimowo Nov 21, 2023 •

edited

Loading

alculquicondor Nov 21, 2023 •

edited

Loading

mimowo Nov 21, 2023

alculquicondor Nov 21, 2023

mimowo Nov 21, 2023

k8s-ci-robot commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

alculquicondor left a comment

k8s-ci-robot commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

k8s-ci-robot commented Nov 21, 2023

tenzen-y Nov 21, 2023

tenzen-y Nov 21, 2023 •

edited

Loading

alculquicondor Nov 21, 2023

tenzen-y Nov 21, 2023 •

edited

Loading

mimowo Nov 22, 2023

mimowo Nov 22, 2023

mwielgus Nov 22, 2023

tenzen-y Nov 23, 2023

tenzen-y Nov 23, 2023

alculquicondor commented Feb 13, 2024

	func (c Controller) getCheckNameToPr(ctx context.Context, wl kueue.Workload, relevantChecks []string, ownedPrs []autoscaling.ProvisioningRequest) map[string]*autoscaling.ProvisioningRequest {
	func (c Controller) activeOrLastPRNameForChecks(ctx context.Context, wl kueue.Workload, relevantChecks []string, ownedPrs []autoscaling.ProvisioningRequest) map[string]*autoscaling.ProvisioningRequest {

Retry provisioning request #1351

Retry provisioning request #1351

Conversation

mimowo commented Nov 20, 2023 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Nov 20, 2023 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

mimowo commented Nov 20, 2023

mimowo commented Nov 20, 2023

mimowo commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

alculquicondor Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

alculquicondor left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

alculquicondor commented Nov 21, 2023

k8s-ci-robot commented Nov 21, 2023

Choose a reason for hiding this comment

tenzen-y Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Feb 13, 2024

mimowo commented Nov 20, 2023 •

edited by k8s-ci-robot

Loading

netlify bot commented Nov 20, 2023 •

edited

Loading

mimowo Nov 21, 2023 •

edited

Loading

alculquicondor Nov 21, 2023 •

edited

Loading

tenzen-y Nov 21, 2023 •

edited

Loading

tenzen-y Nov 21, 2023 •

edited

Loading