Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry clone kill which is flaky #1950

Merged
merged 3 commits into from
Sep 24, 2021

Conversation

awels
Copy link
Member

@awels awels commented Sep 20, 2021

Signed-off-by: Alexander Wels awels@redhat.com

What this PR does / why we need it:
[test_id:4000] Create a data volume and then clone it while killing the
container and verify retry count was pretty flaky lately. The likely culprit was the small amount of data being cloned. This causes the pods to be removed before we have a chance to connect issue the kill command.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

NONE

@kubevirt-bot kubevirt-bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Sep 20, 2021
@awels
Copy link
Member Author

awels commented Sep 20, 2021

/hold
going to retry the runs to ensure this really fixes the problem.

@kubevirt-bot kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 20, 2021
@awels
Copy link
Member Author

awels commented Sep 21, 2021

/test all

Copy link
Contributor

@brybacki brybacki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 1319:
err = utils.WaitTimeoutForPodReady(f.K8sClient, utils.UploadPodName(targetPvc), targetNs.Name, utils.PodWaitForTime)

This wait has polling interval set to 2 seconds. So, in worst case, it can miss any action that has finished in 2 seconds. I think in good conditions our upload server can handle tinyCoreIso in milliseconds.

[test_id:4000] Create a data volume and then clone it while killing the
container and verify retry count was pretty flaky lately. It was failing
on attempting to connect to the upload server pod to kill it. This PR
causes a retry.

Signed-off-by: Alexander Wels <awels@redhat.com>
to kill the process

Signed-off-by: Alexander Wels <awels@redhat.com>
Signed-off-by: Alexander Wels <awels@redhat.com>
@awels
Copy link
Member Author

awels commented Sep 22, 2021

/retest

@awels
Copy link
Member Author

awels commented Sep 23, 2021

/test all

@brybacki
Copy link
Contributor

hpp destructive has a problem in before each, looks like it cannot find CDI

s: "Unable to find pod containing cdi-deployment",
other than that, looks good, trying again

/test all

@brybacki
Copy link
Contributor

/retest

@awels
Copy link
Member Author

awels commented Sep 23, 2021

/test all

@awels
Copy link
Member Author

awels commented Sep 23, 2021

Yeah the destructive lane has a flake where it sometimes messes up the cdi object, which is one of the main reasons we put it in a separate lane to start with. I am mainly interested in seeing several runs where the clone test that is failing often right now doesn't fail.

@awels
Copy link
Member Author

awels commented Sep 23, 2021

/test pull-containerized-data-importer-e2e-k8s-1.19-ceph

@awels
Copy link
Member Author

awels commented Sep 23, 2021

/hold cancel
/test all
/approve

@kubevirt-bot kubevirt-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 23, 2021
@kubevirt-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awels

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 23, 2021
@brybacki
Copy link
Contributor

/lgtm

@kubevirt-bot kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2021
@brybacki
Copy link
Contributor

/retest

@kubevirt-bot kubevirt-bot merged commit 61684d3 into kubevirt:main Sep 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants