Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove non graceful OVN DB pod delete Kuttl test #299

Merged
merged 1 commit into from
Jun 10, 2024
Merged

Remove non graceful OVN DB pod delete Kuttl test #299

merged 1 commit into from
Jun 10, 2024

Conversation

lewisdenny
Copy link
Contributor

@lewisdenny lewisdenny commented Jun 7, 2024 •
edited

We are seeing a failure with the 06-delete-pods-non-graceful.yaml job
in OCP 4.15 Prow CI. After discussing with the ovn-operator maintainers
we decided this test is invalid. With both --force and
--grace-period=0 the database lock file is not being released and this
is something that requires hands on recovery which isn't being covered in
this test case.

JIRA: https://issues.redhat.com/browse/OSPRH-7515

@openshift-ci openshift-ci bot requested review from abays and frenzyfriday June 7, 2024 04:27
@openshift-ci openshift-ci bot added the approved label Jun 7, 2024
@lewisdenny lewisdenny force-pushed the workaround_for_ungraceful_ovndb_delete branch from 30e8cdf to bee0787 Compare June 7, 2024 05:38
@lewisdenny
Copy link
Contributor Author 

    case.go:364: failed in step 6-delete-pods-non-graceful
    case.go:366: command "" exceeded 180 sec timeout, context deadline exceeded

Seems the timeout needs to be increased to add a timeout -_-, can't find any docs but [1]

[1] https://github.com/kudobuilder/kuttl/blob/f36c6725b377b518302f67dae5b4cc321dd33f7a/crds/teststep_crd.yaml#L103

@lewisdenny lewisdenny force-pushed the workaround_for_ungraceful_ovndb_delete branch from bee0787 to a335954 Compare June 7, 2024 06:24
@booxter
Copy link
Contributor

booxter commented Jun 7, 2024

Addition of test case for --force --grace-period 0 was a mistake. This won't kill ovsdb-server, which will leave db file locked. This scenario is invalid to test - force delete should be used only when node is completely dead, I don't think we can emulate it in kuttl.

Please just remove the whole test case instead.

@lewisdenny
Remove non graceful OVN DB pod delete Kuttl test
0c7af4e 
We are seeing a failure with the `06-delete-pods-non-graceful.yaml` job
in OCP 4.15 Prow CI. After discussing with the ovn-operator maintainers
we decided this test is invalid. With both `--force` and
`--grace-period=0` the database lock file is not being released and this is something that requires hands on recovery which isn't being covered in this test case.

JIRA: https://issues.redhat.com/browse/OSPRH-7515
@lewisdenny lewisdenny force-pushed the workaround_for_ungraceful_ovndb_delete branch from a335954 to 0c7af4e Compare June 9, 2024 23:22
@lewisdenny lewisdenny changed the title Add time for OVN DBs to recover before asserting there status Remove non graceful OVN DB pod delete Kuttl test Jun 9, 2024
@lewisdenny
Copy link
Contributor Author

/retest

Seems pvc failed to terminate

NAME                                              STATUS        VOLUME                                  CAPACITY   ACCESS MODES   STORAGECLASS    AGE
ovndbcluster-nb-sample-etc-ovn-ovsdbserver-nb-1   Terminating   local-storage01-oko-01-wnfwc-master-1   10Gi       RWO,ROX,RWX    local-storage   4m53s
ovndbcluster-nb-sample-etc-ovn-ovsdbserver-nb-2   Terminating   local-storage04-oko-01-wnfwc-master-2   10Gi       RWO,ROX,RWX    local-storage   4m53s
``

karelyatin
karelyatin approved these changes Jun 10, 2024
View reviewed changes
Copy link
Contributor

@karelyatin karelyatin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jun 10, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Jun 10, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: karelyatin, lewisdenny

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [karelyatin,lewisdenny]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 2847b4c into openstack-k8s-operators:main Jun 10, 2024
5 checks passed
@lewisdenny lewisdenny mentioned this pull request Jun 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@booxter booxter booxter requested changes

@karelyatin karelyatin karelyatin approved these changes

@abays abays Awaiting requested review from abays

@frenzyfriday frenzyfriday Awaiting requested review from frenzyfriday

Assignees

@booxter booxter

@karelyatin karelyatin

Labels
approved lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@lewisdenny @booxter @karelyatin