Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: don't reuse clusters after test failure #38746

Merged

Conversation

andreimatei
Copy link
Contributor

We've had a case where a cluster got messed up somehow and then a bunch
of tests that tried to reuse it failed. This patch employes a big hammer
and makes it so that we don't reuse a cluster after test failure (which
failure can be cluster related or not).

Release note: None

@andreimatei andreimatei requested a review from tbg July 8, 2019 20:42
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r1.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained

We've had a case where a cluster got messed up somehow and then a bunch
of tests that tried to reuse it failed. This patch employes a big hammer
and makes it so that we don't reuse a cluster after test failure (which
failure can be cluster related or not).

Release note: None
@andreimatei andreimatei force-pushed the roachtest.no-reuse-after-failure branch from 9d7c33d to b5f7a7e Compare July 9, 2019 14:55
Copy link
Contributor Author

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bors r+

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @tbg)

@craig
Copy link
Contributor

craig bot commented Jul 9, 2019

Build failed (retrying...)

@craig
Copy link
Contributor

craig bot commented Jul 9, 2019

Build failed (retrying...)

craig bot pushed a commit that referenced this pull request Jul 9, 2019
38570: opt: fix panic recovery for error handling r=RaduBerinde a=RaduBerinde

The major entry points in the optimizer catch all panics that throw an
error and converts them to errors. Unfortunately, this also catches
runtime errors (in which case we convert them to errors and lose the
stack trace).

This change adds a `ShouldCatch` helper which determines if we should
return a thrown object as an error. If the object is a
`runtime.Error`, it gets wrapped by an AssertionFailed error which
will cause correct error handling (stack trace, sentry reporting, etc).

As part of this change, we are also removing wrappers like
`builderError`, which are no longer useful. We fix the opt tester to
fail with the full error information (using `%+v`) for assertion
errors.

Release note: None

38660: opt: push limit into offset r=ridwanmsharif a=ridwanmsharif

This change pushes the limit into an offset whenever possible.
This shouldn't worsen any plan but does allow the `GetLimitedScans`
rule to fire in more scenarios.

Fixes #30416.
~~This is currently blocked on #38659.~~

Release note: None

38743: roachtest: skip jepsen/multi-register r=god a=nvanbenschoten

There's no use running this every night until #36431 is fixed.

Release note: None

38746: roachtest: don't reuse clusters after test failure r=andreimatei a=andreimatei

We've had a case where a cluster got messed up somehow and then a bunch
of tests that tried to reuse it failed. This patch employes a big hammer
and makes it so that we don't reuse a cluster after test failure (which
failure can be cluster related or not).

Release note: None

38766: scripts/release-notes.py: help the user with --from/--until r=lhirata a=knz

Requested by @lhirata

Release note: None

Co-authored-by: Radu Berinde <radu@cockroachlabs.com>
Co-authored-by: Ridwan Sharif <ridwan@cockroachlabs.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Co-authored-by: Raphael 'kena' Poss <knz@cockroachlabs.com>
@craig
Copy link
Contributor

craig bot commented Jul 9, 2019

Build succeeded

@craig craig bot merged commit b5f7a7e into cockroachdb:master Jul 9, 2019
@andreimatei andreimatei deleted the roachtest.no-reuse-after-failure branch September 11, 2019 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants