roachtest: schemaChangeStep is skipped in acceptance/version-upgrade #58489
Comments
cockroach-teamcity
added
branch-master
|
(roachtest).acceptance/version-upgrade failed on master@339275585b7d30b9ee2d49b0c696b9ddb8d51ad4: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
The first failure has the same stack trace as the mixed version test #58523 |
|
@irfansharif could you take a look at this? |
|
(roachtest).acceptance/version-upgrade failed on master@0d6f0ddd0958a134887623be44da33f6726eac85: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
(roachtest).acceptance/version-upgrade failed on master@eba887a1c9bd96269cabc22a5b16b041a49699c5: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
Same thing as described in #58523 (comment). |
|
Another failure here: https://teamcity.cockroachdb.com/viewLog.html?buildId=2585565&buildTypeId=Cockroach_UnitTests |
|
(roachtest).acceptance/version-upgrade failed on master@cd180429b80542b8fe0c66899bad21f6be5211af: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
This test is flaking quite a bit. I am hesitant to skip it since it feels like an important test (and not super sure how to skip roachtests). Do we think we will have a fix soon? |
|
I will be looking on Monday, before US working hours.
…On Fri, Jan 15, 2021, 20:24 RaduBerinde ***@***.***> wrote:
This test is flaking quite a bit. I am hesitant to skip it since it feels
like an important test (and not super sure how to skip roachtests). Do we
think we will have a fix soon?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#58489 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABGXPZC5IB3ID3GWM2VDYV3S2CI63ANCNFSM4VXCPSJQ>
.
|
|
(roachtest).acceptance/version-upgrade failed on master@cd54fb728636a4ec365dcd14ca921f39f067ea69: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
(roachtest).acceptance/version-upgrade failed on master@a51f2e212a71ee356a01f14510238baac65e76e7: More
Artifacts: /acceptance/version-upgrade
See this test on roachdash |
|
That last failure has no node crashes. I'll need to investigate it separately. |
|
It looks like the "no node crashes" failure is very dominant. I get it 80% of the time (20% of the time it passes - I think the crash failure mode is rarer). The failure is always on cockroach/pkg/cmd/roachtest/versionupgrade.go Lines 257 to 259 in a7afb38 Right after a node has been updated. When I look at the cluster after, it is perfectly healthy. Since this is so frequent, and can't have been going on for very long, I will try my hand at a bisection. Here's what the failure looks like: I am mildly worried that some quirk of my local system is at play. |
|
Re-enabling this isn't trivial the schemachange workload needs version gates in multiple places. Additionally, even without that, this test is exposing real issues in older releases. The right thing here is probably to drop the GA blocker tag and focus on enabling it again on the next release |
rafiss
added
GA-blocker
and removed
branch-release-22.1
|
@fqazi you've closed this, but the issue is still being referenced in the upgrade roachtest, and |
Previously, due to flakes we disabled schema changes inside the version update test. This patch re-enables them, since we are confident that the workload itslef is now stable in a mixed version state. Fixes: cockroachdb#58489 Release note: None
Previously, due to flakes we disabled schema changes inside the version update test. This patch re-enables them, since we are confident that the workload itslef is now stable in a mixed version state. Fixes: cockroachdb#58489 Release note: None
98792: kvserver: unskip `TestNewTruncateDecision` r=erikgrinaker a=erikgrinaker Passed after 10k stress runs. Has been skipped since 2019, issue seems to have been fixed in the meanwhile. Resolves #38584. Epic: none Release note: None 98855: roachtest: enable schema changes in acceptance/version-upgrade r=fqazi a=fqazi Previously, due to flakes we disabled schema changes inside the version update test. This patch re-enables them, since we are confident that the workload itslef is now stable in a mixed version state. Fixes: #58489 Release note: None 99023: kv: add log scope to BenchmarkSingleRoundtripWithLatency r=arulajmani a=nvanbenschoten Informs #98887. Avoids mixing logs with benchmark results, which breaks benchdiff. Release note: None 99033: storepool: set last unavailable on gossip dead r=andrewbaptist a=kvoli Previously, the `LastUnavailable` time was set in most parts of the storepool when a store was considered either `Unavailable`, `Dead`, `Decommissioned` or `Draining`. When `LastUnavailable` is within the last suspect duration (30s default), the node is treated as suspect by other nodes in the cluster. `LastUnavailable` was not being set when a store was considered dead due to the store not gossiping its store descriptor. This commit updates the `status` storepool function to do just that. Informs: #98928 Release note: None 99039: pkg/ccl/backupccl: Remove TestBackupRestoreControlJob r=benbardin a=benbardin This test has was marked skipped for flakiness, in 2018. Fixes: #24136 Release note: None Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com> Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com> Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Austen McClernon <austen@cockroachlabs.com> Co-authored-by: Ben Bardin <bardin@cockroachlabs.com>
exalate-issue-sync
bot
added
T-sql-foundations
(roachtest).acceptance/version-upgrade failed on master@cee475331ca3629b503cd2e7c7919b72c98a5ca5:
More
Artifacts: /acceptance/version-upgrade
Related:
roachtest: acceptance/version-upgrade failed #58329 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-note-script-update release-blocker
roachtest: acceptance/version-upgrade failed: invalid attempted write of database descriptor #58307 roachtest: acceptance/version-upgrade failed A-schema-descriptors C-bug C-test-failure O-roachtest branch-release-20.2
roachtest: acceptance/version-upgrade failed #58139 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-20.1
roachtest: acceptance/version-upgrade failed #53812 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2
See this test on roachdash
powered by pkg/cmd/internal/issues
Jira issue: CRDB-3386
The text was updated successfully, but these errors were encountered: