Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: failover/system-non-liveness/disk-stall failed #99930

Closed
cockroach-teamcity opened this issue Mar 29, 2023 · 1 comment · Fixed by #99963
Closed

roachtest: failover/system-non-liveness/disk-stall failed #99930

cockroach-teamcity opened this issue Mar 29, 2023 · 1 comment · Fixed by #99963
Assignees
@nicktrav
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 29, 2023 •
edited by exalate-issue-sync bot
Loading

roachtest.failover/system-non-liveness/disk-stall failed with artifacts on master @ bb5c97b5d2073a27e7ec88654b05ac913112f610:

test artifacts and logs in: /artifacts/failover/system-non-liveness/disk-stall/run_1
(cluster.go:1977).Run: output in run_132510.563447996_n1-7_echo-0-sudo-blockdev: echo "0 $(sudo blockdev --getsz /dev/sdb) linear /dev/sdb 0" | sudo dmsetup create data1 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_132510.570168842_n1_echo-0-sudo-blockdev.log: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-26146

Epic CRDB-20293
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 29, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 29, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Mar 29, 2023
@nicktrav
Copy link
Collaborator

This is most likely due to #99747. This test uses the infrastructure provided by the disk-stalled test.

@nicktrav nicktrav removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 29, 2023
@nicktrav nicktrav self-assigned this Mar 29, 2023
@nicktrav nicktrav added A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team and removed T-kv KV Team labels Mar 29, 2023
nicktrav added a commit to nicktrav/cockroach that referenced this issue Mar 29, 2023
@nicktrav
roachtest: use local SSDs for disk-stall failover tests
e1eb301 
The disk-stalled roachtests were updated in cockroachdb#99747 to use local SSDs.
This change broke the `failover/*/disk-stall` tests, which look for
`/dev/sdb` on GCE (the used for GCE Persistent Disks), but the tests
still create clusters with local SSDs (the roachtest default).

Fix cockroachdb#99902.
Fix cockroachdb#99926.
Fix cockroachdb#99930.

Touches cockroachdb#97968.

Release note: None.
@nicktrav nicktrav mentioned this issue Mar 29, 2023
Merged
nicktrav added a commit to nicktrav/cockroach that referenced this issue Mar 29, 2023
@nicktrav
roachtest: use local SSDs for disk-stall failover tests
08b5ef6 
The disk-stalled roachtests were updated in cockroachdb#99747 to use PDs in favor
of local SSDs. This change broke the `failover/*/disk-stall` tests,
which look for `/dev/sdb` on GCE (the used for GCE Persistent Disks),
but the tests still create clusters with local SSDs (the roachtest
default).

Fix cockroachdb#99902.
Fix cockroachdb#99926.
Fix cockroachdb#99930.

Touches cockroachdb#97968.

Release note: None.
craig bot pushed a commit that referenced this issue Mar 30, 2023
@btkostner @knz
@maryliag @yuzefovich @nicktrav
Merge #98899 #99607 #99690 #99750 #99962 #99963
331a672 
98899: feat: allow starting docker container via env variable r=rickystewart a=btkostner

Fixes #87043 by allowing you to specify args via the `COCKROACH_ARGS` env value instead of a command. This is required to be able to use the official cockroach image with GitHub actions via a service. More details in the issue.

Note, do to weirdness with merging commands and env values, I decided that setting the `COCKROACH_ARGS` would ignore any command given. This should reduce issues with people trying to use both ways of specifying args and instead force them to pick one.

Release note (general change): Allow setting docker command args via the `COCKROACH_ARGS` environment variable.

99607: sql: block DROP TENANT based on a session var r=stevendanna a=knz

Fixes #97972.
Epic: CRDB-23559

In clusters where we will promote tenant management operations, we would like to ensure there is one extra step needed for administrators to drop a tenant (and thus irremedially lose data). Given that `sql_safe_updates` is not set automatically when users open their SQL session using their own client, we need another mechanism.

This change introduces the new (hidden) session var, `disable_drop_tenant`. When set, tenant deletion fails with the following error message:

```
demo@127.0.0.1:26257/movr> drop tenant foo;
ERROR: rejected (via sql_safe_updates or disable_drop_tenant): DROP TENANT causes irreversible data loss
SQLSTATE: 01000
```

(The session var `sql_safe_updates` is _also_ included as a blocker in the mechanism so that folk using `cockroach sql` get double protection).

The default value of this session var is `false` in single-tenant clusters, for compatibility with CC Serverless. It will be set to `true` via a config profile (#98466) when suitable.

Release note: None

99690: ui: drop index with space r=maryliag a=maryliag

Previously, if the index had a space on its name,
it would fail to drop.
This commit adds quotes so it can be executed.

Fixes #97988

Schema Insights
https://www.loom.com/share/04363b7f83484b5da19c760eb8d0de21

Table Details page
https://www.loom.com/share/1519b897a14440ddb066fb2ab03feb2d

Release note (bug fix): Index recommendation to DROP an index that have a space on its name can now be properly executed.

99750: sql: remove no longer used channel in createStatsNode r=yuzefovich a=yuzefovich

This hasn't been used as of fe6377c. Also mark `create_stats.go` as owned by SQL Queries.

Epic: None

Release note: None

99962: ui: add checks for values r=maryliag a=maryliag

Fixes #99655
Fixes #99538
Fixes #99539

Add checks to usages that could cause
`Cannot read properties of undefined`.

Release note: None

99963: roachtest: use local SSDs for disk-stall failover tests r=andrewbaptist a=nicktrav

The disk-stalled roachtests were updated in #99747 to use local SSDs. This change broke the `failover/*/disk-stall` tests, which look for `/dev/sdb` on GCE (the used for GCE Persistent Disks), but the tests still create clusters with local SSDs (the roachtest default).

Fix #99902.
Fix #99926.
Fix #99930.

Touches #97968.

Release note: None.

Co-authored-by: Blake Kostner <git@btkostner.io>
Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: maryliag <marylia@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Nick Travers <travers@cockroachlabs.com>
@craig craig bot closed this as completed in #99963 Mar 30, 2023
@craig craig bot closed this as completed in 545d49a Mar 30, 2023
@blathers-crl blathers-crl bot mentioned this issue Mar 30, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Mar 30, 2023
@nicktrav
roachtest: use local SSDs for disk-stall failover tests
24d115e 
The disk-stalled roachtests were updated in #99747 to use PDs in favor
of local SSDs. This change broke the `failover/*/disk-stall` tests,
which look for `/dev/sdb` on GCE (the device name used for GCE
Persistent Disks), but the tests still create clusters with local SSDs
(the roachtest default).

Fix #99902.
Fix #99926.
Fix #99930.

Touches #97968.

Release note: None.
@pav-kv pav-kv mentioned this issue Mar 30, 2023
Closed
@pav-kv pav-kv mentioned this issue Mar 30, 2023
Closed
@jbowens jbowens added this to Storage Jun 4, 2024
@jbowens jbowens moved this to Done in Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicktrav nicktrav

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Projects
Storage
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

roachtest: use local SSDs for disk-stall failover tests nicktrav/cockroach
2 participants
@nicktrav @cockroach-teamcity