roachtest: failover/chaos/read-only failed

[cockroach-teamcity]

roachtest.failover/chaos/read-only failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

(cluster.go:2247).Run: output in run_094944.124037951_n1-10_echo-0-sudo-blockdev: echo "0 $(sudo blockdev --getsz /dev/nvme1n1) linear /dev/nvme1n1 0" | sudo dmsetup create data1 returned: COMMAND_PROBLEM: exit status 1
(cluster.go:2247).Run: cluster.RunE: context canceled
(cluster.go:2247).Run: cluster.RunE: context canceled
(cluster.go:2247).Run: cluster.RunE: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-29338

[erikgrinaker]

device-mapper: reload ioctl on data1  failed: Device or resource busy

Unclear why, since we successfully unmounted the device just prior.

[erikgrinaker]

I notice that all of these failures are on AWS. I thought we only ran these on GCE -- since the failure modes are identical across clouds (they're synthetic), there is little benefit in doubling the costs by running on two clouds. I'll submit a PR to only run the nightly in GCE.

We do run separate disk-stall tests on both GCE and AWS though, which fail as well (see e.g. #106009). So we'll need to address the root problem here regardless.

[erikgrinaker]

The nodes use two NVMe drives in a software RAID:

nvme2n1      259:1    0 220.7G  0 disk  
└─md0          9:0    0 720.5G  0 raid0 
nvme1n1      259:4    0   500G  0 disk  
└─md0          9:0    0 720.5G  0 raid0 

Will see if we can disable that somehow.

[cockroach-teamcity]

roachtest.failover/chaos/read-only failed with artifacts on release-23.1 @ e12e85479312972b551677203849d29aeb38ad5f:

(cluster.go:2247).Run: output in run_094718.359838974_n1-10_echo-0-sudo-blockdev: echo "0 $(sudo blockdev --getsz /dev/nvme1n1) linear /dev/nvme1n1 0" | sudo dmsetup create data1 returned: COMMAND_PROBLEM: exit status 1
(cluster.go:2247).Run: cluster.RunE: context canceled
(cluster.go:2247).Run: cluster.RunE: context canceled
(cluster.go:2247).Run: cluster.RunE: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[erikgrinaker]

These tests actually don't use local SSDs, but EBS volumes are exposed as NVMe devices.

ROACHTEST_localSSD=false , ROACHTEST_ssd=0

These tests were switched to use PDs rather than local SSDs in #99747 and #99963.

Don't know yet why we're seeing multiple devices and RAIDing here.

[erikgrinaker]

The problem seems to be that we're creating c6id.xlarge instances even though we haven't requested local SSDs, so we get a local SSD and an EBS volume that's then RAIDed together.

[erikgrinaker]

FWIW, these started failing because of a recent TeamCity change that erroneously enabled all roachtests on AWS. I suspect these disk stall tests have been broken on AWS for some time, probably since #99747.

[erikgrinaker]

I wrote up #106058, handing this over to test-eng. The nightly failures will stop when we correctly omit these tests on AWS.

[blathers-crl]

cc @cockroachdb/test-eng

[erikgrinaker]

The TeamCity param which ran these tests on AWS has been reverted. The roachtest bug that breaks this test on AWS is tracked in #106058.