roachprod: failures due to permission denied on /mnt/data1/cockroach #72635
Assignees
Labels
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Comments
tbg
added
the
C-bug
AlexTalks
added a commit
to AlexTalks/cockroach
that referenced
this issue
Nov 16, 2021
Fixes bugs introduced by cockroachdb#72553, which ported support for multiple stores from AWS to GCP. This change fixes the bugs that caused disks mounted to not be written to `/etc/fstab` on GCP instances, causing mounts to go missing on instance restart. Additionally, this change modifies the defaults on GCP and AWS to prefer creating multiple stores rather than creating a RAID 0 array, thus preserving the existing behavior used in roachtests such as `kv/multi-store-with-overload`. Fixes cockroachdb#72635 Release note: None
AlexTalks
added a commit
to AlexTalks/cockroach
that referenced
this issue
Nov 16, 2021
Fixes bugs introduced by cockroachdb#72553, which ported support for multiple stores from AWS to GCP. This change fixes the bugs that caused disks mounted to not be written to `/etc/fstab` on GCP instances, causing mounts to go missing on instance restart. Additionally, this change modifies roachtest to default to creating multiple stores on GCP and AWS in roachprod invocations rather than mounting a RAID 0 array, thus preserving the existing behavior used in roachtests such as `kv/multi-store-with-overload`. Fixes cockroachdb#72635 Release note: None
craig bot
pushed a commit
that referenced
this issue
Nov 17, 2021
71813: sql: add delete preserving index encoding, use for primary and second… r=rhu713 a=rhu713 …ary index As part of the transition for Bulk Ops to use only MVCC operations, the index backfiller must be rewritten. Part of the rewrite is to introduce a temporary index at the beginning of the backfill encoded with a new delete-preserving index encoding. This patch adds this encoding and uses the encoding for writes to indices configured with this encoding. Release note: None 72803: roachprod,roachtest: fix mounting on GCP and disable RAID0 in roachtests r=AlexTalks a=AlexTalks Fixes bugs introduced by #72553, which ported support for multiple stores from AWS to GCP. This change fixes the bugs that caused disks mounted to not be written to `/etc/fstab` on GCP instances, causing mounts to go missing on instance restart. Additionally, this change modifies roachtest to default to creating multiple stores on GCP in roachprod invocations rather than mounting a RAID 0 array, thus preserving the existing behavior used in roachtests such as `kv/multi-store-with-overload`. Fixes #72635 Release note: None 72817: sql/catalog/resolver: remove an allocation, add a benchmark r=ajwerner a=ajwerner We didn't need to allocate this name except on error paths. Make that lazy. The added benchmark also shows that we do horrifically badly when using user-defined schemas and `search_path`. Yay for benchmarks. ``` name old time/op new time/op delta ResolveExistingObject/CREATE_TABLE_foo_()foo-16 2.18µs ± 5% 2.03µs ± 2% -6.96% (p=0.008 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()sc.foo-16 1.59µs ± 4% 1.53µs ± 4% ~ (p=0.063 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()foo-16 60.5µs ± 4% 60.1µs ± 6% ~ (p=0.690 n=5+5) name old alloc/op new alloc/op delta ResolveExistingObject/CREATE_TABLE_foo_()foo-16 72.4B ± 1% 7.6B ±18% -89.50% (p=0.008 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()sc.foo-16 69.4B ± 1% 5.0B ± 0% -92.80% (p=0.008 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()foo-16 11.2kB ± 0% 11.1kB ± 0% -0.56% (p=0.008 n=5+5) name old allocs/op new allocs/op delta ResolveExistingObject/CREATE_TABLE_foo_()foo-16 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()sc.foo-16 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5) ResolveExistingObject/CREATE_SCHEMA_sc;CREATE_TABLE_sc.foo_()foo-16 75.0 ± 0% 74.0 ± 0% -1.33% (p=0.008 n=5+5) ``` Release note: None Co-authored-by: Rui Hu <rui@cockroachlabs.com> Co-authored-by: Alex Sarkesian <sarkesian@cockroachlabs.com> Co-authored-by: Andrew Werner <awerner32@gmail.com>
blathers-crl bot
pushed a commit
that referenced
this issue
Nov 17, 2021
Fixes bugs introduced by #72553, which ported support for multiple stores from AWS to GCP. This change fixes the bugs that caused disks mounted to not be written to `/etc/fstab` on GCP instances, causing mounts to go missing on instance restart. Additionally, this change modifies roachtest to default to creating multiple stores on GCP in roachprod invocations rather than mounting a RAID 0 array, thus preserving the existing behavior used in roachtests such as `kv/multi-store-with-overload`. Fixes #72635 Release note: None
This was referenced Nov 17, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem
See #72083 (comment) or any other issue linked to this thread, and roachdash
To Reproduce
Still reproducing, but we think it is related to restarting the roachprod vm and then trying to start CRDB again. Perhaps upon restarting, the /mnt/data1/cockroach permissions are not what they were on the initial boot.
@AlexTalks is taking a look.
Impact
Causing families of roachtests to fail predictably, see linked issues.
The text was updated successfully, but these errors were encountered: