Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: enable MVCC stats assertions more aggressively #86649

Open
erikgrinaker opened this issue Aug 23, 2022 · 2 comments
Open

roachtest: enable MVCC stats assertions more aggressively #86649

erikgrinaker opened this issue Aug 23, 2022 · 2 comments
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. O-qa T-storage Storage Team

Comments

@erikgrinaker
Copy link
Contributor

erikgrinaker commented Aug 23, 2022

We should assert MVCC stats in roachtests, by enabling the following environment variables:

  • COCKROACH_CONSISTENCY_AGGRESSIVE=true
  • COCKROACH_ENFORCE_CONSISTENT_STATS=true

This will perform synchronous consistency checks after certain operations, and crash the node if any MVCC stats are incorrect when compared to the real data.

This may surface latent stats issues, so we should be prepared to clean these up.

Jira issue: CRDB-18854

@blathers-crl
Copy link

blathers-crl bot commented Aug 23, 2022

cc @cockroachdb/replication

@tbg
Copy link
Member

tbg commented Aug 30, 2022

Whipped something up in #87106 but a manual run revealed that running the full checks tends to drive nodes into OOMs, so we'll need a few DB-side changes to make that work.

COCKROACH_ENFORCE_CONSISTENT_STATS should be in the cards, though.

@exalate-issue-sync exalate-issue-sync bot assigned erikgrinaker and unassigned tbg Oct 4, 2022
@exalate-issue-sync exalate-issue-sync bot added T-storage Storage Team and removed T-kv-replication labels Nov 30, 2022
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. O-qa T-storage Storage Team
Projects
Status: Backlog
Development

No branches or pull requests

2 participants