Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enable running non-perf roachtests with runtime assertions (crdb_test build tag) #86678

Closed
srosenberg opened this issue Aug 23, 2022 · 3 comments · Fixed by #111949
Closed

enable running non-perf roachtests with runtime assertions (crdb_test build tag) #86678

srosenberg opened this issue Aug 23, 2022 · 3 comments · Fixed by #111949
Labels
A-testing Testing tools and infrastructure branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-testeng TestEng Team

Comments

@srosenberg
Copy link
Member

srosenberg commented Aug 23, 2022 •
edited
Loading

Recent PR [1] added the ability to run roachtests with runtime assertions enabled. TC will now produce cockroach-short-ea. This binary can be staged (instead of cockroach) with the default probability of .5 by invoking maybeUseBuildWithEnabledAssertions.

Let's try to enable this for all non-perf roachtests and conditionally opt out those if the assertions result in longer timeouts.

[1] #86625

Jira issue: CRDB-18869
@srosenberg srosenberg added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-testing Testing tools and infrastructure T-testeng TestEng Team labels Aug 23, 2022
@blathers-crl
Copy link

blathers-crl bot commented Aug 23, 2022

cc @cockroachdb/test-eng

This was referenced Jan 10, 2023
Merged
Open
@srosenberg srosenberg mentioned this issue Jan 13, 2023
Open
@renatolabs renatolabs mentioned this issue Mar 15, 2023
Closed
renatolabs added a commit to renatolabs/cockroach that referenced this issue Mar 15, 2023
@renatolabs
roachtest: randomly run with runtime assertions by default
1cb357f 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions enabled by default. Performance tests (i.e., tests
that export to roachperf) will be able to continue using the standard
binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions enabled, most often due to timeouts.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
renatolabs added a commit to renatolabs/cockroach that referenced this issue Mar 17, 2023
@renatolabs
roachtest: randomly run with runtime assertions by default
bc05547 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions enabled by default. Performance tests (i.e., tests
that export to roachperf) will be able to continue using the standard
binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions enabled, most often due to timeouts.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
renatolabs added a commit to renatolabs/cockroach that referenced this issue Mar 22, 2023
@renatolabs
roachtest: randomly run with runtime assertions by default
e42fcc9 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions enabled by default. Performance tests (i.e., tests
that export to roachperf) will be able to continue using the standard
binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions enabled, most often due to timeouts.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
@DarrylWong DarrylWong mentioned this issue Oct 6, 2023
Merged
@erikgrinaker erikgrinaker mentioned this issue Oct 13, 2023
Closed
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 17, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
325f77c 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions enabled by default. Performance tests (indicated by
the benchmark TestSpec) will continue using the standard binary, without
runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions enabled, most often due to timeouts.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 26, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
05598c3 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 30, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
540a18f 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 31, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
3c00c4d 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 31, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
10f3709 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 31, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
c58ba1a 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Oct 31, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
339f002 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Nov 1, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
68e5974 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
craig bot pushed a commit that referenced this issue Nov 2, 2023
@renatolabs @DarrylWong @itsbilal
Merge #111949 #113693
722f902 
111949: roachtest: randomly run with runtime assertions by default r=renatolabs,mikeCRL,rickystewart,herkolategan a=DarrylWong

This changes the semantics of `t.Cockroach()` to use a binary with runtime assertions enabled by default. Performance tests (indicated by the benchmark TestSpec) will be able to continue using the standard binary, without runtime assertions or metamorphic constants, as usual.

This commit also opts-out other tests that cannot easily be run with runtime assertions or metamorphic constants enabled, most often due to timeouts or metamorphic constant incompatibility.

Resolves #86678.
Informs #94986.

Epic: none

Release note: None

113693: go.mod: bump Pebble to 844f0582c2eb r=RaduBerinde a=itsbilal

```
844f0582 tool: fix tool to tolerate T in the log context
58cbdb46 internal/dsl: new package
481da041 db: add KeyStatistics.LatestKindsCount
86ec1c41 db: move size check for remote files to table stats job
424019fb db: guard against concurrent batch writes during Commit
5f131c12 db: default to zeroing fields in Batch.Reset
```

Release note: None.

Epic: none

Co-authored-by: Renato Costa <renato@cockroachlabs.com>
Co-authored-by: DarrylWong <darryl@cockroachlabs.com>
Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
@craig craig bot closed this as completed in 581e671 Nov 2, 2023
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Nov 3, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
4fa2af1 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
@DarrylWong DarrylWong mentioned this issue Nov 3, 2023
Merged
DarrylWong pushed a commit to DarrylWong/fork that referenced this issue Nov 6, 2023
@renatolabs @DarrylWong
roachtest: randomly run with runtime assertions by default
a82ef08 
This changes the semantics of `t.Cockroach()` to use a binary with
runtime assertions and metamorphic constants enabled by default.
Performance tests (indicated by the benchmark TestSpec) will continue
using the standard binary, without runtime assertions, as usual.

This commit also opts-out other tests that cannot easily be run with
runtime assertions or metamorphic constants enabled, most often due to
timeouts and metamorphic constant incompatibility.

Resolves cockroachdb#86678.
Informs cockroachdb#94986.

Epic: none

Release note: None
@DarrylWong DarrylWong mentioned this issue Nov 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-testing Testing tools and infrastructure branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-testeng TestEng Team
Projects
No open projects
Test Engineering
Status: Done
Milestone
No milestone
2 participants
@erikgrinaker @srosenberg