Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

teamcity: failed test: TestCleanupIntents #35064

Closed
cockroach-teamcity opened this issue Feb 19, 2019 · 2 comments · Fixed by #35085
Closed

teamcity: failed test: TestCleanupIntents #35064

cockroach-teamcity opened this issue Feb 19, 2019 · 2 comments · Fixed by #35085
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed on master (testrace): TestCleanupIntents/#2, TestCleanupIntents/#00, TestCleanupIntents/#1, TestCleanupIntents

You may want to check for open issues.

#1144299:

TestCleanupIntents/#00
--- FAIL: testrace/TestCleanupIntents/#00 (0.000s)
Test ended in panic.




TestCleanupIntents/#02
...git.luolix.top/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:187 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0002a2b40, 0x16a8f80, 0xc000034118, 0xc00003c330, 0x22, 0x0, 0x0, 0xc00046c170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x14e

goroutine 1140 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0005440d0, 0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:368 +0x3e3
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:187 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0002a2b40, 0x16a8f80, 0xc000034118, 0xc00003c2d0, 0x22, 0x0, 0x0, 0xc00046c0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x14e

goroutine 1143 [runnable]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000544270, 0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:368 +0x3e3
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:187 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0002a2b40, 0x16a8f80, 0xc000034118, 0xc00003c360, 0x22, 0x0, 0x0, 0xc00046c190)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x14e

goroutine 1116 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0000a2d00, 0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:368 +0x3e3
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:187 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0002a2b40, 0x16a8f80, 0xc000034118, 0xc00044e000, 0x22, 0x0, 0x0, 0xc0001e4070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x14e

goroutine 1139 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000544000, 0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:368 +0x3e3
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x16a8f80, 0xc000034118)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:187 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0002a2b40, 0x16a8f80, 0xc000034118, 0xc00003c2a0, 0x22, 0x0, 0x0, 0xc00046c0d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x14e



TestCleanupIntents
--- FAIL: test/TestCleanupIntents (0.040s)
Test ended in panic.




TestCleanupIntents/#01
--- FAIL: testrace/TestCleanupIntents/#01 (0.000s)
Test ended in panic.




TestCleanupIntents
--- FAIL: testrace/TestCleanupIntents (0.000s)
Test ended in panic.





Please assign, take a look and update the issue accordingly.

@cockroach-teamcity cockroach-teamcity added this to the 2.2 milestone Feb 19, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Feb 19, 2019
@nvanbenschoten
Copy link
Member

@ajwerner looks related to the new RequestBatcher. Mind taking a look?

@ajwerner
Copy link
Contributor

Bummer. Yes I’ll deal with it.

ajwerner added a commit to ajwerner/cockroach that referenced this issue Feb 20, 2019
The unit tests which failed assumed that all messages intents in a single call
to ResolveIntents could be queued within the batch timeout and idle window which
is by default rather short (10 and 5 ms respectively). The testing validation
logic is rather strict and assumes that the batch will fire due to size rather
than time constraints. The fix is to allow the test to increase these values to
make the test more robust to load. Before this change the flake was reproducible
running testrace within several hundred iterations. After the test it has not
failed after running over 20k test race iterations from two concurrently running
executions.

Fixes cockroachdb#35064.

Release note: None
ajwerner added a commit to ajwerner/cockroach that referenced this issue Feb 20, 2019
The unit tests which failed assumed that all messages intents in a single call
to ResolveIntents could be queued within the batch timeout and idle window which
is by default rather short (10 and 5 ms respectively). The testing validation
logic is rather strict and assumes that the batch will fire due to size rather
than time constraints. The fix is to allow the test to increase these values to
make the test more robust to load. Before this change the flake was reproducible
running testrace within several hundred iterations. Running testrace now seems
to not provoke any failures.

All that being said, I can see an argument that the testing logic should be
made less rigid and should accept that the batches may be split up.

Fixes cockroachdb#35064.

Release note: None
ajwerner added a commit to ajwerner/cockroach that referenced this issue Feb 20, 2019
The unit tests which failed assumed that all messages intents in a single call
to ResolveIntents could be queued within the batch timeout and idle window which
is by default rather short (10 and 5 ms respectively). The testing validation
logic is rather strict and assumes that the batch will fire due to size rather
than time constraints. The fix is to allow the test to increase these values to
make the test more robust to load. Before this change the flake was reproducible
running testrace within several hundred iterations. Running testrace now seems
to not provoke any failures.

All that being said, I can see an argument that the testing logic should be
made less rigid and should accept that the batches may be split up.

Fixes cockroachdb#35064.

Release note: None
craig bot pushed a commit that referenced this issue Feb 21, 2019
35085: intentresolver: fix testrace flake by extending timeouts r=ajwerner a=ajwerner

The unit tests which failed assumed that all messages intents in a single call
to ResolveIntents could be queued within the batch timeout and idle window which
is by default rather short (10 and 5 ms respectively). The testing validation
logic is rather strict and assumes that the batch will fire due to size rather
than time constraints. The fix is to allow the test to increase these values to
make the test more robust to load. Before this change the flake was reproducible
running testrace within several hundred iterations. After the test it has not
failed after running over 20k test race iterations from two concurrently running
executions.

Fixes #35064.

Release note: None

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
@craig craig bot closed this as completed in #35085 Feb 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants