Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What's the purpose of setting shard_count in java_test? #2113

Closed
GinFungYJF opened this issue Nov 21, 2016 · 1 comment
Closed

What's the purpose of setting shard_count in java_test? #2113

GinFungYJF opened this issue Nov 21, 2016 · 1 comment

Comments

@GinFungYJF
Copy link
Contributor

GinFungYJF commented Nov 21, 2016

I want to make "bazel test" run the methods in a single java class concurrently in order to speed up the test. And then, I set the shard_count=n(n is the total number of methods in the java class) in java_test rule. But I found that setting shard_count takes more time than no setting shard_count. So, is the option shard_count aimed at speeding up the test? If not, how could I make "bazel test" run the methods in a single java class concurrently? What's more, my test framework is spring and the Test Runner is JUnit4.

Environment info

  • Operating System:
    ubuntu 14.04

  • Bazel version (output of bazel info release):
    0.4.0

@ulfjack
Copy link
Contributor

ulfjack commented Dec 1, 2016

Yes, shard_count is aimed at speeding up the test. The idea is indeed that Bazel runs multiple instances with each only running a subset of the tests. That said, if the tests are all very quick-running, then the overhead of starting up multiple Jvms can be larger than the benefits of sharding. Also, if you're running it on a single machine, you're limited by the parallelism you can get on that machine, especially if the tests are internally using multiple threads. Even for single-threaded tests, setting shard_count higher than the number of cores on your machine is (very) unlikely to be a win. At Google, we distribute tests to multiple machines, which allows us to run with larger number of shards.

All that said, if you see no reduction in per-shard times when comparing one vs. multiple shards, then that's an indication that there's still a bug in Bazel or the Java test runner wrt. the sharding protocol. Can you check the logs from the shards to see if each shard is actually only running a single test method?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants