Skip to content
This repository has been archived by the owner on Sep 26, 2023. It is now read-only.

fix: fix flaky tests and non blocking semaphore #1365

Merged
merged 3 commits into from
May 4, 2021

Conversation

mutianf
Copy link
Contributor

@mutianf mutianf commented May 4, 2021

Fix for #1359

In NonBlockingSemaphore, updating both limit and avaliablePermits in increase and decrease limits may cause problems. If the limit is lowered but avaliablePermits is not updated yet, and release is called, this statement in release: if (availablePermits.compareAndSet(old, Math.min(old + permits, limit.get()))) may update the availablePermits incorrectly.

Since we're already tracking the limit in the semaphore class, instead of tracking what's available, we could track how many permits are acquired. So When we're adjusting the limit, we won't need to update both variables and should avoid this problem.

After the change the test succeeded in all 100 runs:

//gax:com.google.api.gax.batching.FlowControllerTest                     PASSED in 1.7s
  Stats over 100 runs: max = 1.7s, min = 0.9s, avg = 1.2s, dev = 0.1s

@mutianf mutianf requested review from a team as code owners May 4, 2021 18:07
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label May 4, 2021
Copy link
Contributor

@vam-google vam-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but can you please confirm that it is still green on 1,000 and maybe 10,000 runs?

@mutianf
Copy link
Contributor Author

mutianf commented May 4, 2021

10000 runs with flag

--test_filter=com.google.api.gax.batching.FlowControllerTest#
--runs_per_test=10000
--cache_test_results=no
--sandbox_tmpfs_path=/tmp

all passed

INFO: 10001 processes: 1 internal, 10000 linux-sandbox.
INFO: Build completed successfully, 10001 total actions
//gax:com.google.api.gax.batching.FlowControllerTest                     PASSED in 2.4s
  Stats over 10000 runs: max = 2.4s, min = 0.9s, avg = 1.3s, dev = 0.2s

@vam-google vam-google merged commit fc8e520 into googleapis:master May 4, 2021
@mutianf mutianf deleted the fix_flaky_test branch June 4, 2021 14:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants