Race condition when creating HystrixThreadPool #340
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Symptoms
Different thread pool is referenced in
AbstractCommand
and a different one inHystrixThreadPoolMetrics
for the same key. This means monitoring always shows0
in active thread, etc. because that pool is never touched. This happens only when pool is first initialized from multiple threads. Pseudo code exposing bug:Code above shows correct pool statistics because second line safely initialized one thread pool. However if second line is commented, race condition causes different thread pool to be registered for statistics and different one for actual execution. I believe I can turn this into a test-case if necessary.
Steps to reproduce:
HystrixThreadPool.Factory.getInstance()
viaAbstractCommand
c-torthreadPools
static map is empty for both threads, thus they both hitthreadPools.putIfAbsent(key, new HystrixThreadPoolDefault(...));
HystrixThreadPoolDefault
are created eagerly, first one (arbitrary) registers itself inHystrixThreadPoolMetrics.metrics
insideHystrixThreadPoolMetrics#getInstance(...)
HystrixThreadPool.Factory.getInstance()
this:threadPools.putIfAbsent(key, ...)
is executed. However there is no guarantee thatthreadPools.putIfAbsent()
will put the same thread pool as the one inHystrixThreadPoolMetrics.metrics
Severity
This bug is not critical, but causes dashboard to show incorrect statistics.
Fix
My fix is straightforward: just put
synchronized
around lazy initialization. Previous code was more clever and probably faster - but broken. I understandsynchronized
was almost absent in this class, moreover we already deal with thread-safe collections. I considered another approach: instead of eagerly creating newHystrixThreadPoolDefault
inputIfAbsent()
, use another (new) implementation ofHystrixThreadPool
as a marker:But this seems hacky and hard to read. Also by introducing second implementation of
HystrixThreadPool
JVM can't make such aggressive optimizations (inlining) due to multiple implementations loaded. This is largely a premature optimization.