Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use HdrHistogram for latency percentile calculation #779

Merged
merged 2 commits into from
Apr 29, 2015
Merged

Use HdrHistogram for latency percentile calculation #779

merged 2 commits into from
Apr 29, 2015

Conversation

mattrjacobs
Copy link
Contributor

Partially addresses #698

Matt Jacobs added 2 commits April 29, 2015 10:44
Added dependency on HdrHistogram 2.1.4
2779e89
Replaced internal data structure of HystrixRollingPercentile with Hdr…
e3ea425 
…Histogram

* Also fixed some incorrect assertions around how percentile was being calculated
@mattrjacobs
Copy link
Contributor Author

Performance numbers:

Benchmark (forceCircuitOpen) (isolationStrategy) BaselineScore BaselineError HDR Score HDR Error Units Diff
CommandExecutionPerfTest.baselineAsyncComputationObserve N/A N/A 330.703 6.169 326.357 5.608 ops/ms -1.31%
CommandExecutionPerfTest.baselineAsyncComputationObserve:·gc.alloc.rate.norm N/A N/A 1273.857 8.669 1270.978 6.75 B/op -0.23%
CommandExecutionPerfTest.baselineAsyncComputationObserve:·gc.count N/A N/A 376 367 counts -2.39%

CommandExecutionPerfTest.baselineAsyncCustomThreadPoolObserve N/A N/A 247.147 5.138 243.765 3.359 ops/ms -1.37%
CommandExecutionPerfTest.baselineAsyncCustomThreadPoolObserve:·gc.alloc.rate.norm N/A N/A 1843.866 433.512 1753.755 550.925 B/op -4.89%
CommandExecutionPerfTest.baselineAsyncCustomThreadPoolObserve:·gc.count N/A N/A 391 367 counts -6.14%

CommandExecutionPerfTest.baselineExecute N/A N/A 1438688.894 5143.102 1422630.068 8143.436 ops/ms -1.12%
CommandExecutionPerfTest.baselineExecute:·gc.alloc.rate.norm N/A N/A 0 0 B/op
CommandExecutionPerfTest.baselineExecute:·gc.count N/A N/A 0 0 counts

CommandExecutionPerfTest.baselineQueue N/A N/A 213.75 3.827 220.422 3.891 ops/ms 3.12%
CommandExecutionPerfTest.baselineQueue:·gc.alloc.rate.norm N/A N/A 106.799 4.783 106.042 4.962 B/op -0.71%
CommandExecutionPerfTest.baselineQueue:·gc.count N/A N/A 9 10 counts 11.11%

CommandExecutionPerfTest.baselineSyncObserve N/A N/A 12086.749 90.029 12005.763 92.88 ops/ms -0.67%
CommandExecutionPerfTest.baselineSyncObserve:·gc.alloc.rate.norm N/A N/A 801.61 5.318 803.208 4.974 B/op 0.20%
CommandExecutionPerfTest.baselineSyncObserve:·gc.count N/A N/A 658 649 counts -1.37%

CommandExecutionPerfTest.hystrixExecute TRUE THREAD 756.134 9.987 750.421 14.344 ops/ms -0.76%
CommandExecutionPerfTest.hystrixExecute:·gc.alloc.rate.norm TRUE THREAD 3895.17 10.443 3895.156 10.512 B/op 0.00%
CommandExecutionPerfTest.hystrixExecute:·gc.count TRUE THREAD 597 499 counts -16.42%

CommandExecutionPerfTest.hystrixExecute TRUE SEMAPHORE 763.237 13.021 761.161 11.188 ops/ms -0.27%
CommandExecutionPerfTest.hystrixExecute:·gc.alloc.rate.norm TRUE SEMAPHORE 3899.973 10.936 3909.503 13.57 B/op 0.24%
CommandExecutionPerfTest.hystrixExecute:·gc.count TRUE SEMAPHORE 532 561 counts 5.45%

CommandExecutionPerfTest.hystrixExecute FALSE THREAD 105.68 6.129 104.066 7.903 ops/ms -1.53%
CommandExecutionPerfTest.hystrixExecute:·gc.alloc.rate.norm FALSE THREAD 5112.851 13.55 5149.32 24.735 B/op 0.71%
CommandExecutionPerfTest.hystrixExecute:·gc.count FALSE THREAD 20 21 counts 5.00%

CommandExecutionPerfTest.hystrixExecute FALSE SEMAPHORE 310.77 8.657 312.867 6.848 ops/ms 0.67%
CommandExecutionPerfTest.hystrixExecute:·gc.alloc.rate.norm FALSE SEMAPHORE 4038.634 6.86 4079.235 17.498 B/op 1.01%
CommandExecutionPerfTest.hystrixExecute:·gc.count FALSE SEMAPHORE 29 30 counts 3.45%

CommandExecutionPerfTest.hystrixObserve TRUE THREAD 694.001 12.909 697.921 8.388 ops/ms 0.56%
CommandExecutionPerfTest.hystrixObserve:·gc.alloc.rate.norm TRUE THREAD 4690.376 13.54 4701.676 17.512 B/op 0.24%
CommandExecutionPerfTest.hystrixObserve:·gc.count TRUE THREAD 564 531 counts -5.85%

CommandExecutionPerfTest.hystrixObserve TRUE SEMAPHORE 695.901 9.682 701.939 12.944 ops/ms 0.87%
CommandExecutionPerfTest.hystrixObserve:·gc.alloc.rate.norm TRUE SEMAPHORE 4690.431 4.884 4707.174 12.204 B/op 0.36%
CommandExecutionPerfTest.hystrixObserve:·gc.count TRUE SEMAPHORE 576 503 counts -12.67%

CommandExecutionPerfTest.hystrixObserve FALSE THREAD 109.284 2.681 107.554 2.879 ops/ms -1.58%
CommandExecutionPerfTest.hystrixObserve:·gc.alloc.rate.norm FALSE THREAD 5922.235 18.193 5946.152 21.623 B/op 0.40%
CommandExecutionPerfTest.hystrixObserve:·gc.count FALSE THREAD 24 20 counts -16.67%

CommandExecutionPerfTest.hystrixObserve FALSE SEMAPHORE 278.146 3.883 280.664 4.046 ops/ms 0.91%
CommandExecutionPerfTest.hystrixObserve:·gc.alloc.rate.norm FALSE SEMAPHORE 4844.64 12.237 4873.286 27.067 B/op 0.59%
CommandExecutionPerfTest.hystrixObserve:·gc.count FALSE SEMAPHORE 30 32 counts 6.67%

CommandExecutionPerfTest.hystrixQueue TRUE THREAD 752.702 10.624 780.142 13.745 ops/ms 3.65%
CommandExecutionPerfTest.hystrixQueue:·gc.alloc.rate.norm TRUE THREAD 3892.722 6.507 3899.943 13.146 B/op 0.19%
CommandExecutionPerfTest.hystrixQueue:·gc.count TRUE THREAD 597 512 counts -14.24%

CommandExecutionPerfTest.hystrixQueue TRUE SEMAPHORE 750.394 11.706 767.212 12.961 ops/ms 2.24%
CommandExecutionPerfTest.hystrixQueue:·gc.alloc.rate.norm TRUE SEMAPHORE 3897.52 10.824 3899.941 13.139 B/op 0.06%
CommandExecutionPerfTest.hystrixQueue:·gc.count TRUE SEMAPHORE 595 523 counts -12.10%

CommandExecutionPerfTest.hystrixQueue FALSE THREAD 109.66 1.806 103.133 8.17 ops/ms -5.95%
CommandExecutionPerfTest.hystrixQueue:·gc.alloc.rate.norm FALSE THREAD 5117.543 13.825 5134.532 25.749 B/op 0.33%
CommandExecutionPerfTest.hystrixQueue:·gc.count FALSE THREAD 20 20 counts 0.00%

CommandExecutionPerfTest.hystrixQueue FALSE SEMAPHORE 309.703 6.823 311.532 8.319 ops/ms 0.59%
CommandExecutionPerfTest.hystrixQueue:·gc.alloc.rate.norm FALSE SEMAPHORE 4051.504 6.497 4070.294 18.109 B/op 0.46%
CommandExecutionPerfTest.hystrixQueue:·gc.count FALSE SEMAPHORE 30 30 counts 0.00%

CommandExecutionPerfTest.hystrixToObservable TRUE THREAD 737.15 14.007 740.795 12.562 ops/ms 0.49%
CommandExecutionPerfTest.hystrixToObservable:·gc.alloc.rate.norm TRUE THREAD 4156.812 10.583 4158.418 13.024 B/op 0.04%
CommandExecutionPerfTest.hystrixToObservable:·gc.count TRUE THREAD 575 513 counts -10.78%

CommandExecutionPerfTest.hystrixToObservable TRUE SEMAPHORE 739.351 12.943 735.285 10.498 ops/ms -0.55%
CommandExecutionPerfTest.hystrixToObservable:·gc.alloc.rate.norm TRUE SEMAPHORE 4149.615 13.57 4151.243 14.648 B/op 0.04%
CommandExecutionPerfTest.hystrixToObservable:·gc.count TRUE SEMAPHORE 510 471 counts -7.65%

CommandExecutionPerfTest.hystrixToObservable FALSE THREAD 113.566 1.413 111.999 2.029 ops/ms -1.38%
CommandExecutionPerfTest.hystrixToObservable:·gc.alloc.rate.norm FALSE THREAD 5341.183 16.168 5363.177 16.357 B/op 0.41%
CommandExecutionPerfTest.hystrixToObservable:·gc.count FALSE THREAD 20 20 counts 0.00%

CommandExecutionPerfTest.hystrixToObservable FALSE SEMAPHORE 301.764 5.037 301.471 5.837 ops/ms -0.10%
CommandExecutionPerfTest.hystrixToObservable:·gc.alloc.rate.norm FALSE SEMAPHORE 4274.184 8.048 4306.701 17.878 B/op 0.76%
CommandExecutionPerfTest.hystrixToObservable:·gc.count FALSE SEMAPHORE 30 30 counts 0.00%

MultiThreadedMetricsTest.evenSplit:evenSplitOfWritesAndReadsCommandExecution N/A THREAD 88.055 3.119 81.455 3.558 ops/ms -7.50%
MultiThreadedMetricsTest.evenSplit:evenSplitOfWritesAndReadsReadMetrics N/A THREAD 27825.522 1272.008 28046.7 1469.442 ops/ms 0.79%
MultiThreadedMetricsTest.evenSplit:·gc.alloc.rate.norm N/A THREAD 503.473 8.865 498.219 10.574 B/op -1.04%
MultiThreadedMetricsTest.evenSplit:·gc.count N/A THREAD 47 47 counts 0.00%

MultiThreadedMetricsTest.evenSplit:evenSplitOfWritesAndReadsCommandExecution N/A SEMAPHORE 229.168 11.314 223.642 16.355 ops/ms -2.41%
MultiThreadedMetricsTest.evenSplit:evenSplitOfWritesAndReadsReadMetrics N/A SEMAPHORE 37350.112 787.822 37024.293 976.228 ops/ms -0.87%
MultiThreadedMetricsTest.evenSplit:·gc.alloc.rate.norm N/A SEMAPHORE 519.382 7.935 516.507 11.445 B/op -0.55%
MultiThreadedMetricsTest.evenSplit:·gc.count N/A SEMAPHORE 87 86 counts -1.15%

MultiThreadedMetricsTest.readHeavy:readHeavyCommandExecution N/A THREAD 25.409 1.039 25.435 1.056 ops/ms 0.10%
MultiThreadedMetricsTest.readHeavy:readHeavyReadMetrics N/A THREAD 60430.379 990.994 59941.233 779.843 ops/ms -0.81%
MultiThreadedMetricsTest.readHeavy:·gc.alloc.rate.norm N/A THREAD 404.312 0.584 404.602 0.606 B/op 0.07%
MultiThreadedMetricsTest.readHeavy:·gc.count N/A THREAD 250 237 counts -5.20%

MultiThreadedMetricsTest.readHeavy:readHeavyCommandExecution N/A SEMAPHORE 107.26 2.433 105.018 3.124 ops/ms -2.09%
MultiThreadedMetricsTest.readHeavy:readHeavyReadMetrics N/A SEMAPHORE 60305.59 930.555 59679.441 878.39 ops/ms -1.04%
MultiThreadedMetricsTest.readHeavy:·gc.alloc.rate.norm N/A SEMAPHORE 430.933 0.971 431.488 1.267 B/op 0.13%
MultiThreadedMetricsTest.readHeavy:·gc.count N/A SEMAPHORE 104 103 counts -0.96%

MultiThreadedMetricsTest.writeHeavy:writeHeavyCommandExecution N/A THREAD 96.301 6.428 97.505 6.774 ops/ms 1.25%
MultiThreadedMetricsTest.writeHeavy:writeHeavyReadMetrics N/A THREAD 9613.169 681.297 8877.695 1125.974 ops/ms -7.65%
MultiThreadedMetricsTest.writeHeavy:·gc.alloc.rate.norm N/A THREAD 880.629 62.747 956.569 95.963 B/op 8.62%
MultiThreadedMetricsTest.writeHeavy:·gc.count N/A THREAD 26 27 counts 3.85%

MultiThreadedMetricsTest.writeHeavy:writeHeavyCommandExecution N/A SEMAPHORE 275.957 12.397 252.638 14.648 ops/ms -8.45%
MultiThreadedMetricsTest.writeHeavy:writeHeavyReadMetrics N/A SEMAPHORE 16725.509 545.664 16914.949 668.144 ops/ms 1.13%
MultiThreadedMetricsTest.writeHeavy:·gc.alloc.rate.norm N/A SEMAPHORE 797.9 21.139 756.466 22.334 B/op -5.19%
MultiThreadedMetricsTest.writeHeavy:·gc.count N/A SEMAPHORE 49 49 counts 0.00%

@mattrjacobs
Copy link
Contributor Author

I will reformat the above comment when I have more time, but the takeaway is that using HdrHistogram for percentiles gained correctness and didn't appear to affect performance negatively

@cloudbees-pull-request-builder
Copy link

NetflixOSS » Hystrix » Hystrix-pull-requests #112 SUCCESS
This pull request looks good

mattrjacobs added a commit that referenced this pull request Apr 29, 2015
@mattrjacobs
Merge pull request #779 from mattrjacobs/rebased-hdr-histogram
89c2569 
Use HdrHistogram for latency percentile calculation
@mattrjacobs mattrjacobs merged commit 89c2569 into Netflix:master Apr 29, 2015
@mattrjacobs mattrjacobs deleted the rebased-hdr-histogram branch April 29, 2015 20:32
This was referenced Apr 29, 2015
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mattrjacobs @cloudbees-pull-request-builder