Compare Subcommand Only Reports P100 for Latency and Service Time

[IanHoang]

When running OSB with compare subcommand, users only have P100 latency and service time displayed in the results. Support was added a while back for percentiles to be passed through but there are no CLI options for users to pass in percentiles. Because no custom or default percentiles are passed in, the ComparisonResultsPublisher only uses the percentile of 100 in this line.

We need to add support for users to pass in custom percentiles for latency and service time. When none is given, a default list is provided.

Example:

Comparing baseline
  TestExecution ID: 2aae2c19-68f7-4d3e-bd42-860d0b344df7
  TestExecution timestamp: 2024-03-28 18:28:31
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: external
  User tags: jenkins-build-id=20835, security-enabled=false
 
with contender
  TestExecution ID: a57cc647-27fc-4cbd-ac53-55aadc6843ad
  TestExecution timestamp: 2024-02-27 21:46:57
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: external
  User tags: jenkins-build-id=18134, security-enabled=false
 
------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
 
|                                                        Metric |                                       Task |    Baseline |   Contender |     Diff |   Unit |
|--------------------------------------------------------------:|-------------------------------------------:|------------:|------------:|---------:|-------:|
|                    Cumulative indexing time of primary shards |                                            |     1.80267 |     1.66932 | -0.13335 |    min |
|             Min cumulative indexing time across primary shard |                                            |           0 |           0 |        0 |    min |
|          Median cumulative indexing time across primary shard |                                            |    0.589417 | 1.66667e-05 |  -0.5894 |    min |
|             Max cumulative indexing time across primary shard |                                            |    0.611133 |      1.6693 |  1.05817 |    min |
|           Cumulative indexing throttle time of primary shards |                                            |           0 |           0 |        0 |    min |
|    Min cumulative indexing throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
| Median cumulative indexing throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
|    Max cumulative indexing throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
|                       Cumulative merge time of primary shards |                                            |    0.111633 |     0.11115 | -0.00048 |    min |
|                      Cumulative merge count of primary shards |                                            |           7 |           9 |        2 |        |
|                Min cumulative merge time across primary shard |                                            |           0 |           0 |        0 |    min |
|             Median cumulative merge time across primary shard |                                            |   0.0282333 |           0 | -0.02823 |    min |
|                Max cumulative merge time across primary shard |                                            |      0.0459 |     0.11115 |  0.06525 |    min |
|              Cumulative merge throttle time of primary shards |                                            |           0 |           0 |        0 |    min |
|       Min cumulative merge throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
|    Median cumulative merge throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
|       Max cumulative merge throttle time across primary shard |                                            |           0 |           0 |        0 |    min |
|                     Cumulative refresh time of primary shards |                                            |    0.149167 |   0.0964167 | -0.05275 |    min |
|                    Cumulative refresh count of primary shards |                                            |          50 |          28 |      -22 |        |
|              Min cumulative refresh time across primary shard |                                            |           0 |           0 |        0 |    min |
|           Median cumulative refresh time across primary shard |                                            |   0.0414833 |      0.0001 | -0.04138 |    min |
|              Max cumulative refresh time across primary shard |                                            |      0.0625 |   0.0963167 |  0.03382 |    min |
|                       Cumulative flush time of primary shards |                                            | 0.000166667 | 0.000183333 |    2e-05 |    min |
|                      Cumulative flush count of primary shards |                                            |           5 |           3 |       -2 |        |
|                Min cumulative flush time across primary shard |                                            |           0 |           0 |        0 |    min |
|             Median cumulative flush time across primary shard |                                            |           0 |           0 |        0 |    min |
|                Max cumulative flush time across primary shard |                                            | 0.000166667 | 0.000183333 |    2e-05 |    min |
|                                       Total Young Gen GC time |                                            |       2.119 |       1.851 |   -0.268 |      s |
|                                      Total Young Gen GC count |                                            |         306 |         304 |       -2 |        |
|                                         Total Old Gen GC time |                                            |           0 |           0 |        0 |      s |
|                                        Total Old Gen GC count |                                            |           0 |           0 |        0 |        |
|                                                    Store size |                                            |    0.118504 |    0.119278 |  0.00077 |     GB |
|                                                 Translog size |                                            | 2.56114e-07 | 1.53668e-07 |       -0 |     GB |
|                                        Heap used for segments |                                            |           0 |           0 |        0 |     MB |
|                                      Heap used for doc values |                                            |           0 |           0 |        0 |     MB |
|                                           Heap used for terms |                                            |           0 |           0 |        0 |     MB |
|                                           Heap used for norms |                                            |           0 |           0 |        0 |     MB |
|                                          Heap used for points |                                            |           0 |           0 |        0 |     MB |
|                                   Heap used for stored fields |                                            |           0 |           0 |        0 |     MB |
|                                                 Segment count |                                            |          30 |          13 |      -17 |        |
|                                                Min Throughput |                                      index |       89357 |     82997.7 | -6359.37 | docs/s |
|                                               Mean Throughput |                                      index |      102119 |       94966 | -7152.56 | docs/s |
|                                             Median Throughput |                                      index |      104656 |     94934.4 |  -9721.5 | docs/s |
|                                                Max Throughput |                                      index |      110741 |      105688 | -5052.69 | docs/s |
|                                      100th percentile latency |                                      index |      600.84 |     623.757 |  22.9174 |     ms |
|                                 100th percentile service time |                                      index |      600.84 |     623.757 |  22.9174 |     ms |
|                                                    error rate |                                      index |           0 |           0 |        0 |      % |
|                                                Min Throughput |                   wait-until-merges-finish |     74.9532 |     99.4052 |  24.4521 |  ops/s |
|                                               Mean Throughput |                   wait-until-merges-finish |     74.9532 |     99.4052 |  24.4521 |  ops/s |
|                                             Median Throughput |                   wait-until-merges-finish |     74.9532 |     99.4052 |  24.4521 |  ops/s |
|                                                Max Throughput |                   wait-until-merges-finish |     74.9532 |     99.4052 |  24.4521 |  ops/s |
|                                      100th percentile latency |                   wait-until-merges-finish |     12.8589 |     9.60831 | -3.25064 |     ms |
|                                 100th percentile service time |                   wait-until-merges-finish |     12.8589 |     9.60831 | -3.25064 |     ms |
|                                                    error rate |                   wait-until-merges-finish |           0 |           0 |        0 |      % |
|                                                Min Throughput |     percolator_with_content_president_bush |     43.2868 |     37.2193 | -6.06751 |  ops/s |
|                                               Mean Throughput |     percolator_with_content_president_bush |     45.5868 |     39.3985 | -6.18827 |  ops/s |
|                                             Median Throughput |     percolator_with_content_president_bush |     45.5868 |     39.3985 | -6.18827 |  ops/s |
|                                                Max Throughput |     percolator_with_content_president_bush |     47.8868 |     41.5777 | -6.30903 |  ops/s |
|                                      100th percentile latency |     percolator_with_content_president_bush |      459.37 |     599.534 |  140.163 |     ms |
|                                 100th percentile service time |     percolator_with_content_president_bush |     14.7142 |     25.7474 |  11.0333 |     ms |
|                                                    error rate |     percolator_with_content_president_bush |           0 |           0 |        0 |      % |
|                                                Min Throughput |     percolator_with_content_saddam_hussein |     50.1498 |     50.1305 | -0.01934 |  ops/s |
|                                               Mean Throughput |     percolator_with_content_saddam_hussein |     50.1834 |     50.1565 | -0.02688 |  ops/s |
|                                             Median Throughput |     percolator_with_content_saddam_hussein |     50.1834 |     50.1565 | -0.02688 |  ops/s |
|                                                Max Throughput |     percolator_with_content_saddam_hussein |      50.217 |     50.1826 | -0.03442 |  ops/s |
|                                      100th percentile latency |     percolator_with_content_saddam_hussein |     7.61227 |     5.70896 | -1.90331 |     ms |
|                                 100th percentile service time |     percolator_with_content_saddam_hussein |     6.60175 |     5.08692 | -1.51484 |     ms |
|                                                    error rate |     percolator_with_content_saddam_hussein |           0 |           0 |        0 |      % |
|                                                Min Throughput |  percolator_with_content_hurricane_katrina |     50.0474 |     49.9546 | -0.09282 |  ops/s |
|                                               Mean Throughput |  percolator_with_content_hurricane_katrina |     50.0479 |     49.9573 | -0.09056 |  ops/s |
|                                             Median Throughput |  percolator_with_content_hurricane_katrina |     50.0479 |     49.9573 | -0.09056 |  ops/s |
|                                                Max Throughput |  percolator_with_content_hurricane_katrina |     50.0484 |       49.96 | -0.08831 |  ops/s |
|                                      100th percentile latency |  percolator_with_content_hurricane_katrina |     12.3883 |     21.3538 |  8.96552 |     ms |
|                                 100th percentile service time |  percolator_with_content_hurricane_katrina |     11.2262 |      20.558 |  9.33177 |     ms |
|                                                    error rate |  percolator_with_content_hurricane_katrina |           0 |           0 |        0 |      % |
|                                                Min Throughput |             percolator_with_content_google |     22.5889 |      10.428 | -12.1609 |  ops/s |
|                                               Mean Throughput |             percolator_with_content_google |      22.684 |     10.4677 | -12.2163 |  ops/s |
|                                             Median Throughput |             percolator_with_content_google |     22.6804 |     10.4697 | -12.2107 |  ops/s |
|                                                Max Throughput |             percolator_with_content_google |      22.782 |     10.5058 | -12.2762 |  ops/s |
|                                      100th percentile latency |             percolator_with_content_google |     1417.56 |     11657.3 |  10239.7 |     ms |
|                                 100th percentile service time |             percolator_with_content_google |        93.2 |     125.075 |  31.8753 |     ms |
|                                                    error rate |             percolator_with_content_google |           0 |           0 |        0 |      % |
|                                                Min Throughput |    percolator_no_score_with_content_google |     99.4634 |     99.6988 |  0.23539 |  ops/s |
|                                               Mean Throughput |    percolator_no_score_with_content_google |     99.4634 |     99.6988 |  0.23539 |  ops/s |
|                                             Median Throughput |    percolator_no_score_with_content_google |     99.4634 |     99.6988 |  0.23539 |  ops/s |
|                                                Max Throughput |    percolator_no_score_with_content_google |     99.4634 |     99.6988 |  0.23539 |  ops/s |
|                                      100th percentile latency |    percolator_no_score_with_content_google |     6.49906 |     5.81908 | -0.67998 |     ms |
|                                 100th percentile service time |    percolator_no_score_with_content_google |      5.6017 |     5.42238 | -0.17932 |     ms |
|                                                    error rate |    percolator_no_score_with_content_google |           0 |           0 |        0 |      % |
|                                                Min Throughput |               percolator_with_highlighting |      49.469 |     49.5002 |  0.03119 |  ops/s |
|                                               Mean Throughput |               percolator_with_highlighting |     49.5304 |     49.5641 |  0.03369 |  ops/s |
|                                             Median Throughput |               percolator_with_highlighting |     49.5304 |     49.5641 |  0.03369 |  ops/s |
|                                                Max Throughput |               percolator_with_highlighting |     49.5918 |      49.628 |   0.0362 |  ops/s |
|                                      100th percentile latency |               percolator_with_highlighting |     9.03186 |     9.93936 |  0.90749 |     ms |
|                                 100th percentile service time |               percolator_with_highlighting |     7.77886 |     8.28948 |  0.51062 |     ms |
|                                                    error rate |               percolator_with_highlighting |           0 |           0 |        0 |      % |
|                                                Min Throughput |          percolator_with_content_ignore_me |   0.0836749 |     0.05905 | -0.02462 |  ops/s |
|                                               Mean Throughput |          percolator_with_content_ignore_me |    0.084229 |   0.0591155 | -0.02511 |  ops/s |
|                                             Median Throughput |          percolator_with_content_ignore_me |   0.0839568 |   0.0590722 | -0.02488 |  ops/s |
|                                                Max Throughput |          percolator_with_content_ignore_me |   0.0868816 |   0.0594171 | -0.02746 |  ops/s |
|                                      100th percentile latency |          percolator_with_content_ignore_me |     6650.64 |      554974 |   548323 |     ms |
|                                 100th percentile service time |          percolator_with_content_ignore_me |     6646.96 |     17065.5 |  10418.5 |     ms |
|                                                    error rate |          percolator_with_content_ignore_me |           0 |           0 |        0 |      % |
|                                                Min Throughput | percolator_no_score_with_content_ignore_me |      15.063 |     15.0636 |  0.00065 |  ops/s |
|                                               Mean Throughput | percolator_no_score_with_content_ignore_me |     15.0852 |     15.0871 |  0.00188 |  ops/s |
|                                             Median Throughput | percolator_no_score_with_content_ignore_me |     15.0818 |     15.0839 |  0.00207 |  ops/s |
|                                                Max Throughput | percolator_no_score_with_content_ignore_me |     15.1164 |     15.1197 |  0.00327 |  ops/s |
|                                      100th percentile latency | percolator_no_score_with_content_ignore_me |     5.90333 |     4.64105 | -1.26229 |     ms |
|                                 100th percentile service time | percolator_no_score_with_content_ignore_me |      4.8496 |     3.54795 | -1.30165 |     ms |
|                                                    error rate | percolator_no_score_with_content_ignore_me |           0 |           0 |        0 |      % |