-
Notifications
You must be signed in to change notification settings - Fork 39
Performance
siimsuu1 edited this page Dec 1, 2020
·
61 revisions
In order to produce comparable test results all performance tests are done in the same server and with same methods. Tests are written as unit tests by using Contiperf2 and JUnit libraries and are located under the PerformanceTest class. Performance tests are run as Jenkins jobs in a Linux based environment and the latest digidoc4j release is compared against three earlier versions. After running the performance test job in Jenkins, Contiperf2 created a HTML-report under target/contiperf-report. Results are exported manually and combined in the table below.
Values in the test results are in milliseconds: 1: Average: Average response time for 1 sample 2: Max: Maximum response time for 1 sample
| Test (Test in Code) | Conditions | 4.1.0-RC.1 19.11.2020 |
4.0.3 19.11.2020 |
|---|---|---|---|
| Validate Bdoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 47 ms (Max 3326 ms) | Average 36 ms (Max 1739 ms) |
| Validate ASiC-E (validateTestAsiceSignatures) | 1000 samples in 1 thread | Average 90 ms (Max 164 ms) | Average 81 ms (Max 155 ms) |
| Validate ASiC-S (validateTestAsicsSignatures) | 1000 samples in 1 thread | Average 1 ms (Max 10 ms) | Average 1 ms (Max 8 ms) |
| Validate Ddoc (validateTestDDocSignatures) | 1000 samples in 1 thread | Average 7 ms (Max 20 ms) | Average 7 ms (Max 28 ms) |
| Validate Bdoc-TM in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 149 ms (Max 395 ms) | Average 107 ms (Max 249 ms) |
| Validate ASiC-E in 20 threads (validateTestAsiceSignaturesInThreads) | 1000 samples in 20 threads | Average 277 ms (Max 1169 ms) | Average 222 ms (Max 992 ms) |
| Validate ASiC-S in 20 threads (validateTestAsicsSignaturesInThreads) | 1000 samples in 20 threads | Average 4 ms (Max 33 ms) | Average 4 ms (Max 27 ms) |
| Validate Ddoc in 20 threads (validateTestDDocSignaturesInThreads) | 1000 samples in 20 threads | Average 37 ms (Max 192 ms) | Average 23 ms (Max 136 ms) |
| Validate large Bdoc-TM (validateTestBdocLargeContainer) | 1000 samples in 1 thread | Average 193 ms (Max 310 ms) | Average 183 ms (Max 296 ms) |
| Validate large ASiC-E (validateTestAsiceLargeContainer) | 1000 samples in 1 thread | Average 195 ms (Max 263 ms) | Average 183 ms (Max 296 ms) |
| Validate large ASiC-S (validateTestAsicsLargeContainer) | 1000 samples in 1 thread | Average 73 ms (Max 84 ms) | Average 71 ms (Max 127 ms) |
| Validate large Ddoc (validateTestDdocLargeContainer) | 1000 samples in 1 thread | Average 157 ms (Max 186 ms) | Average 146 ms (Max 230 ms) |
| Validate Bdoc-TM with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | Average 15458 ms (Max 15458 ms) | Average 15673 ms (Max 15673 ms) |
| Validate ASiC-E with 1000 signatures (validateAsiceWith1000Signatures) | 1 sample in 1 thread | Average 25541 ms (Max 25541 ms) | Average 23842 ms (Max 23842 ms) |
| Validate ASiC-S with 1000 signatures (validateAsicsWith1000Signatures) | 1 sample in 1 thread | Average 133 ms (Max 133 ms) | Average 128 ms (Max 128 ms) |
| Validate Ddoc with 1000 signatures (validateDddocWith1000Signatures) | 1 sample in 1 thread | Average 10315 ms (Max 10315 ms) | Average 10247 ms (Max 10247 ms) |
| Open Bdoc-TM details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 12 ms (Max 103 ms) | Average 10 ms (Max 79 ms) |
| Open ASiC-E details (openAsiceContainerDetails) | 1000 samples in 1 thread | Average 16 ms (Max 202 ms) | Average 14 ms (Max 155 ms) |
| Open ASiC-S details (openAsicsContainerDetails) | 1000 samples in 1 thread | Average 0 ms (Max 20 ms) | Average 0 ms (Max 1 ms) |
| Open Ddoc details (openDdocContainerDetails) | 1000 samples in 1 thread | Average 4 ms (Max 16 ms) | Average 4 ms (Max 29 ms) |
| Save container on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 67 ms (Max 511 ms) | Average 53 ms (Max 527 ms) |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 7744 ms (Max 8648 ms) | Average 8353 ms (Max 9809 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 124 ms (Max 343 ms) | Average 121 ms (Max 331 ms) |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 235 ms (Max 596 ms) | Average 217 ms (Max 485 ms) |
Performance tests are run in an Jenkins Linux VM environment with the following parameters: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz 8 virtual cores with 8GB RAM.
Last run in 10.01.2020
| Test (Test in Code) | Conditions | 4.0.0 10.02.2020 |
4.0.0-RC.1 11.12.2019 |
3.3.1 11.02.2020 |
3.3.0 21.08.2019 |
3.2.0 17.06.2019 |
3.1.1 11.04.2019 |
3.1.0 04.03.2019 |
3.0.0 18.11.2018 |
2.1.1 18.11.2018 |
2.1.0 18.11.2018 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Validate Bdoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 33 ms (Max 110 ms) | Average 28 ms (Max 101 ms) | Average 30 ms (Max 115 ms) | Average 28 ms (Max 95 ms) | Average 28 ms (Max 109 ms) | Average 29 ms (Max 92 ms) | Average 38 ms (Max 105 ms) | Average 28 ms (Max 1,431 ms) | Average 31 ms (Max 55 ms) | Average 38 ms (Max 1,374 ms) |
| Validate ASiC-E (validateTestAsiceSignatures) | 1000 samples in 1 thread | Average 68 ms (Max 151 ms) | Average 38 ms (Max 110 ms) | Average 32 ms (Max 114 ms) | Average 29 ms (Max 84 ms) | Average 29 ms (Max 81 ms) | Average 30 ms (Max 104 ms) | Average 38 ms (Max 92 ms) | Average 21 ms (Max 73 ms) | Average 29 ms (Max 43 ms) | Average 31 ms (Max 48 ms) |
| Validate ASiC-S (validateTestAsicsSignatures) | 1000 samples in 1 thread | Average 1 ms (Max 17 ms) | Average 0 ms (Max 9 ms) | Average 1 ms (Max 27 ms) | Average 1 ms (Max 8 ms) | Average 1 ms (Max 6 ms) | Average 1 ms (Max 12 ms) | Average 1 ms (Max 12 ms) | Average 1 ms (Max 15 ms) | Average 1 ms (Max 15 ms) | Average 1 ms (Max 9 ms) |
| Validate Ddoc (validateTestDDocSignatures) | 1000 samples in 1 thread | Average 8 ms (Max 24 ms) | Average 6 ms (Max 32 ms) | Average 7 ms (Max 39 ms) | Average 7 ms (Max 12 ms) | Average 7 ms (Max 13 ms) | Average 7 ms (Max 15 ms) | Average 7 ms (Max 28 ms) | Average 6 ms (Max 430 ms) | Average 5 ms (Max 16 ms) | Average 6 ms (Max 425 ms) |
| Validate Bdoc-TM in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 127 ms (Max 303 ms) | Average 122 ms (Max 362 ms) | Average 113 ms (Max 251 ms) | Average 104 ms (Max 269ms) | Average 104 ms (Max 282 ms) | Average 110 ms (Max 271 ms) | Average 139 ms (Max 393 ms) | Average 90 ms (Max 223 ms) | Average 259 ms (Max 975 ms) | Average 249 ms (Max 636 ms) |
| Validate ASiC-E in 20 threads (validateTestAsiceSignaturesInThreads) | 1000 samples in 20 threads | Average 212 ms (Max 550 ms) | Average 134 ms (Max 330 ms) | Average 136 ms (Max 301 ms) | Average 114 ms (Max 362 ms) | Average 114 ms (Max 232 ms) | Average 121 ms (Max 312 ms) | Average 150 ms (Max 789 ms) | Average 103 ms (Max 264 ms) | Average 260 ms (Max 669 ms) | Average 249 ms (Max 669 ms) |
| Validate ASiC-S in 20 threads (validateTestAsicsSignaturesInThreads) | 1000 samples in 20 threads | Average 6 ms (Max 59 ms) | Average 4 ms (Max 40 ms) | Average 6 ms (Max 58 ms) | Average 5 ms (Max 62 ms) | Average 5 ms (Max 67 ms) | Average 4 ms (Max 68 ms) | Average 5 ms (Max 72 ms) | Average 5 ms (Max 53 ms) | Average 4 ms (Max 61 ms) | Average 5 ms (Max 57 ms) |
| Validate Ddoc in 20 threads (validateTestDDocSignaturesInThreads) | 1000 samples in 20 threads | Average 26 ms (Max 159 ms) | Average 21 ms (Max 125 ms) | Average 28 ms (Max 125 ms) | Average 24ms (Max 116 ms) | Average 26 ms (Max 143 ms) | Average 25 ms (Max 129 ms) | Average 26 ms (Max 135 ms) | Average 19 ms (Max 99 ms) | Average 16 ms (Max 100 ms) | Average 16 ms (Max 93 ms) |
| Validate large Bdoc-TM (validateTestBdocLargeContainer) | 1000 samples in 1 thread | Average 178 ms (Max 295 ms) | Average 143 ms (Max 276 ms) | Average 124 ms (Max 250 ms) | Average 115 ms (Max 239 ms) | Average 115 ms (Max 167 ms) | Average 121 ms (Max 188 ms) | Average 128 ms (Max 179 ms) | Average 102 ms (Max 152 ms) | Average 110 ms (Max 158 ms) | Average 75 ms (Max 94 ms) |
| Validate large ASiC-E (validateTestAsiceLargeContainer) | 1000 samples in 1 thread | Average 180 ms (Max 280 ms) | Average 143 ms (Max 226 ms) | Average 126 ms (Max 327 ms) | Average 115 ms (Max 168 ms) | Average 115 ms (Max 165 ms) | Average 121 ms (Max 175 ms) | Average 129 ms (Max 187 ms) | Average 107 ms (Max 161 ms) | Average 110 ms (Max 125 ms) | Average 76 ms (Max 289 ms) |
| Validate large ASiC-S (validateTestAsicsLargeContainer) | 1000 samples in 1 thread | Average 74 ms (Max 117 ms) | Average 73 ms (Max 305 ms) | Average 79 ms (Max 106 ms) | Average 77 ms (Max 107 ms) | Average 77 ms (Max 101 ms) | Average 77 ms (Max 100 ms) | Average 77 ms (Max 101 ms) | Average 77 ms (Max 93 ms) | Average 77 ms (Max 110 ms) | Average 76 ms (Max 98 ms) |
| Validate large Ddoc (validateTestDdocLargeContainer) | 1000 samples in 1 thread | Average 145 ms (Max 302 ms) | Average 147 ms (Max 306 ms) | Average 149 ms (Max 191 ms) | Average 145 ms (Max 239 ms) | Average 149 ms (Max 265 ms) | Average 141 ms (Max 279 ms) | Average 138 ms (Max 269 ms) | Average 146 ms (Max 264 ms) | Average 166 ms (Max 398 ms) | Average 145 ms (Max 230 ms) |
| Validate Bdoc-TM with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | Average 15,056 ms (Max 15,056 ms) | Average 12,383 ms (Max 12,383 ms) | Average 11,260 ms (Max 11,260 ms) | Average 10,516 ms (Max 10,516 ms) | Average 10,526 ms (Max 10,526 ms) | Average 10,729 ms (Max 10,729 ms) | Average 12,176 ms (Max 12,176 ms) | Average 9,228 ms (Max 9,228 ms) | Average 8,791 ms (Max 8,791 ms) | Average 10,454 ms (Max 10,454 ms) |
| Validate ASiC-E with 1000 signatures (validateAsiceWith1000Signatures) | 1 sample in 1 thread | Average 21,423 ms (Max 21,423 ms) | Average 15,923 ms (Max 15,923 ms) | Average 14,159 ms (Max 14,159 ms) | Average 12,336 ms (Max 12,336 ms) | Average 12,842 ms (Max 12,842 ms) | Average 13,354 ms (Max 13,354 ms) | Average 14,254 ms (Max 14,254 ms) | Average 11,967 ms (Max 11,967 ms) | Average 11,049 ms (Max 11,049 ms) | Average 12,947 ms (Max 12,947 ms) |
| Validate ASiC-S with 1000 signatures (validateAsicsWith1000Signatures) | 1 sample in 1 thread | Average 136 ms (Max 136 ms) | Average 135 ms (Max 135 ms) | Average 327 ms (Max 327 ms) | Average 143 ms (Max 143 ms) | Average 162 ms (Max 162 ms) | Average 144 ms (Max 144 ms) | Average 143 ms (Max 143 ms) | Average 154 ms (Max 154 ms) | Average 145 ms (Max 145 ms) | Average 152 ms (Max 152 ms) |
| Validate Ddoc with 1000 signatures (validateDddocWith1000Signatures) | 1 sample in 1 thread | Average 10,070 ms (Max 10,070 ms) | Average 8,162 ms (Max 8,162 ms) | Average 11,454 ms (Max 11,454 ms) | Average 9,507 ms (Max 9,507 ms) | Average 10,319 ms (Max 10,319 ms) | Average 10,601 ms (Max 10,601 ms) | Average 9,720 ms (Max 9,720 ms) | Average 7,691 ms (Max 7,691 ms) | Average 7,237 ms (Max 7,237 ms) | Average 7,133 ms (Max 7,133 ms) |
| Open Bdoc-TM details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 12 ms (Max 76 ms) | Average 10 ms (Max 76 ms) | Average 8 ms (Max 67 ms) | Average 7 ms (Max 68 ms) | Average 7 ms (Max 65 ms) | Average 7 ms (Max 59 ms) | Average 7 ms (Max 58 ms) | Average 6 ms (Max 64 ms) | Average 6 ms (Max 19 ms) | Average 7 ms (Max 18 ms) |
| Open ASiC-E details (openAsiceContainerDetails) | 1000 samples in 1 thread | Average 14 ms (Max 87 ms) | Average 12 ms (Max 76 ms) | Average 10 ms (Max 67 ms) | Average 9 ms (Max 67 ms) | Average 9 ms (Max 61 ms) | Average 10 ms (Max 69 ms) | Average 9 ms (Max 65 ms) | Average 8 ms (Max 137 ms) | Average 8 ms (Max 22 ms) | Average 9 ms (Max 85 ms) |
| Open ASiC-S details (openAsicsContainerDetails) | 1000 samples in 1 thread | Average 0 ms (Max 1 ms) | Average 0 ms (Max 14 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 19 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 7 ms) | Average 0 ms (Max 1 ms) |
| Open Ddoc details (openDdocContainerDetails) | 1000 samples in 1 thread | Average 4 ms (Max 25 ms) | Average 3 ms (Max 15 ms) | Average 4 ms (Max 34 ms) | Average 4 ms (Max 11 ms) | Average 4 ms (Max 20 ms) | Average 4 ms (Max 11 ms) | Average 4 ms (Max 28 ms) | Average 3 ms (Max 24 ms) | Average 3 ms (Max 19 ms) | Average 3 ms (Max 19 ms) |
| Save container on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 57 ms (Max 576 ms) | Average 56 ms (Max 710 ms) | Average 41 ms (Max 247 ms) | Average 59 ms (Max 330 ms) | Average 60 ms (Max 363 ms) | Average 82 ms (Max 613 ms) | Average 50 ms (Max 247 ms) | Average 44 ms (Max 351 ms) | Average 53 ms (Max 1,119 ms) | Average 54 ms (Max 521 ms) |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 6,027 ms (Max 6,516 ms) | Average 5,563 ms (Max 5,843 ms) | Average 2,864 ms (Max 3,133 ms) | Average 3,234 ms (Max 5,081 ms) | Average 3,003 ms (Max 3,144 ms) | Average 3,801 ms (Max 5,960 ms) | Average 2,915 ms (Max 3,815 ms) | Average 2,908 ms (Max 3,985 ms) | Average 2,930 ms (Max 4,115 ms) | Average 3,007 ms (Max 4,145 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 502 ms (Max 1,410 ms) | Average 378 ms (Max 1,006 ms) | Average 364 ms (Max 895 ms) | Average 343 ms (Max 1,096 ms) | Average 309 ms (Max 541 ms) | Average 303 ms (Max 578 ms) | Average 332 ms (Max 2,298 ms) | Average 320 ms (Max 2,084 ms) | Average 293 ms (Max 731 ms) | Average 275 ms (Max 420 ms) |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 599 ms (Max 2,268 ms) | Average 442 ms (Max 819 ms) | Average 500 ms (Max 1,466 ms) | Average 447 ms (Max 1,198 ms) | Average 375 ms (Max 729 ms) | Average 354 ms (Max 730 ms) | Average 383 ms (Max 2,124 ms) | Average 330 ms (Max 399 ms) | Average 321 ms (Max 461 ms) | Average 322 ms (Max 377 ms) |
In versions 2.1.1 and prior logging level must be at least INFO, otherwise DDOC validation is much slower.
Performance tests are run in an Jenkins Linux VM environment with the following parameters: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz 8 virtual cores with 4GB RAM.
- Validate Bdoc-TM : Validation of one small BDOC container with one LT_TM signature.
- Validate ASiC-E : Validation of one small ASiC-E container with one LT signature.
- Validate ASiC-S : Validation of one small ASiC-S container.
- Validate Ddoc : Validation of one small DDOC container with one signature.
- Validate Bdoc-TM in 20 threads : Validation of one small BDOC container with one LT_TM signature.
- Validate ASiC-E in 20 threads : Validation of one small ASiC-E container with one LT signature.
- Validate ASiC-S in 20 threads : Validation of one small ASiC-S container.
- Validate Ddoc in 20 threads : Validation of one small DDOC container with one signature.
- Validate large Bdoc-TM : Validation of one large BDOC container (size is 2.8MB) with one LT_TM signature.
- Validate large ASiC-E : Validation of one large ASiC-E container (size is 2.8MB) with one LT signature.
- Validate large ASiC-S : Validation of one large ASiC-S container (size is 2.8MB).
- Validate large Ddoc : Validation of one large DDOC container (size is 2.8MB) with one signature.
- Save container on Disk : Saving of one BDOC container to test folder and checking the existence of the file on disk.
- TSL loading : Refreshing TSL data in context of PROD-environment.
- Create TM signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT_TM profile.
- Create TS signature : Signing of one existing ASiC-E container with PKCS12 token, creating a signature with LT profile.
Official builds are provided through releases. If you want support, you need to be using official builds. For assistance, contact us by email help@ria.ee. Additional information can be found in wiki Q&A and on ID.ee portal.
For staying up to date with news impacting services and applications that use the DigiDoc4j library, join DigiDoc4j library newsletter.
Source code is provided on "as is" terms with no warranty (see license for more information). Do not file GitHub issues with generic support requests.