-
Notifications
You must be signed in to change notification settings - Fork 40
Performance
naare edited this page Aug 21, 2019
·
61 revisions
In order to produce comparable test results all performance tests are done in the same server and with same methods. Tests are written as unit tests by using Contiperf2 and JUnit libraries and are located under the PerformanceTest class. Performance tests are run as Jenkins jobs in a Linux based environment and the latest digidoc4j release is compared against three earlier versions. After running the performance test job in Jenkins, Contiperf2 created a HTML-report under target/contiperf-report. Results are exported manually and combined in the table below.
Values in the test results are in milliseconds: 1: Average: Average response time for 1 sample 2: Max: Maximum response time for 1 sample
Last run in 21.08.2019
| Test (Test in Code) | Conditions | 3.3.0 21.08.2019 |
3.2.0 17.06.2019 |
3.1.1 11.04.2019 |
3.1.0 04.03.2019 |
3.0.0 18.11.2018 |
2.1.1 18.11.2018 |
2.1.0 18.11.2018 |
|---|---|---|---|---|---|---|---|---|
| Validate Bdoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 28 ms (Max 95 ms) | Average 28 ms (Max 109 ms) | Average 29 ms (Max 92 ms) | Average 38 ms (Max 105 ms) | Average 28 ms (Max 1,431 ms) | Average 31 ms (Max 55 ms) | Average 38 ms (Max 1,374 ms) |
| Validate ASiC-E (validateTestAsiceSignatures) | 1000 samples in 1 thread | Average 29 ms (Max 84 ms) | Average 29 ms (Max 81 ms) | Average 30 ms (Max 104 ms) | Average 38 ms (Max 92 ms) | Average 21 ms (Max 73 ms) | Average 29 ms (Max 43 ms) | Average 31 ms (Max 48 ms) |
| Validate ASiC-S (validateTestAsicsSignatures) | 1000 samples in 1 thread | Average 1 ms (Max 8 ms) | Average 1 ms (Max 6 ms) | Average 1 ms (Max 12 ms) | Average 1 ms (Max 12 ms) | Average 1 ms (Max 15 ms) | Average 1 ms (Max 15 ms) | Average 1 ms (Max 9 ms) |
| Validate Ddoc (validateTestDDocSignatures) | 1000 samples in 1 thread | Average 7 ms (Max 12 ms) | Average 7 ms (Max 13 ms) | Average 7 ms (Max 15 ms) | Average 7 ms (Max 28 ms) | Average 6 ms (Max 430 ms) | Average 5 ms (Max 16 ms) | Average 6 ms (Max 425 ms) |
| Validate Bdoc-TM in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 104 ms (Max 269ms) | Average 104 ms (Max 282 ms) | Average 110 ms (Max 271 ms) | Average 139 ms (Max 393 ms) | Average 90 ms (Max 223 ms) | Average 259 ms (Max 975 ms) | Average 249 ms (Max 636 ms) |
| Validate ASiC-E in 20 threads (validateTestAsiceSignaturesInThreads) | 1000 samples in 20 threads | Average 114 ms (Max 362 ms) | Average 114 ms (Max 232 ms) | Average 121 ms (Max 312 ms) | Average 150 ms (Max 789 ms) | Average 103 ms (Max 264 ms) | Average 260 ms (Max 669 ms) | Average 249 ms (Max 669 ms) |
| Validate ASiC-S in 20 threads (validateTestAsicsSignaturesInThreads) | 1000 samples in 20 threads | Average 5 ms (Max 62 ms) | Average 5 ms (Max 67 ms) | Average 4 ms (Max 68 ms) | Average 5 ms (Max 72 ms) | Average 5 ms (Max 53 ms) | Average 4 ms (Max 61 ms) | Average 5 ms (Max 57 ms) |
| Validate Ddoc in 20 threads (validateTestDDocSignaturesInThreads) | 1000 samples in 20 threads | Average 24ms (Max 116 ms) | Average 26 ms (Max 143 ms) | Average 25 ms (Max 129 ms) | Average 26 ms (Max 135 ms) | Average 19 ms (Max 99 ms) | Average 16 ms (Max 100 ms) | Average 16 ms (Max 93 ms) |
| Validate large Bdoc-TM (validateTestBdocLargeContainer) | 1000 samples in 1 thread | Average 115 ms (Max 239 ms) | Average 115 ms (Max 167 ms) | Average 121 ms (Max 188 ms) | Average 128 ms (Max 179 ms) | Average 102 ms (Max 152 ms) | Average 110 ms (Max 158 ms) | Average 75 ms (Max 94 ms) |
| Validate large ASiC-E (validateTestAsiceLargeContainer) | 1000 samples in 1 thread | Average 115 ms (Max 168 ms) | Average 115 ms (Max 165 ms) | Average 121 ms (Max 175 ms) | Average 129 ms (Max 187 ms) | Average 107 ms (Max 161 ms) | Average 110 ms (Max 125 ms) | Average 76 ms (Max 289 ms) |
| Validate large ASiC-S (validateTestAsicsLargeContainer) | 1000 samples in 1 thread | Average 77 ms (Max 107 ms) | Average 77 ms (Max 101 ms) | Average 77 ms (Max 100 ms) | Average 77 ms (Max 101 ms) | Average 77 ms (Max 93 ms) | Average 77 ms (Max 110 ms) | Average 76 ms (Max 98 ms) |
| Validate large Ddoc (validateTestDdocLargeContainer) | 1000 samples in 1 thread | Average 145 ms (Max 239 ms) | Average 149 ms (Max 265 ms) | Average 141 ms (Max 279 ms) | Average 138 ms (Max 269 ms) | Average 146 ms (Max 264 ms) | Average 166 ms (Max 398 ms) | Average 145 ms (Max 230 ms) |
| Validate Bdoc-TM with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | Average 10,516 ms (Max 10,516 ms) | Average 10,526 ms (Max 10,526 ms) | Average 10,729 ms (Max 10,729 ms) | Average 12,176 ms (Max 12,176 ms) | Average 9,228 ms (Max 9,228 ms) | Average 8,791 ms (Max 8,791 ms) | Average 10,454 ms (Max 10,454 ms) |
| Validate ASiC-E with 1000 signatures (validateAsiceWith1000Signatures) | 1 sample in 1 thread | Average 12,336 ms (Max 12,336 ms) | Average 12,842 ms (Max 12,842 ms) | Average 13,354 ms (Max 13,354 ms) | Average 14,254 ms (Max 14,254 ms) | Average 11,967 ms (Max 11,967 ms) | Average 11,049 ms (Max 11,049 ms) | Average 12,947 ms (Max 12,947 ms) |
| Validate ASiC-S with 1000 signatures (validateAsicsWith1000Signatures) | 1 sample in 1 thread | Average 143 ms (Max 143 ms) | Average 162 ms (Max 162 ms) | Average 144 ms (Max 144 ms) | Average 143 ms (Max 143 ms) | Average 154 ms (Max 154 ms) | Average 145 ms (Max 145 ms) | Average 152 ms (Max 152 ms) |
| Validate Ddoc with 1000 signatures (validateDddocWith1000Signatures) | 1 sample in 1 thread | Average 9,507 ms (Max 9,507 ms) | Average 10,319 ms (Max 10,319 ms) | Average 10,601 ms (Max 10,601 ms) | Average 9,720 ms (Max 9,720 ms) | Average 7,691 ms (Max 7,691 ms) | Average 7,237 ms (Max 7,237 ms) | Average 7,133 ms (Max 7,133 ms) |
| Open Bdoc-TM details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 68 ms) | Average 7 ms (Max 65 ms) | Average 7 ms (Max 59 ms) | Average 7 ms (Max 58 ms) | Average 6 ms (Max 64 ms) | Average 6 ms (Max 19 ms) | Average 7 ms (Max 18 ms) |
| Open ASiC-E details (openAsiceContainerDetails) | 1000 samples in 1 thread | Average 9 ms (Max 67 ms) | Average 9 ms (Max 61 ms) | Average 10 ms (Max 69 ms) | Average 9 ms (Max 65 ms) | Average 8 ms (Max 137 ms) | Average 8 ms (Max 22 ms) | Average 9 ms (Max 85 ms) |
| Open ASiC-S details (openAsicsContainerDetails) | 1000 samples in 1 thread | Average 0 ms (Max 19 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 1 ms) | Average 0 ms (Max 7 ms) | Average 0 ms (Max 1 ms) |
| Open Ddoc details (openDdocContainerDetails) | 1000 samples in 1 thread | Average 4 ms (Max 11 ms) | Average 4 ms (Max 20 ms) | Average 4 ms (Max 11 ms) | Average 4 ms (Max 28 ms) | Average 3 ms (Max 24 ms) | Average 3 ms (Max 19 ms) | Average 3 ms (Max 19 ms) |
| Save container on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 59 ms (Max 330 ms) | Average 60 ms (Max 363 ms) | Average 82 ms (Max 613 ms) | Average 50 ms (Max 247 ms) | Average 44 ms (Max 351 ms) | Average 53 ms (Max 1,119 ms) | Average 54 ms (Max 521 ms) |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 3,234 ms (Max 5,081 ms) | Average 3,003 ms (Max 3,144 ms) | Average 3,801 ms (Max 5,960 ms) | Average 2,915 ms (Max 3,815 ms) | Average 2,908 ms (Max 3,985 ms) | Average 2,930 ms (Max 4,115 ms) | Average 3,007 ms (Max 4,145 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 343 ms (Max 1,096 ms) | Average 309 ms (Max 541 ms) | Average 303 ms (Max 578 ms) | Average 332 ms (Max 2,298 ms) | Average 320 ms (Max 2,084 ms) | Average 293 ms (Max 731 ms) | Average 275 ms (Max 420 ms) |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 447 ms (Max 1,198 ms) | Average 375 ms (Max 729 ms) | Average 354 ms (Max 730 ms) | Average 383 ms (Max 2,124 ms) | Average 330 ms (Max 399 ms) | Average 321 ms (Max 461 ms) | Average 322 ms (Max 377 ms) |
In versions 2.1.1 and prior logging level must be at least INFO, otherwise DDOC validation is much slower.
Performance tests are run in an Jenkins Linux VM environment with the following parameters: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz 8 virtual cores with 4GB RAM.
- Validate Bdoc-TM : Validation of one small BDOC container with one LT_TM signature.
- Validate ASiC-E : Validation of one small ASiC-E container with one LT signature.
- Validate ASiC-S : Validation of one small ASiC-S container.
- Validate Ddoc : Validation of one small DDOC container with one signature.
- Validate Bdoc-TM in 20 threads : Validation of one small BDOC container with one LT_TM signature.
- Validate ASiC-E in 20 threads : Validation of one small ASiC-E container with one LT signature.
- Validate ASiC-S in 20 threads : Validation of one small ASiC-S container.
- Validate Ddoc in 20 threads : Validation of one small DDOC container with one signature.
- Validate large Bdoc-TM : Validation of one large BDOC container (size is 2.8MB) with one LT_TM signature.
- Validate large ASiC-E : Validation of one large ASiC-E container (size is 2.8MB) with one LT signature.
- Validate large ASiC-S : Validation of one large ASiC-S container (size is 2.8MB).
- Validate large Ddoc : Validation of one large DDOC container (size is 2.8MB) with one signature.
- Save container on Disk : Saving of one BDOC container to test folder and checking the existence of the file on disk.
- TSL loading : Refreshing TSL data in context of PROD-environment.
- Create TM signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT_TM profile.
- Create TS signature : Signing of one existing ASiC-E container with PKCS12 token, creating a signature with LT profile.
Test 06.03.2018
| Test (Test in Code) | Conditions | 2.1.0 | 2.0.0 | 1.0.7 | 1.0.6 |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 79 ms (Max 1734 ms) | Average 48 ms (Max 581 ms) | Average 45 ms (Max 521 ms) | Average 60 ms (max 856 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 140 ms (Max 217 ms) | Average 112 ms (Max 180 ms) | Average 606 ms (Max 3093 ms) | Average 120 ms (max 209 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 2544 ms (Max 5357 ms) | Average 731 ms (Max 1444 ms) | Average 851 ms (Max 1415 ms) | Average 1078 ms (Max 1766 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15953 ms | 14839 ms | 16872 ms | 18267 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 23 ms) | Average 5 ms (Max 17 ms) | Average 6 ms (Max 25 ms) | Average 11 ms (max 31 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 55 ms (Max 396 ms) | Average 25 ms (Max 127 ms) | Average 22 ms (Max 55 ms) | Average 33 ms (max 93 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 318 ms (Max 504 ms) | Average 324 ms (Max 586 ms) | Average 333 ms (Max 583 ms) | Average 355 ms (Max 674 ms) |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 392 ms (Max 685 ms) | Average 469 ms (Max 780 ms) | Average 404 ms (Max 1400 ms) | Average 406 ms (Max 581 ms) |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 353 ms (Max 709 ms) | Average 470 ms (Max 1852 ms) | Average 388 ms (Max 601 ms) | Average 430 ms (Max 1588 ms) |
Test 25.01.2018
| Test (Test in Code) | Conditions | 2.0.0 | 1.0.7 | 1.0.6 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 69 ms (Max 868 ms) | Average 21 ms (Max 786 ms) | Average 85 ms (Max 974 ms) | 25 ms (max 54 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 124 ms (Max 199 ms) | Average 766 ms (Max 1614 ms) | Average 139 ms (Max 215 ms) | 333 ms (max 906 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 794 ms (Max 1333 ms) | Average 1068 ms (Max 1688 ms) | Average 1255 ms (Max 2324 ms) | Average 922 ms (Max 1652 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15519 ms | 19912 ms | 23110 ms | 8638 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 19 ms) | Average 5 ms (Max 18 ms) | Average 14 ms (Max 33 ms) | 32 ms (max 85 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 25 ms (Max 65 ms) | Average 19 ms (Max 66 ms) | Average 38 ms (Max 107 ms) | 11 ms (max 114 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 344 ms (Max 515 ms) | Average 347 ms (Max 957 ms) | Average 356 ms (Max 653 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 422 ms (Max 671 ms) | Average 414 ms (Max 620 ms) | Average 417 ms (Max 1428 ms) | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 6233 ms (Max 7224 ms) | Average 620 ms (Max 770 ms) | Average 618 ms (Max 740) | - |
Test 19.10.2017
Changed VM config, upgraded from 2 to 8 virtual cores.
| Test (Test in Code) | Conditions | 1.0.7 | 1.0.6 | 1.0.5 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 55 ms (Max 721 ms) | Average 70 ms (Max 751 ms) | Average 69 ms (Max 500 ms) | 6ms (max 26 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 709 ms (Max 1910 ms) | Average 125 ms (Max 218 ms) | Average 126 ms (Max 215 ms) | 147 ms (max 164 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 881 ms (Max 1380 ms) | Average 1005 ms (Max 1976 ms) | - | Average 92 ms (Max 374 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15859 ms | 15222 ms | 26871 ms | 5782 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 5 ms (Max 20 ms) | Average 11 ms (Max 36 ms) | Average 10 ms (Max 71 ms) | 8 ms (max 35 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 22 ms (Max 81 ms) | Average 31 ms (Max 137 ms) | Average 35 ms (Max 53 ms) | 5 ms (max 65 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 326 ms (Max 580 ms) | Average 475 ms (Max 1408 ms) | Average 391 ms (Max 678 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 419 ms (Max 722 ms) | Average 779 ms (Max 3713 ms) | Average 506 ms (Max 3424 ms) | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 655 ms (Max 2300 ms) | Average 823 ms (Max 2290 ms) | Average 476 ms (Max 914) | - |
Test 17.10.2017
| Test (Test in Code) | Conditions | 1.0.7 | 1.0.6 | 1.0.5 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 51 ms (Max 581 ms) | Average 68 ms (Max 911 ms) | Average 68 ms (Max 644 ms) | 10ms (max 17 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 920 ms (Max 2622 ms) | Average 111 ms (Max 198 ms) | Average 68 ms (Max 644 ms) | 147 ms (max 164 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 879 ms (Max 1557 ms) | - | - | 365 ms (max 669 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 17764 ms | 21446 ms | 19585 ms | 2127 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 21 ms) | Average 10 ms (Max 52 ms) | Average 14 ms (Max 205 ms) | 10 ms (max 22 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 13 ms (Max 71 ms) | Average 21 ms (Max 146 ms) | Average 40 ms (Max 164 ms) | 6 ms (max 170 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 366 ms (Max 651 ms) | Average 359 ms (Max 655 ms) | Average 358 ms (Max 499 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 428 ms (Max 1365 ms) | Average 412 ms (Max 638 ms) | Average 403 ms (Max 530 ms) | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 991 ms (Max 2277 ms) | Average 991 ms (Max 2277 ms) | 2465 ms | - |
As there is no SLA for digidoc4j response times are evaluated against jDigidoc where applicable.
Performance tests are run in an Jenkins Linux VM environment with the following parameters: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz 8 virtual cores with 4GB RAM.
- Validate BDoc-TM : Validation of one small BDOC container with one LT_TM signature.
- Validate large BDoc : Validation of one large BDOC container (size is 2.8MB) with one LT_TM signature.
- Validate 3 BDocs in 20 threads : Validation of two valid BDOC containers and one non-valid BDOC container.
- Validate BDoc with 1000 signatures : Validation of one BDOC container with 1000 LT_TM signature.
- Open BDoc details : Opening of one BDOC container and checking name of the datafile and content of the signature.
- Save BDoc on Disk : Saving of one BDOC container to test folder and checking the existence of the file on disk.
- Create TM signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT_TM profile.
- Create TS signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT profile.
- TSL loading : Refreshing TSL data in context of PROD-environment.
Official builds are provided through releases. If you want support, you need to be using official builds. For assistance, contact us by email help@ria.ee. Additional information can be found in wiki Q&A and on ID.ee portal.
For staying up to date with news impacting services and applications that use the DigiDoc4j library, join DigiDoc4j library newsletter.
Source code is provided on "as is" terms with no warranty (see license for more information). Do not file GitHub issues with generic support requests.