-
Notifications
You must be signed in to change notification settings - Fork 40
Performance
luhtarur edited this page Mar 6, 2018
·
61 revisions
In order to produce comparable test results all performance tests are done in the same server and with same methods. Tests are written as unit tests by using Contiperf2 and JUnit libraries and are located under the PerformanceTest class. Performance tests are run as Jenkins jobs in a Linux based environment and the latest digidoc4j release is compared against three earlier versions. After running the performance test job in Jenkins, Contiperf2 created a HTML-report under target/contiperf-report. Results are exported manually and combined in the table below.
Test 06.03.2018
| Test (Test in Code) | Conditions | 2.1.0 | 2.0.0 | 1.0.7 | 1.0.6 |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 79 ms (Max 1734 ms) | Average 48 ms (Max 581 ms) | Average 45 ms (Max 521 ms) | Average 60 ms (max 856 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 140 ms (Max 217 ms) | Average 112 ms (Max 180 ms) | Average 606 ms (Max 3093 ms) | Average 120 ms (max 209 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 2544 ms (Max 5357 ms) | Average 731 ms (Max 1444 ms) | Average 851 ms (Max 1415 ms) | Average 1078 ms (Max 1766 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15953 ms | 14839 ms | 16872 ms | 18267 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 23 ms) | Average 5 ms (Max 17 ms) | Average 6 ms (Max 25 ms) | Average 11 ms (max 31 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 55 ms (Max 396 ms) | Average 25 ms (Max 127 ms) | Average 22 ms (Max 55 ms) | Average 33 ms (max 93 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 318 ms (Max 504 ms) | Average 324 ms (Max 586 ms) | Average 333 ms (Max 583 ms) | Average 355 ms (Max 674 ms) |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 392 ms (Max 685 ms) | Average 469 ms (Max 780 ms) | Average 404 ms (Max 1400 ms) | Average 406 ms (Max 581 ms) |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 353 ms (Max 709 ms) | Average 470 ms (Max 1852 ms) | Average 388 ms (Max 601 ms) | Average 430 ms (Max 1588 ms) |
Test 25.01.2018
| Test (Test in Code) | Conditions | 2.0.0 | 1.0.7 | 1.0.6 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 69 ms (Max 868 ms) | Average 21 ms (Max 786 ms) | Average 85 ms (Max 974 ms) | 25 ms (max 54 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 124 ms (Max 199 ms) | Average 766 ms (Max 1614 ms) | Average 139 ms (Max 215 ms) | 333 ms (max 906 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 794 ms (Max 1333 ms) | Average 1068 ms (Max 1688 ms) | Average 1255 ms (Max 2324 ms) | Average 922 ms (Max 1652 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15519 ms | 19912 ms | 23110 ms | 8638 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 19 ms) | Average 5 ms (Max 18 ms) | Average 14 ms (Max 33 ms) | 32 ms (max 85 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 25 ms (Max 65 ms) | Average 19 ms (Max 66 ms) | Average 38 ms (Max 107 ms) | 11 ms (max 114 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 344 ms (Max 515 ms) | Average 347 ms (Max 957 ms) | Average 356 ms (Max 653 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 422 ms (Max 671 ms) | Average 414 ms (Max 620 ms) | Average 417 ms (Max 1428 ms) | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 6233 ms (Max 7224 ms) | Average 620 ms (Max 770 ms) | Average 618 ms (Max 740) | - |
Test 19.10.2017
Changed VM config, upgraded from 2 to 8 virtual cores.
| Test (Test in Code) | Conditions | 1.0.7 | 1.0.6 | 1.0.5 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 55 ms (Max 721 ms) | Average 70 ms (Max 751 ms) | Average 69 ms (Max 500 ms) | 6ms (max 26 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 709 ms (Max 1910 ms) | Average 125 ms (Max 218 ms) | Average 126 ms (Max 215 ms) | 147 ms (max 164 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 881 ms (Max 1380 ms) | Average 1005 ms (Max 1976 ms) | - | Average 92 ms (Max 374 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 15859 ms | 15222 ms | 26871 ms | 5782 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 5 ms (Max 20 ms) | Average 11 ms (Max 36 ms) | Average 10 ms (Max 71 ms) | 8 ms (max 35 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 22 ms (Max 81 ms) | Average 31 ms (Max 137 ms) | Average 35 ms (Max 53 ms) | 5 ms (max 65 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 326 ms (Max 580 ms) | Average 475 ms (Max 1408 ms) | Average 391 ms (Max 678 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 419 ms (Max 722 ms) | Average 779 ms (Max 3713 ms) | Average 506 ms (Max 3424 ms | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 655 ms (Max 2300 ms) | Average 823 ms (Max 2290 ms) | Average 476 ms (Max 914) | - |
Test 17.10.2017
| Test (Test in Code) | Conditions | 1.0.7 | 1.0.6 | 1.0.5 | jDigidoc |
|---|---|---|---|---|---|
| Validate BDoc-TM (validateTestBDocTmSignatures) | 1000 samples in 1 thread | Average 51 ms (Max 581 ms) | Average 68 ms (Max 911 ms) | Average 68 ms (Max 644 ms) | 10ms (max 17 ms) |
| Validate large BDoc (validateLargeContainer) | 1000 samples in 1 thread | Average 920 ms (Max 2622 ms) | Average 111 ms (Max 198 ms) | Average 68 ms (Max 644 ms) | 147 ms (max 164 ms) |
| Validate 3 BDocs in 20 threads (validateTestBDocTmSignaturesInThreads) | 1000 samples in 20 threads | Average 879 ms (Max 1557 ms) | - | - | 365 ms (max 669 ms) |
| Validate BDoc with 1000 signatures (validateBDocWith1000Signatures) | 1 sample in 1 thread | 17764 ms | 21446 ms | 19585 ms | 2127 ms |
| Open BDoc details (openBDocTmContainerDetails) | 1000 samples in 1 thread | Average 7 ms (Max 21 ms) | Average 10 ms (Max 52 ms) | Average 14 ms (Max 205 ms) | 10 ms (max 22 ms) |
| Save BDoc on Disk (saveExistingContainerOnDisk) | 1000 samples in 20 threads | Average 13 ms (Max 71 ms) | Average 21 ms (Max 146 ms) | Average 40 ms (Max 164 ms) | 6 ms (max 170 ms) |
| Create TM signature (createBDocTmSignature) | 50 samples in 1 thread | Average 366 ms (Max 651 ms) | Average 359 ms (Max 655 ms) | Average 358 ms (Max 499 ms) | - |
| Create TS signature (createAsicSignature) | 50 samples in 1 thread | Average 428 ms (Max 1365 ms) | Average 412 ms (Max 638 ms) | Average 403 ms (Max 530 ms | - |
| TSL loading (loadingTSL) | 10 samples in 1 thread | Average 991 ms (Max 2277 ms) | Average 991 ms (Max 2277 ms) | 2465 ms | - |
Values in the test results are in milliseconds: 1: Average: Average response time for 1 sample 2: Max: Maximum response time for 1 sample
As there is no SLA for digidoc4j response times are evaluated against jDigidoc where applicable.
Performance tests are run in an Jenkins Linux VM environment with the following parameters: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz 8 virtual cores with 4GB RAM.
- Validate BDoc-TM : Validation of one small BDOC container with one LT_TM signature.
- Validate large BDoc : Validation of one large BDOC container (size is 2.8MB) with one LT_TM signature.
- Validate 3 BDocs in 20 threads : Validation of two valid BDOC containers and one non-valid BDOC container.
- Validate BDoc with 1000 signatures : Validation of one BDOC container with 1000 LT_TM signature.
- Open BDoc details : Opening of one BDOC container and checking name of the datafile and content of the signature.
- Save BDoc on Disk : Saving of one BDOC container to test folder and checking the existence of the file on disk.
- Create TM signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT_TM profile.
- Create TS signature : Signing of one existing BDOC container with PKCS12 token, creating a signature with LT profile.
- TSL loading : Refreshing TSL data in context of PROD-environment.
Official builds are provided through releases. If you want support, you need to be using official builds. For assistance, contact us by email help@ria.ee. Additional information can be found in wiki Q&A and on ID.ee portal.
For staying up to date with news impacting services and applications that use the DigiDoc4j library, join DigiDoc4j library newsletter.
Source code is provided on "as is" terms with no warranty (see license for more information). Do not file GitHub issues with generic support requests.