Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use optimised OpenSSL SHA256 function for Merkle Tree hashing #5621

Merged
merged 14 commits into from
Sep 11, 2023
Merged

Use optimised OpenSSL SHA256 function for Merkle Tree hashing #5621

merged 14 commits into from
Sep 11, 2023

Conversation

jumaffre
Copy link
Contributor

@jumaffre jumaffre commented Sep 5, 2023 •
edited
Loading

Follow up from #5548.

This may speed up end-to-end performance tests.

Merkle bench results: (

Before

append_retract:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
         append_retract * |    1000 |     0.025 |      25 |      - | 39255711.7
         append_retract * |   10000 |     0.175 |      17 |      - | 57061992.1
===============================================================================
append_flush:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
           append_flush * |    1000 |     0.019 |      18 |      - | 53824210.1
           append_flush * |   10000 |     2.830 |     282 |      - |  3533986.0
===============================================================================
append_get_proof_verify:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
append_get_proof_verify * |    1000 |     2.442 |    2441 |      - |   409558.3
append_get_proof_verify * |   10000 |    31.211 |    3121 |      - |   320404.3
===============================================================================
append_get_proof_verify_v:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
append_get_proof_verify_v * |    1000 |     3.267 |    3267 |      - |   306044.8
append_get_proof_verify_v * |   10000 |    41.074 |    4107 |      - |   243462.0
===============================================================================
serialise_deserialise:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
  serialise_deserialise * |    1000 |     0.355 |     355 |      - |  2815434.2
  serialise_deserialise * |   10000 |     3.628 |     362 |      - |  2756446.7
===============================================================================
serialised_size:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
        serialised_size * |       1 |     0.002 |    2166 |      - |   461680.5
        serialised_size * |       2 |     0.002 |     787 |      - |  1269841.3
        serialised_size * |      10 |     0.004 |     418 |      - |  2388344.9
        serialised_size * |     100 |     0.036 |     356 |      - |  2803004.8
        serialised_size * |    1000 |     0.323 |     323 |      - |  3094633.9
        serialised_size * |   10000 |     3.395 |     339 |      - |  2945784.9
===============================================================================

After

append_retract: 
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
         append_retract * |    1000 |     0.026 |      26 |      - | 38099592.3
         append_retract * |   10000 |     0.178 |      17 |      - | 56220519.4   
===============================================================================
append_flush
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
           append_flush * |    1000 |     0.019 |      18 |      - | 53659583.6
           append_flush * |   10000 |     2.223 |     222 |      - |  4497685.0
===============================================================================
append_get_proof_verify:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
append_get_proof_verify * |    1000 |     1.952 |    1951 |      - |   512423.5
append_get_proof_verify * |   10000 |    24.689 |    2468 |      - |   405032.3
===============================================================================
append_get_proof_verify_v:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
append_get_proof_verify_v * |    1000 |     2.778 |    2778 |      - |   359927.8
append_get_proof_verify_v * |   10000 |    34.563 |    3456 |      - |   289328.2
===============================================================================
serialise_deserialise:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
  serialise_deserialise * |    1000 |     0.295 |     294 |      - |  3391808.1
  serialise_deserialise * |   10000 |     3.018 |     301 |      - |  3313827.0
===============================================================================
serialised_size:
===============================================================================
   Name (baseline is *)   |   Dim   |  Total ms |  ns/op  |Baseline| Ops/second
===============================================================================
        serialised_size * |       1 |     0.001 |    1312 |      - |   762195.1
        serialised_size * |       2 |     0.002 |     906 |      - |  1103144.0
        serialised_size * |      10 |     0.004 |     447 |      - |  2234636.9
        serialised_size * |     100 |     0.047 |     468 |      - |  2134745.1
        serialised_size * |    1000 |     0.260 |     259 |      - |  3850745.1
        serialised_size * |   10000 |     3.215 |     321 |      - |  3110659.9
===============================================================================

@ghost
Copy link

ghost commented Sep 5, 2023 •
edited by ghost
Loading

merklecpp_use_faster_hashes@75679 aka 20230911.7 vs main ewma over 20 builds from 75332 to 75672

Click to see table

main

build_id build_number Commit latency factor pi_basic_mt_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem pi_basic_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_virtual_cft^ ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
75332 20230905.15 0.795074 73261.1 16686.1 1.88908e+07 32237.6 2.51822e+07 16806.5 1.05021e+07 18374.4 1.46964e+07 45805 48010.4 53472.6 1663.1 1.25993e+07 4468.6 8145.3 1.88908e+07 8056.1 6.30784e+06 17303.8 19203.1 7062.03 1.67936e+07 17495.4 14979.2 6644.82 1.67936e+07 9760 4549.04 1.88908e+07 47860 830160 1.17693e+06 8.15631e+06 3.07813e+07
75348 20230905.21 0.818926 69941.9 16897 1.88908e+07 31697 2.51822e+07 16860.5 1.05021e+07 18475.4 1.46964e+07 45750.9 48204.9 52524 1657.2 1.25993e+07 4466 8211.83 1.88908e+07 8045.4 6.30784e+06 17413.2 19456.4 7058.49 1.67936e+07 17519 15001.3 6594.25 1.67936e+07 10200.3 4559.4 1.88908e+07 43975.4 820294 1.17696e+06 8.17128e+06 3.07581e+07
75360 20230905.26 0.800555 75493.7 16639.2 1.88908e+07 32429.6 2.51822e+07 16865.9 1.05021e+07 18094.7 1.46964e+07 45554.4 42866.8 51334.5 1629.4 1.25993e+07 4480 8097.92 1.88908e+07 8244.9 6.30784e+06 17462.1 19193.1 6577.72 1.67936e+07 16930.1 15179.2 6573.38 1.67936e+07 9898.05 4545.58 1.67936e+07 47617.4 834158 1.17952e+06 8.1542e+06 3.0816e+07
75413 20230905.41 0.822033 84700.2 16619.4 1.88908e+07 32149.1 2.51822e+07 16634.2 1.05021e+07 18150.8 1.46964e+07 42002.8 47678.8 53684.4 1640.4 1.25993e+07 4426.7 8128.27 1.88908e+07 7918 6.30784e+06 17180.1 19169.9 6668.1 1.67936e+07 17447.2 14839.1 6561.8 1.67936e+07 10272.6 4544.2 1.67936e+07 46378.9 831657 1.18593e+06 8.15381e+06 3.25544e+07
75427 20230905.45 0.79923 72646.1 16851.9 1.88908e+07 32303 2.51822e+07 16843.5 1.05021e+07 18380.2 1.46964e+07 43818.6 47219.3 54172.4 1660.9 1.25993e+07 4474.6 8204.56 1.88908e+07 8033.3 6.30784e+06 17207 20035.7 7055.71 1.67936e+07 17125.1 16863.6 6604.32 1.67936e+07 9757.72 4591.15 1.88908e+07 49899.7 831718 1.18507e+06 8.16148e+06 3.07891e+07
75432 20230906.1 0.792911 72254 16953.5 1.88908e+07 32134 2.72794e+07 16886.8 1.05021e+07 18426.2 1.46964e+07 44002.6 47749.6 54054.4 1656.7 1.25993e+07 4440.1 8197.08 1.88908e+07 8067.4 6.30784e+06 17184.3 19123.7 7062.48 1.67936e+07 17476.7 14970.7 6650.18 1.67936e+07 10416.6 4558.5 1.88908e+07 47511.8 836786 1.17882e+06 8.13596e+06 3.09679e+07
75453 20230906.8 0.786039 62844.4 16911.9 1.88908e+07 31842.2 2.51822e+07 16854.2 1.05021e+07 18339 1.46964e+07 45758.2 48251.8 54707.9 1666.3 1.25993e+07 4474.7 8195.46 1.88908e+07 8236.7 6.30784e+06 17380.4 18840.6 7050.11 1.67936e+07 17589.3 14994.8 6651.5 1.67936e+07 9887.4 4550.56 1.88908e+07 47836.8 831787 1.17789e+06 8.12511e+06 3.08063e+07
75489 20230906.23 0.800124 82560 16654.9 1.88908e+07 31743.9 2.51822e+07 16776.6 1.05021e+07 18256.7 1.25993e+07 43804.6 47435.6 54050.3 1636.7 1.25993e+07 4459.5 8153.42 1.88908e+07 8125.5 6.30784e+06 17073.3 18929 6677.29 1.67936e+07 17464.5 14865.8 6628.14 1.67936e+07 10073.8 4550.28 1.88908e+07 49990.9 835269 1.17747e+06 8.13363e+06 3.07143e+07
75522 20230907.3 0.772057 66260.1 16673.7 1.88908e+07 32075.6 2.51822e+07 16929.9 1.05021e+07 18283.4 1.46964e+07 45714.3 48111.2 54314.5 1656.8 1.25993e+07 4463.3 8168.97 1.88908e+07 8023.9 6.30784e+06 17295.6 19589.4 7052.03 1.67936e+07 17065.4 14677.8 6594.49 1.67936e+07 10062.2 4565.33 1.88908e+07 49662.8 828056 1.18636e+06 8.1552e+06 3.07808e+07
75527 20230907.5 0.781303 86171.1 16807.7 1.88908e+07 32308 2.51822e+07 16890.6 1.05021e+07 18259.9 1.46964e+07 43523 47340.4 53737.5 1648.4 1.25993e+07 4404.9 8151.7 1.88908e+07 7992.3 6.30784e+06 17097.4 19462.4 7014.17 1.67936e+07 17630.6 15065 6599.19 1.67936e+07 10283.8 4558.03 1.88908e+07 42500.3 823830 1.18114e+06 8.1554e+06 3.07249e+07
75554 20230907.14 0.784733 89286.5 16776.5 1.88908e+07 32296.2 2.51822e+07 16911.3 1.05021e+07 18490.9 1.46964e+07 45865.6 47959.1 54950.9 1659.7 1.25993e+07 4492.7 8214.84 1.88908e+07 8100.2 6.30784e+06 17191.6 19221.2 7060.74 1.67936e+07 17286.2 16906.8 6600.24 1.67936e+07 9966.44 4579.56 1.88908e+07 47720.1 832953 1.17863e+06 8.1447e+06 3.05827e+07
75570 20230907.20 0.799646 62416.2 16832.6 1.88908e+07 32177.9 2.51822e+07 17001.3 1.05021e+07 18451.6 1.46964e+07 45705.3 48300.1 55108.2 1661.9 1.25993e+07 4494.8 8246.8 1.88908e+07 8192.6 6.30784e+06 17134.4 19570.6 7051.77 1.67936e+07 17098.8 16872.1 6608.3 1.67936e+07 9890.69 4556.94 1.88908e+07 46267.7 831516 1.18245e+06 8.13564e+06 3.11644e+07
75578 20230908.2 0.828259 78823.5 16699.5 1.88908e+07 32091.1 2.51822e+07 16847.4 1.05021e+07 18400.6 1.46964e+07 43942.1 48082.5 54705.4 1654.5 1.25993e+07 4443.4 8136.67 1.88908e+07 8377.3 6.30784e+06 17216 18878.3 7061.63 1.67936e+07 17041.1 16926.4 6591.31 1.67936e+07 9811.7 4554.19 1.88908e+07 48501.8 833497 1.1874e+06 8.15413e+06 3.08021e+07
75585 20230908.5 0.797482 74515.3 16880.3 1.88908e+07 32334.7 2.51822e+07 16912.1 1.05021e+07 18498.4 1.25993e+07 43449.2 48049.2 54134.4 1654.4 1.25993e+07 4445.7 8214.32 1.88908e+07 8178.3 6.30784e+06 16982.9 19441.6 6689.4 1.67936e+07 17178.3 14747.2 6602.85 1.67936e+07 10169.7 4556.48 1.88908e+07 48501 840060 1.18261e+06 8.15384e+06 3.08624e+07
75619 20230908.14 0.792552 71512.6 13892.6 1.88908e+07 27206.7 2.51822e+07 13918.6 1.05021e+07 15383.3 1.46964e+07 45711.8 47721.1 53565.9 1447 1.25993e+07 4169.8 6791.3 1.67936e+07 6879.8 6.30784e+06 17033 19433.3 5749.43 1.67936e+07 17333.6 14800.1 5721.31 1.67936e+07 9828.73 3985.08 1.67936e+07 46035.4 831032 1.18459e+06 8.14194e+06 3.08067e+07
75633 20230908.19 0.814514 87196.1 13841.4 1.88908e+07 26426.9 2.30851e+07 13952.4 1.05021e+07 15293 1.46964e+07 43679.4 47559.5 53848.3 1426.1 1.25993e+07 4427.3 6803.4 1.67936e+07 6796.8 6.30784e+06 17198.8 19457.1 5780.48 1.67936e+07 17581.4 14768.9 5455.24 1.67936e+07 10298.4 3984.61 1.67936e+07 40163.3 833682 1.18594e+06 8.13551e+06 3.05845e+07
75639 20230908.21 0.780733 86236.2 13880 1.88908e+07 27149.3 2.51822e+07 13884.3 1.05021e+07 15315.7 1.46964e+07 45714.2 47797.3 54112.7 1435.7 1.25993e+07 4489.1 7182.95 1.67936e+07 6940.1 6.30784e+06 17346.3 19833.8 5791.25 1.67936e+07 17596.5 14765.9 5708.59 1.67936e+07 9895.98 3967.22 1.67936e+07 46797.6 836115 1.1819e+06 8.14551e+06 3.06335e+07
75654 20230908.26 0.800096 89588.7 13871.4 1.67936e+07 27195 2.51822e+07 13956.2 1.05021e+07 15367 1.46964e+07 43869.4 47485.2 54275.4 1435.2 1.25993e+07 4469.5 6841.62 1.88908e+07 6895.5 6.30784e+06 17192.2 19753.5 5790.81 1.67936e+07 17307.3 15019.4 5688.99 1.67936e+07 9817.01 3967.13 1.67936e+07 43470.8 831294 1.18063e+06 8.16519e+06 3.08276e+07
75662 20230911.2 0.808688 83834.8 13935.4 1.88908e+07 27181.8 2.51822e+07 13957 1.05021e+07 15499.6 1.46964e+07 45601.5 47690.4 53934.4 1440.2 1.25993e+07 4406 6826.9 1.67936e+07 7172.6 6.30784e+06 17267 18849.8 5746.37 1.67936e+07 17581 14898.7 5756.56 1.67936e+07 9856.76 3997.11 1.67936e+07 44575.1 836042 1.18753e+06 8.15413e+06 3.15135e+07
75672 20230911.6 0.819773 75501.5 13880 1.67936e+07 26990.3 2.30851e+07 13947.8 1.05021e+07 15361.7 1.46964e+07 43845.5 47682.4 54141 1436.3 1.25993e+07 4457.3 6809.97 1.88908e+07 6953.1 6.30784e+06 17348 19305 5793.21 1.67936e+07 15082.7 14637.5 5761.62 1.67936e+07 10070.8 3995.63 1.67936e+07 43290.9 837710 1.17811e+06 8.15426e+06 3.07175e+07

merklecpp_use_faster_hashes

build_id build_number Commit latency factor pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem pi_basic_mt_virtual_cft^ ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ pi_basic_js_virtual_cft^ ls_jwt_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_jwt_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_js_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
75604 20230908.10 0.776146 32794.7 2.51822e+07 60203 44087.3 48589.9 54953.8 4450.9 17127.4 16872.2 1.88908e+07 19243 16998.2 1.05021e+07 17165.2 18547.1 1.25993e+07 14747.4 10292.5 1653.4 1.25993e+07 8213.56 1.88908e+07 8069.8 6.30784e+06 7127.21 1.67936e+07 43444.9 6651.29 1.67936e+07 4601.57 1.88908e+07 829284 1.18206e+06 8.14917e+06 3.2218e+07
75610 20230908.11 0.782553 32775 2.51822e+07 78474.6 45726 48438.1 55000.8 4446.9 16987.2 16886.1 1.88908e+07 19750.7 16988.7 1.05021e+07 17303.3 18763.1 1.25993e+07 14638.8 9875.38 1654.1 1.25993e+07 8262.61 1.88908e+07 8285.3 6.30784e+06 7083.52 1.67936e+07 47584 6626.05 1.67936e+07 4595.28 1.88908e+07 810852 1.17972e+06 8.15488e+06 3.07998e+07
75630 20230908.18 0.788298 27852.9 2.51822e+07 62129.4 45540.2 48211.1 55010.2 4463.8 17165.9 14049.7 1.88908e+07 19153.1 14177.2 1.05021e+07 17098.4 15688.4 1.46964e+07 14833.7 9782.42 1441.1 1.25993e+07 6846.39 1.88908e+07 6941.6 6.30784e+06 5780.43 1.67936e+07 45481.8 5737.36 1.67936e+07 4004.44 1.67936e+07 826009 1.17548e+06 8.15465e+06 3.06403e+07
75651 20230908.25 0.775248 28012.3 2.30851e+07 92011.9 44119.4 48588.7 54789.1 4466.3 17089.4 14051.2 1.88908e+07 19603.6 14191.3 1.05021e+07 17458.7 15644.2 1.25993e+07 14677.4 10055.9 1435.8 1.25993e+07 7259.6 1.67936e+07 7084.9 6.30784e+06 5796.74 1.67936e+07 47116.3 5738.8 1.67936e+07 3985.65 1.67936e+07 838042 1.18082e+06 8.14327e+06 3.0974e+07
75679 20230911.7 0.825868 28338.5 2.30851e+07 76833 43726.8 47424.6 54224.8 4432.2 17078.8 14038.4 1.88908e+07 19593.6 14107.6 1.05021e+07 17411.6 15625 1.46964e+07 14882.8 9849.9 1446.8 1.25993e+07 6884.99 1.88908e+07 6965.7 6.30784e+06 5814.92 1.67936e+07 44940.2 5749.48 1.67936e+07 4003.64 1.67936e+07 837230 1.18369e+06 8.13357e+06 3.08634e+07

images

achamayou
achamayou reviewed Sep 7, 2023
View reviewed changes
src/node/history.h Outdated Show resolved Hide resolved
@jumaffre jumaffre marked this pull request as ready for review September 8, 2023 10:15
@jumaffre jumaffre requested a review from a team September 8, 2023 10:15
@jumaffre jumaffre added auto-backport Automatically backport this PR to LTS branch 4.x-todo PRs which should be backported to 4.x labels Sep 8, 2023
@jumaffre
Copy link
Contributor Author

jumaffre commented Sep 8, 2023

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@achamayou achamayou enabled auto-merge (squash) September 8, 2023 15:58
@achamayou achamayou merged commit d473d75 into microsoft:main Sep 11, 2023
24 of 31 checks passed
@ghost ghost mentioned this pull request Sep 11, 2023
Merged
ghost pushed a commit that referenced this pull request Sep 11, 2023
@jumaffre
Use optimised OpenSSL SHA256 function for Merkle Tree hashing (#5621)
a5e7828 
(cherry picked from commit d473d75)
@ghost ghost added the backported This PR was successfully backported to LTS branch label Sep 11, 2023
jumaffre pushed a commit that referenced this pull request Sep 11, 2023
[release/4.x] Cherry pick: Use optimised OpenSSL SHA256 function for …
ac59518 
…Merkle Tree hashing (#5621) (#5647)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@achamayou achamayou achamayou approved these changes

Assignees
No one assigned
Labels
4.x-todo PRs which should be backported to 4.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jumaffre @achamayou