Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add end-to-end benchmark results to documentation #935

Merged
merged 7 commits into from
Jun 19, 2023

Conversation

ghost
Copy link

@ghost ghost commented Jun 15, 2023

Adds a page containing result(s) of running end-to-end benchmarks in the CI. At the moment, it contains a single benchmark with a very simple content displayed:

Screenshot 2023-06-16 at 11 23 41

Future works might include:

  • Add more pages/sections for different scenarios
  • Include graphs displaying the execution profile of each benchmark

Fix #186


  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

@github-actions
Copy link

github-actions bot commented Jun 15, 2023

Test Results

312 tests   - 11   306 ✔️  - 11   19m 16s ⏱️ -29s
106 suites  -   4       6 💤 ±  0 
    5 files    -   1       0 ±  0 

Results for commit 8a2b785. ± Comparison against base commit cac437c.

This pull request removes 11 tests.
Hydra.TUI.Options ‑ parses --cardano-signing-key option
Hydra.TUI.Options ‑ parses --connect option
Hydra.TUI.Options ‑ parses --node-socket option
Hydra.TUI.Options ‑ parses --testnet-magic option
Hydra.TUI/end-to-end smoke tests ‑ display feedback long enough
Hydra.TUI/end-to-end smoke tests ‑ doesn't allow multiple initializations
Hydra.TUI/end-to-end smoke tests ‑ starts & renders
Hydra.TUI/end-to-end smoke tests ‑ supports the full Head life cycle
Hydra.TUI/end-to-end smoke tests ‑ supports the init & abort Head life cycle
Hydra.TUI/text rendering errors ‑ should show not enough fuel message and suggestion
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Jun 15, 2023

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2023-06-16 12:32:03.153124888 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial 2212a4ee618434b9b2f366d7c330dbdfb5c7072e793a850fd0de6ddd 4294
νCommit 69e1ccf9ad73dc6d37a5bc8de5aec86f3c4c1710fe5fd334e0e16b18 2124
νHead 8ae095dca4d14a1b8edffb37faa6c84ec60340fbf389a62f027e0b76 9355
μHead 33642a45c7fbb955ce1704ef09229bb211bf9af9980530db28c6aafe* 4148
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 4741 12.09 4.71 0.49
2 4947 15.07 5.86 0.53
3 5155 18.99 7.39 0.58
5 5561 23.24 8.98 0.65
10 6587 33.25 12.69 0.80
37 12124 99.36 37.70 1.76

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 599 14.89 5.70 0.34
2 784 19.66 7.73 0.40
3 972 24.75 9.88 0.46
5 1345 36.07 14.56 0.61
10 2284 71.73 28.85 1.04
13 2848 98.03 39.15 1.35

Cost of CollectCom Transaction

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 814 27.63 10.73 0.49
2 114 1133 43.76 17.13 0.68
3 171 1455 61.49 24.22 0.89
4 226 1782 82.31 32.57 1.13

Cost of Close Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 635 18.58 8.31 0.39
2 804 19.97 9.56 0.42
3 969 21.67 10.94 0.45
5 1299 24.46 13.45 0.50
10 2120 31.12 19.62 0.64
50 8725 87.17 69.99 1.74

Cost of Contest Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 709 25.22 10.98 0.46
2 840 26.49 12.01 0.49
3 1005 28.20 13.38 0.52
5 1337 31.63 16.12 0.58
10 2162 39.77 22.80 0.73
45 7928 99.74 70.74 1.81

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 4714 19.19 7.79 0.57
2 5176 36.68 15.56 0.79
3 5360 48.96 20.72 0.93
4 5817 73.47 31.59 1.23
5 6138 96.51 41.64 1.50

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
5 0 0 4769 8.66 3.57 0.46
5 1 56 4800 10.06 4.39 0.47
5 5 284 4944 15.64 7.69 0.55
5 10 568 5123 22.61 11.82 0.64
5 20 1138 5479 36.56 20.07 0.83
5 30 1710 5846 50.52 28.33 1.02
5 40 2273 6200 64.49 36.60 1.21
5 50 2845 6562 78.46 44.87 1.40
5 65 3699 7096 99.42 57.28 1.68

@ghost ghost marked this pull request as ready for review June 16, 2023 09:27
@ghost ghost requested review from ch1bo, ffakenz, v0d1ch and pgrange and removed request for ffakenz June 16, 2023 09:27
Copy link
Collaborator

@ch1bo ch1bo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the avoidable unsafePerformIO makes me not approve this right away. Also had some other suggestions.

Result looks great

hydra-cluster/bench/Bench/EndToEnd.hs Outdated Show resolved Hide resolved
hydra-cluster/bench/Bench/EndToEnd.hs Outdated Show resolved Hide resolved
hydra-cluster/bench/Bench/EndToEnd.hs Outdated Show resolved Hide resolved
hydra-cluster/bench/Main.hs Outdated Show resolved Hide resolved
hydra-cluster/bench/Main.hs Outdated Show resolved Hide resolved
hydra-cluster/src/CardanoNode.hs Show resolved Hide resolved
hydra-cluster/bench/Main.hs Outdated Show resolved Hide resolved
@ghost ghost force-pushed the abailly-iohk/add-ete-benchmarks-to-doc branch from d3ce45d to f70e6da Compare June 16, 2023 12:10
@ghost ghost requested a review from ch1bo June 16, 2023 12:10
@ghost ghost force-pushed the abailly-iohk/add-ete-benchmarks-to-doc branch from f70e6da to 8a2b785 Compare June 16, 2023 12:23
@ghost
Copy link
Author

ghost commented Jun 16, 2023

When the benchmark fails we get some info on the source location:

Starting benchmark
Seeding network
Logfile written to: /tmp/bench-cf4d443a06740abf/logs/cardano-node.log
Benchmark failed at bench/Bench/EndToEnd.hs:90:7 in main:Bench.EndToEnd: Test timed out after 1s seconds
To re-run with same dataset, pass '--work-directory=/tmp/bench-cf4d443a06740abf' to the executable
Benchmark bench-e2e: ERROR

hydra-cluster/src/CardanoNode.hs Show resolved Hide resolved
@ch1bo ch1bo merged commit 4a27e4c into master Jun 19, 2023
@ch1bo ch1bo deleted the abailly-iohk/add-ete-benchmarks-to-doc branch June 19, 2023 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Benchmark performance of a Hydra Head
2 participants