NEWS 2019-Feb-23 - video!
Explaining software v55 video released on youtube:
--> watch
TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.
The brand new release v55 is now presented & explained in a useful video on youtube.
hammer/- submits many transactions, while watching the recent blocksreader/- reads blocks; visualizes TPS, blocktime, gas, bytes - see reader/README.mddocs/- see esp. reproduce.md, cloud.md, FAQ.md, new: azure.mdresults/- for each client one markdown file;results/runs/- auto-generated pageslogs/- check this first if problemsnetworks/- network starters & external repos via install script, see belowscripts/- installers and other iseful bash scriptsenv/- Python virtualenv, created via install script, see belowtests/- start whole integration test suite via./pytest.sh
See the results/ folder:
- log.md: initial steps; also tried Quorum's private transactions
- quorum.md: raft consensus, quorum is a geth fork
- tobalaba.md: parity fork of EnergyWebFoundation
- quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
- geth.md: geth clique PoA algorithm
- parity.md: parity aura PoA algorithm, many attempts to accelerate
- eos.md: not begun yet
- substrate.md: not begun yet
Outdated table in which I had run each of the experiments manually in autumn 2018; soon re-done completely, using the below automation. So please contact me now, if you know how to accelerate any of these clients:
| hardware | node type | #nodes | config | peak TPS_av | final TPS_av |
|---|---|---|---|---|---|
| t2.micro | parity aura | 4 | (D) | 45.5 | 44.3 |
| t2.large | parity aura | 4 | (D) | 53.5 | 52.9 |
| t2.xlarge | parity aura | 4 | (J) | 57.1 | 56.4 |
| t2.2xlarge | parity aura | 4 | (D) | 57.6 | 57.6 |
| t2.micro | parity instantseal | 1 | (G) | 42.3 | 42.3 |
| t2.xlarge | parity instantseal | 1 | (J) | 48.1 | 48.1 |
| t2.2xlarge | geth clique | 3+1 +2 | (B) | 421.6 | 400.0 |
| t2.xlarge | geth clique | 3+1 +2 | (B) | 386.1 | 321.5 |
| t2.xlarge | geth clique | 3+1 | (K) | 372.6 | 325.3 |
| t2.large | geth clique | 3+1 +2 | (B) | 170.7 | 169.4 |
| t2.small | geth clique | 3+1 +2 | (B) | 96.8 | 96.5 |
| t2.micro | geth clique | 3+1 | (H) | 124.3 | 122.4 |
| t2.micro SWAP | quorum crux IBFT | 4 | (I) SWAP! | 98.1 | 98.1 |
| t2.micro | quorum crux IBFT | 4 | (F) | lack of RAM | |
| t2.large | quorum crux IBFT | 4 | (F) | 207.7 | 199.9 |
| t2.xlarge | quorum crux IBFT | 4 | (F) | 439.5 | 395.7 |
| t2.xlarge | quorum crux IBFT | 4 | (L) | 389.1 | 338.9 |
| t2.2xlarge | quorum crux IBFT | 4 | (F) | 435.4 | 423.1 |
| c5.4xlarge | quorum crux IBFT | 4 | (F) test_getNearestEntry() | 536.4 | 524.3 |
Reproduce these results easily; for the config column also see there.
Quickest reproduction with my Amazon AMI readymade image.
And see that bottom of parity.md and geth.md
and quorum-IBFT.md for the latest runs, issues, and additional details.
- how I initially got this faster, on Quorum, step by step, please do read the 1st logbook log.md
- then I improved per client, see each in #chronology above
- (possible TODOs - any other ideas?)
but not much more needed = the current version is already fully automated. Use it! May it help you to improve the speed of your Ethereum client!
Add yourself to other-projects.md using chainhammer, or projects which are similar to this.
(Especially if you work in one of the dev teams, you know your client code best - ) please try to improve the above results, e.g. by varying the CLI arguments with which the nodes are started; I don't see that as my job, you will be much more successful with that.
See parity PE#9393, parity SE#58521, geth GE#17447, quorum Q#479.
Please report back when you have done other / new measurements.
All this is developed and much tested on Debian, locally and in the AWS cloud. New: Ubuntu now also supported, see below.
N.B.: Better do this on a disposable cloud, or virtualbox machine; because the installation makes lasting changes and needs sudo!
After unpacking a ZIP of the downloaded repo, or by
git clone https://github.com/drandreaskrueger/chainhammer drandreaskrueger_chainhammer
ln -s drandreaskrueger_chainhammer CH
cd CH
you now only need these two lines to prepare and run the 1st experiment!
scripts/install.sh
CH_TXS=1000 CH_THREADING="sequential" ./run.sh $HOSTNAME-TestRPC testrpc
You will then have a diagram, and a HTML and MD page about this run!
(on Ubuntu instead: scripts/install.sh docker ubuntu )
Better now logout & login, or close the terminal, and open a new terminal, because the above scripts/install.sh might have enabled docker for the the first time for this user. Then:
For the full integration test, run each client for a short moment:
export CH_MACHINE=yourChoice
./run-all_small.sh
For detailed instructions, please see docs/, esp. reproduce.md, and for troubleshooting FAQ.md and github issues.
Chainhammer can now be stripped down to its pure benchmarking abilities, i.e. without the installation of docker and without the three local network starters (parity-deploy, geth-dev, quorum-crux). It was successfully used to benchmark the Microsoft Azure blockchain-as-a-service product. The essential difference is to start the installation with the switch nodocker:
scripts/install.sh nodocker
So, if you just want to benchmark your existing Ethereum node or network, have a look at the manual docs/azure.md .
./pytest.sh
enables the virtualenv,
then starts a testrpc-py Ethereum simulator on http://localhost:8545 in the background,
logging into tests/logs/;
then runs ./deploy.py andtests;
and finally runs all the unittests, also logging into tests/logs/.
(Instead of testrpc-py) if you want to run tests with another node,
just start that; and run pytest manually:
source env/bin/activate
py.test -v --cov
There were 98 tests on January 23rd, all 98 PASSED
(see this logfile -->
cat tests/logs/*.ansi because colors) on these different Ethereum providers:
- testrpc instantseal (
testrpc-py) 13 seconds - geth Clique (
geth-dev) 63 seconds - quorum IBFT (
blk-io/crux) 59 seconds - parity instantseal (
parity-deploy) 8 seconds - parity aura (
parity-deploy) 72 seconds
Please credit this as:
benchmarking scripts "chainhammer"
maintainer: Dr Andreas Krueger 2018-2020
https://github.com/drandreaskrueger/chainhammer
Consider to submit your improvements & usage as pull request. Thanks.
v01-v35 financed by Electron.org.uk 2018
v40-v55 financed by Web3Foundation 2018-2019
v58-v59 financed by Microsoft Azure 2019
Thank you very much!
The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer
# The following diagrams are outdated! Just make your own, new ones, with:
CH_MACHINE=yourChoice ./run-all_large.sh
examples:
geth.md = geth (go ethereum client), "Clique" consensus.
50,000 transactions to an Amazon t2.xlarge machine.
Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.
reader/img/geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png
quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)
OLD RUN on a desktop machine.
quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second)
OLD RUN on a desktop machine.
tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.
parity.md#run-18 = using parity-deploy.sh dockerized network of 4 local nodes with increased gasLimit, and 5 seconds blocktime; 20k transactions; ~ 60 TPS on an Amazon t2.xlarge machine.
N.B.: Could not work with parity v2 yet because of bugs PD#76 and PE#9582 --> everything still on parity v1.11.11
parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png
Calling all parity experts: How to improve these too slow TPS results?
See issue PE#9393, and the detailed log of what I've tried already, and the 2 shortest routes to reproducing the results: reproduce.md.
Thanks.