Skip to content

fire many transactions at Ethereum node, then produce diagrams of TPS, blocktime, gasUsed and gasLimit, and blocksize.

License

Notifications You must be signed in to change notification settings

drandreaskrueger/chainhammer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NEWS 2019-Feb-23 - video!
Explaining software v55 video released on youtube:

--> watch

chainhammer-logo.png


chainhammer v59

TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.

instructions

video

The brand new release v55 is now presented & explained in a useful video on youtube.

folders

  • hammer/ - submits many transactions, while watching the recent blocks
  • reader/ - reads blocks; visualizes TPS, blocktime, gas, bytes - see reader/README.md
  • docs/ - see esp. reproduce.md, cloud.md, FAQ.md, new: azure.md
  • results/ - for each client one markdown file; results/runs/ - auto-generated pages
  • logs/ - check this first if problems
  • networks/ - network starters & external repos via install script, see below
  • scripts/ - installers and other iseful bash scripts
  • env/ - Python virtualenv, created via install script, see below
  • tests/ - start whole integration test suite via ./pytest.sh

chronology

See the results/ folder:

  1. log.md: initial steps; also tried Quorum's private transactions
  2. quorum.md: raft consensus, quorum is a geth fork
  3. tobalaba.md: parity fork of EnergyWebFoundation
  4. quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
  5. geth.md: geth clique PoA algorithm
  6. parity.md: parity aura PoA algorithm, many attempts to accelerate
  7. eos.md: not begun yet
  8. substrate.md: not begun yet

results summary

Outdated table in which I had run each of the experiments manually in autumn 2018; soon re-done completely, using the below automation. So please contact me now, if you know how to accelerate any of these clients:

hardware node type #nodes config peak TPS_av final TPS_av
t2.micro parity aura 4 (D) 45.5 44.3
t2.large parity aura 4 (D) 53.5 52.9
t2.xlarge parity aura 4 (J) 57.1 56.4
t2.2xlarge parity aura 4 (D) 57.6 57.6
t2.micro parity instantseal 1 (G) 42.3 42.3
t2.xlarge parity instantseal 1 (J) 48.1 48.1
t2.2xlarge geth clique 3+1 +2 (B) 421.6 400.0
t2.xlarge geth clique 3+1 +2 (B) 386.1 321.5
t2.xlarge geth clique 3+1 (K) 372.6 325.3
t2.large geth clique 3+1 +2 (B) 170.7 169.4
t2.small geth clique 3+1 +2 (B) 96.8 96.5
t2.micro geth clique 3+1 (H) 124.3 122.4
t2.micro SWAP quorum crux IBFT 4 (I) SWAP! 98.1 98.1
t2.micro quorum crux IBFT 4 (F) lack of RAM
t2.large quorum crux IBFT 4 (F) 207.7 199.9
t2.xlarge quorum crux IBFT 4 (F) 439.5 395.7
t2.xlarge quorum crux IBFT 4 (L) 389.1 338.9
t2.2xlarge quorum crux IBFT 4 (F) 435.4 423.1
c5.4xlarge quorum crux IBFT 4 (F) test_getNearestEntry() 536.4 524.3

Reproduce these results easily; for the config column also see there. Quickest reproduction with my Amazon AMI readymade image. And see that bottom of parity.md and geth.md and quorum-IBFT.md for the latest runs, issues, and additional details.

faster wider more

  • how I initially got this faster, on Quorum, step by step, please do read the 1st logbook log.md
  • then I improved per client, see each in #chronology above
  • (possible TODOs - any other ideas?)

but not much more needed = the current version is already fully automated. Use it! May it help you to improve the speed of your Ethereum client!

you

Add yourself to other-projects.md using chainhammer, or projects which are similar to this.

(Especially if you work in one of the dev teams, you know your client code best - ) please try to improve the above results, e.g. by varying the CLI arguments with which the nodes are started; I don't see that as my job, you will be much more successful with that.

See parity PE#9393, parity SE#58521, geth GE#17447, quorum Q#479.

Please report back when you have done other / new measurements.

install and run

All this is developed and much tested on Debian, locally and in the AWS cloud. New: Ubuntu now also supported, see below.

quickstart

N.B.: Better do this on a disposable cloud, or virtualbox machine; because the installation makes lasting changes and needs sudo!

After unpacking a ZIP of the downloaded repo, or by

git clone https://github.com/drandreaskrueger/chainhammer drandreaskrueger_chainhammer
ln -s drandreaskrueger_chainhammer CH
cd CH

you now only need these two lines to prepare and run the 1st experiment!

scripts/install.sh
CH_TXS=1000 CH_THREADING="sequential" ./run.sh $HOSTNAME-TestRPC testrpc

You will then have a diagram, and a HTML and MD page about this run!

(on Ubuntu instead: scripts/install.sh docker ubuntu )

activate docker

Better now logout & login, or close the terminal, and open a new terminal, because the above scripts/install.sh might have enabled docker for the the first time for this user. Then:

All supported clients in one go:

For the full integration test, run each client for a short moment:

export CH_MACHINE=yourChoice
./run-all_small.sh

For detailed instructions, please see docs/, esp. reproduce.md, and for troubleshooting FAQ.md and github issues.

benchmarking a remote node

Chainhammer can now be stripped down to its pure benchmarking abilities, i.e. without the installation of docker and without the three local network starters (parity-deploy, geth-dev, quorum-crux). It was successfully used to benchmark the Microsoft Azure blockchain-as-a-service product. The essential difference is to start the installation with the switch nodocker:

scripts/install.sh nodocker

So, if you just want to benchmark your existing Ethereum node or network, have a look at the manual docs/azure.md .

unittests

./pytest.sh

enables the virtualenv, then starts a testrpc-py Ethereum simulator on http://localhost:8545 in the background, logging into tests/logs/; then runs ./deploy.py andtests; and finally runs all the unittests, also logging into tests/logs/.

(Instead of testrpc-py) if you want to run tests with another node, just start that; and run pytest manually:

source env/bin/activate
py.test -v --cov

There were 98 tests on January 23rd, all 98 PASSED (see this logfile --> cat tests/logs/*.ansi because colors) on these different Ethereum providers:

  • testrpc instantseal (testrpc-py) 13 seconds
  • geth Clique (geth-dev) 63 seconds
  • quorum IBFT (blk-io/crux) 59 seconds
  • parity instantseal (parity-deploy) 8 seconds
  • parity aura (parity-deploy) 72 seconds

credits

Please credit this as:

benchmarking scripts "chainhammer"
maintainer: Dr Andreas Krueger 2018-2020
https://github.com/drandreaskrueger/chainhammer

Consider to submit your improvements & usage as pull request. Thanks.

development was supported by

v01-v35 financed by Electron.org.uk 2018
v40-v55 financed by Web3Foundation 2018-2019
v58-v59 financed by Microsoft Azure 2019

logo

Thank you very much!

short summary

The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer




# The following diagrams are outdated! Just make your own, new ones, with:
CH_MACHINE=yourChoice ./run-all_large.sh

chainhammer: hammer --> reader --> diagrams

examples:

geth clique on AWS t2.xlarge

geth.md = geth (go ethereum client), "Clique" consensus.

50,000 transactions to an Amazon t2.xlarge machine.

Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.

geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png
reader/img/geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png

quorum IBFT on AWS t2.xlarge

quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)

quorum-crux-IBFT_t2xlarge_tps-bt-bs-gas_blks320-395.png

quorum raft

OLD RUN on a desktop machine.

quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second) reader/img/quorum_tps-bt-bs-gas_blks242-357.png

tobalaba

OLD RUN on a desktop machine.

tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.

reader/img/tobalaba_tps-bt-bs-gas_blks5173630-5173671.png

parity aura v1.11.11 on AWS t2.xlarge

parity.md#run-18 = using parity-deploy.sh dockerized network of 4 local nodes with increased gasLimit, and 5 seconds blocktime; 20k transactions; ~ 60 TPS on an Amazon t2.xlarge machine.

N.B.: Could not work with parity v2 yet because of bugs PD#76 and PE#9582 --> everything still on parity v1.11.11

parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png
parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png

Calling all parity experts: How to improve these too slow TPS results?

See issue PE#9393, and the detailed log of what I've tried already, and the 2 shortest routes to reproducing the results: reproduce.md.

Thanks.

About

fire many transactions at Ethereum node, then produce diagrams of TPS, blocktime, gasUsed and gasLimit, and blocksize.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published