~90 tps?

[drandreaskrueger]

I see a bit more than ...

90 TPS

in my TPS measurement: https://gitlab.com/electronDLT/chainhammer with:

• raft
• 7nodes example
• only speaking to node 1
• python, 1000 threads

What else should I try now? How to accelerate this?

Thanks.

System information

geth version
Geth
Version: 1.7.2-stable
Git Commit: ee498061b5a74bf1f3290139a53840345fa038cb
Quorum Version: 2.0.1
Architecture: amd64
Network Id: 1
Go Version: go1.9.3
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

[drandreaskrueger]

yes, the 90tps were averaged over 1000 transactions; will additionally measure intermediate peak rates - next time, probably Thursday

[fixanoid]

@drandreaskrueger can you also post specs of the hardware you tested it on? Also, assuming the testing was in the vagrant?

[jpmsam]

Hi @drandreaskrueger good start and thanks for sharing the code. You can achieve significantly higher TPS:

1. Not running in vagrant
2. Sending the transaction through RPC, you should bypass web3
3. If possible do an async transaction submission through eth_sendTransactionAsync.

Please share your numbers once done. Depending on the environment, permissioned vs public, you will be running in, there are other client optimizations that can improve your throughput.

[drandreaskrueger]

Nice one, thanks.

Yes, I want to keep trying, if time allows. Possibly continue on Thursday.

The 90 tps were only a first average over 1000 transactions, so peak values were higher. I will measure them too, i.e. moving average with short window sizes.

specs of the hardware you tested it on?

Will do, when I am back in the office. It is a pretty standard late 2016 desktop machine, with SSD, and 16GB RAM.

Not running in vagrant

I can try, however the beauty of the current setup is ... how quickly I could get it up and running.

For this test, could I perhaps simply add an 8th node to the existing "7nodes" example; which connects to the other 7 nodes, but lives outside the vagrant virtualmachine, i.e. runs on my host machine. What would be the full geth command to do that?

I suppose, I need to compile quorum then myself, what about #322?

Sending the transaction through RPC, you should bypass web3

This is using RPC, no?

You think that direct sending via e.g. curl would be considerably faster than via web3 ?

async transaction submission through eth_sendTransactionAsync.

Interesting, I will look into that.

there are other client optimizations that can improve your throughput.

Cool. What about starting a wikiwiki page about all those optimizations, that can be tried?

thanks for sharing the code.

feel free to run it yourself, and make the changes that you think might help to increase the TPS. Happy to accept pull requests. I say that because I am often not given enough time to try out everything, so I will probably have to simply drop this at some stage, when I have some early tentative ballpark numbers.

[drandreaskrueger]

plus, I want to also benchmark IBFT of course.

And perhaps also compare with the classical PoA, and PoW.

[jpmsam]

It shouldn't be complicated to build and configure Quorum if you want to do a bit more testing than what we provide in the examples vagrant setup.

You can reference the bootstrap script for how you can build and configure quorum directly

For load testing, I'd suggest something like Jmeter with direct RPC calls so that you don't have to worry about script optimizations.

I will close this as an issue but feel free to share your finding and follow up on slack.

[drandreaskrueger]

@jpmsam - Perfect, thanks.

Yes, those pointers are very helpful. Much appreciated.

I had been rushing through this, so some of the more obvious parts I had simply overlooked.

Especially the hint with quorum-examples/examples/7nodes/raft-start.sh - great!
I can simply adapt that script, to e.g. try out PoA for comparison, and also simple PoW.

At first I was wondering - how do the 7 nodes find each other?
But then I realized there is this: static-nodes.json = all done in raft-init.sh - clever.

Thanks.

Will keep this issue updated, when I will have worked on my repo.

[drandreaskrueger]

Did some more testing to understand how the different threading setups in Python are influencing the throughput: https://gitlab.com/electronDLT/chainhammer/blob/master/log.md

Have started to look into installing everything on hostmachine (instead of vagrant), along this script - but then the internet collapsed, and the work day was over. Next week, probably....

[drandreaskrueger]

@jpmsam I have tried to make your suggestion working

Not running in vagrant

but at the end of it ...

I am now getting a "runtime error: ... segmentation violation", see issue #352

[drandreaskrueger]

Hello @jpmsam, I think I have disproved this point:

Sending the transaction through RPC, you should bypass web3

at least for now.

Perhaps with enabled privateFor field that changes.
But honestly, I would not know why.

The whole time I had thought - why would you assume the web3 coders are so bad that they slow down TPS ?
But you suggested it, so I tried it out.

Please see my today's updates:

https://gitlab.com/electronDLT/chainhammer/blob/master/log.md#sending-via-web3-versus-sending-via-rpc

Thanks.

[jpmsam]

@drandreaskrueger web3 coders are actually very good and we use their code throughout our applications. Ideally, removing the extra translation hop from your transaction creation should have increased the throughput.

[drandreaskrueger]

thx a lot.

sorry for my tired & slightly bitter words up there - but I was actually disappointed that all that work today had not improved the TPS rate in any way, quite the opposite. And since your suggestion, I had actually been wondering the whole time why and how direct-RPC calls should be able to be faster than web3-calls.

what do you mean by translation hop? Thx.

What I have not yet used is eth_sendTransactionAsync.

[drandreaskrueger]

I had timed the encoding, see this test_argument_encoding() loop - and 10,000 of those took less than 0.5 seconds; so that cannot be it.

what do you mean by extra translation hop?

what in my code you @jpmsam (or anyone) are spotting ... which I could optimize?

Thx.

[jpmsam]

@drandreaskrueger No worries, thanks for making your scripts available for others to use.

"extra translation hop" I meant the extra step that was added to do the encoding of the methods and params whether done by web3 or your code. In your case, that doesn't seem to be the issue.

We'll look into the issue #352.

[drandreaskrueger]

Perfect, thanks!

[drandreaskrueger]

IBFT: 175 TPS

Also benchmarked the istanbul consensus now:

https://gitlab.com/electronDLT/chainhammer/blob/master/chainreader/img/quorum-istanbul_tps-bt-bs-gas_blks3680-3755.png

looks like I am hitting the hard wall of maximum gas limit per block at 175 TPS

... so it might be interesting to allow for a much larger gas limit, and then see how fast it can be pushed.

[drandreaskrueger]

raft: 230 - 240 TPS

Now benchmarked quorum raft once more, with 10,000 transactions

looks like above 200 TPS are possible easily. And that is then not even close to the block gas limit.

What is a bit odd: 3917 transactions are crammed into the one last block of these over 300 blocks.

For more details about this, see https://gitlab.com/electronDLT/chainhammer/blob/master/chainreader/blocksDB_analyze_quorum-raft.ipynb

[drandreaskrueger]

@jpmsam ...
We'll look into the issue #352.

yes, please.

[drandreaskrueger]

now when pushing the limits higher and higher, I do see big gains using your idea @jpmsam

Sending the transaction through RPC, you should bypass web3

have a look at this: chainhammer quorum-IBFT.md#direct-rpc-call-instead-of-web3-call

over 400 TPS, initially!

thanks a lot!

[jpmsam]

@drandreaskrueger great to hear and thanks for the update! We have also been using chainhammer in our testing.

[drandreaskrueger]

If possible do an async transaction submission through eth_sendTransactionAsync.

@jpmsam your eth_sendTransactionAsync is quorum specific? Not even in geth, right?

I suggested it to the parity guys --> PE#9471 but got the answer:

This would need to go through an EIP.

What about creating such an EIP? You are the experts on this functionality.