The TPS of Quorum

[bigstar119]

Hi, I want to know How many transactions do Quorum accomplish in a second?

I build a 3 node test environment, the device info is
CPU: 2 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Mem: 4GB
Disk:80GB
in my test , TOP TPS is 137 , is the result reasonable? and do you have any methd to increase the TPS,eg ... change some parameter or so ?

[patrickmn]

Operations are, in the majority of cases, I/O (disk)-bound, and in some cases CPU-bound.

137 sounds pretty low. Can you please share details about how you're benchmarking (are you sending private or public transactions, what are the contents of your transactions, etc.), how fast the harddrive is (e.g. sudo hdparm -t /dev/sda) and where the nodes are located geographically in relation to each other?

[sitano]

@bigstar119 publish your bench plz

[bigstar119]

hi .thank you for answer.. i deploy 3 Quorum nodes ,use raft consensus.
3 nodes are in 100M Ethernet ,net work latency is low. the contract is a Virtual bank, the tx is saveingMoney to an account. tx is private transactions from node1 to node3. i send 1000 transactions in 1000 thread from 10 Ethereum Account and i managed the nonce myself use a memory based increaser, each Account sendTransaction complete then send next Transaction, so each Ethereum Account have nearly 100 txs in final .because of upstream geth issue i must send tx one by one.
at last i got 137 transactions per sencond ..i also feel this result is pretty low,but i do not know why.

2017-05-22 next Monday I will upload the Detail resource include:
I/O (disk)-bound use sudo hdparm -t /dev/sda
Quorum nodes deploy scripts
testcase projects use java web3j-Qrorum
and so on

[bigstar119]

is anyone ever do the same test like me ? what is your test result ? How many transactions per second in you testcase? can your share something detail to me ? thank you .

[sitano]

@bigstar119 I doing some tests currently. Will share in a while as soon as will get any results.

[patrickmn]

@bigstar119 so you're aware, we added another API endpoint called sendTransactionAsync which does not return the transactionhash immediately (but can push to a callback URL with it): https://github.com/jpmorganchase/quorum/releases/tag/v1.0.2

This should significantly increase performance for sequential sending of private transactions (i.e. for each thread) since it does not wait for the recipient Constellation nodes to acknowledge receipt before returning. Barring CPU/thread starvation, it should increase your number significantly.

[sitano]

I just ran a test based on Public transactions of Wei.
The same default configuration as in 7nodes examples.

7 nodes on 7 AWS spot instances c4.4xlarge (16 cpu cores per instance, compute opimized) in the same network group, optimized IO (up to 3K iops),
single block maker 3-5 seconds.

I set the separate machine and send transaction sequentially to the node 1 which reuses (unlock 0) default account. I use eth.sendTransaction call. Its fine. Async only helps for private txs. So, my load test tries to push load evenly, scheduling pushes through time (without coordinated omission correction, but fine though).

Results: even 100 tx/s (from 1 account to new accounts) loads cluster with a CPU spikes from 30% to 150-300% on each node every committed block. txpool.status reports pending txs spikes and node 1 breaks after a certain amount of time due to the overload of the tx pool, facing a pending tx limit of 4096, while a long cpu spike on a single core.

It can eat more with decreased blockmaker window, but not much.

[sitano]

PProf shows just a big spent in applyTransaction, keccak and update of internal state.

quorum-geth-tree-cum.txt
quorum-geth-tree-flat.txt

[patrickmn]

Unfortunately, each vcpu on EC2 is fairly slow, and geth benefits much more from faster clock speeds than more cores because of its sequential nature. (Constellation does parallelize its operations, so if you're sending asynchronous private transactions, you should benefit somewhat from having more cores.)

It's definitely worth looking more at performance on many-slow-core setups, but it may ultimately be something that has to be addressed in geth itself.

[sitano]

@patrickmn totally agree. was just interested in theoretical perf cap avg. btw, 60% of CPU wasted on RLP encoding (rlp.EncodeToBytes) and Trie (Patricia Merkle Tree) updates.

[bigstar119]

@patrickmn
@sitano
hi, because Quorum do not provide sendRawTransaction method now . in other word rlpEncode can only done at the geth node ,the 60% of CPU costed can not be avoided.. is means that if quorum provide sendRawTransaction method ,and rlpEncode in the App，not on geth node ,the TPS will increase ?
then when will Quorum provide sendRawTransaction method?

[bigstar119]

@patrickmn
@sitano
hi,do you have the official Data about The TPS of Quorum? and the hardware、software configuration?

[conor10]

@patrickmn in the release notes it states that eth_sendTransactionAsync is temporary. Once raw transaction signing is supported, do you realistically envisage getting rid of it?

I'm thinking out loud if it may be worthwhile adding support for it in web3j-quorum.

[patrickmn]

@bigstar119 Yes, we do plan to support sendRawTransaction. The tricky part is that it's the identifier returned from Constellation that needs to be signed, not the original payload. Hopefully we will have an easy approach for this in the next release.

@conor10 Yes, eventually it will be deprecated in favor of something that doesn't force you to feed the plaintext of your transaction to the geth node just to have it feed it to Constellation to get the identifier. Our plan is to make libraries available that call out to Constellation or geth respectively only when necessary, in the same vein as what's happening in upstream Ethereum.

[LuoZijun]

@bigstar119

this problem is about safe to be concurrently.
in ethereum go client go-ethereum, that's not safe, so many transcation maybe got one same nonce.

the problem link: ethereum/go-ethereum#2950

tips:
use method sendRawTransaction or sendTransaction method with nonce option.

by the way, in ethereum rust client parity , that's problem shouldn't happen. All methods in RPC are safe to be called concurrently.

[sitano]

if you need to send concurrent requests with a single originating address, you can emit wallet nonce manually. largely, it's not a client issue.

[karalabe]

There were some concurrent nonce handling issues in go-ethereum, but they were fixed a few releases ago. Please update Quorum accordingly if it did not pull in those updates.

[LuoZijun]

@karalabe this commit is about that fixed , ethereum/go-ethereum@ea11f7d right ?

[jpmsam]

Hi @karalabe, thanks for the follow up! We are in the process of upgrading to 1.6 which will include this and multiple other fixes and separately, @bas-vk is working on a PR that will include the nonce fix.

[kissshan]

@bigstar119, @patrickmn ,@sitano
on my local machine i got 55 TPS only please give some instruction to improve my TPS by 100 or more. anyone have any document about TPS improvement ??

[Ryanmtate]

@sitano -- Based on your results, it does not seem reasonable to expect 100 TPS in production.

Results: even 100 tx/s (from 1 account to new accounts) loads cluster with a CPU spikes from 30% to 150-300% on each node every committed block. txpool.status reports pending txs spikes and node 1 breaks after a certain amount of time due to the overload of the tx pool, facing a pending tx limit of 4096, while a long cpu spike on a single core.

I really think your results represent a good stress test of the capabilities of the system, using AWS spot instances with plenty of power.

@patrickmn @jpmsam @tylobban

What do you think a better, more reasonable expectation is for TPS using quorum as it is today?

As a simple benchmark, I tested against sending 1500 transactions each from 3 separate accounts to 3 nodes running on one VM instance-- the 7node example deployment in azure. Each transaction process was run on a separate thread and the http RPC calls were made in parallel with a manually incrementing nonce for each account passed into the send object.

I averaged ~12.5 tx/s for 4500 total transactions per benchmark.

I've pasted the scripts I wrote this morning to run a crude benchmark. I welcome any thoughts or improvements to the benchmarking methods.

Many thanks--

// manager.js used to fork the processes and run on each thread

const Promise = require('bluebird')
const fork = require('child_process').fork


const workers = {
  0: fork('./sendTransaction.js'),
  1: fork('./sendTransaction.js'),
  2: fork('./sendTransaction.js')
}

const accounts = [{
    account: "REPLACE WITH THE SENDING ACCOUNT",
    rpc: "http://0.0.0.0:22000" // NOTE: actual IPs used were remote hosts
  }, {
    account: "REPLACE WITH THE SENDING ACCOUNT",
    rpc: "http://0.0.0.0:22001"
  }, {
    account: "REPLACE WITH THE SENDING ACCOUNT",
    rpc: "http://0.0.0.0:22003"
  }]

function sendWork({ worker, fromAccount, rpc }) {
  return new Promise((resolve, reject) => {
    try {
      workers[worker].on('message', (msg) => {
        return resolve(true)
      });

      workers[worker].on('error', (error) => {
        console.log('error', error)
      });

      workers[worker].send({ fromAccount, rpc });
    } catch (error) {
      return reject(error)
    }
  })
}


Promise.resolve(accounts).map((data, i) => {
  const { account, rpc } = data
  console.time('sendTransaction-benchmark')
  return sendWork({
    worker: i,
    fromAccount: account,
    rpc
  })
}).then(() => {
  console.timeEnd('sendTransaction-benchmark')
  console.log('finished')
  process.exit(0)
}).catch((error) => {
  console.log(error)
  process.exit(1)
})

// sendTransaction.js script used to process the transactions

const Web3 = require('web3')
const Promise = require('bluebird')

const NUM_TX_PER_ACCOUNT = 500

// Receiving Accounts
const RECEIVING_ACCOUNTS = [
  "0x8520de2650e91063882a2f45dD16CC80224BC451",
  "0x7BdD73C8A76B67B2E78dDE46BD18468341063Ed5",
  "0xAf2554cBd0f1E6ecEd368A206a8714bdE162327a",
]

function benchSendTransaction({ web3, from, to, nonce, counter }) {
  return new Promise((resolve, reject) => {
    if (counter == 0) {
      return resolve(true)
    } else {
      // console.time(`eth.sendTransaction, nonce: ${nonce}`)
      web3.eth.sendTransaction({ from, to, value: 1e2, nonce })
        // .once('transactionHash', (txHash) => { console.log('txHash', txHash) })
        // .once('receipt', (receipt) => { console.log('receipt', receipt) })
        // .on('confirmation', (confirmation, receipt) => { console.log('confirmation, receipt', confirmation, receipt) })
        .on('error', (error) => { return reject(error) })
        .then((receipt) => {
          // console.timeEnd(`eth.sendTransaction, nonce: ${nonce}`)
          return resolve(benchSendTransaction({
            web3,
            from,
            to,
            nonce: nonce += 1,
            counter: counter -= 1
          }))
        })
    }
  })
}

process.on('message', (msg) => {
  const { fromAccount, rpc} = msg
  const web3 = new Web3(rpc);

  let nonce;

  Promise.resolve(web3.eth.getTransactionCount(fromAccount)).then((count) => {
    nonce = count
    return RECEIVING_ACCOUNTS
  }).map((toAccount) => {
    console.time('eth.sendTransaction')
    return benchSendTransaction({
      web3,
      from: fromAccount,
      to: toAccount,
      nonce,
      counter: NUM_TX_PER_ACCOUNT
    })
  }).then((finished) => {
    console.timeEnd('eth.sendTransaction')
    process.send({ finished: true });
  }).catch((error) => {
    console.log('error', error)
    process.send({ error: true });
  })

});

[jpmsam]

Hi @Ryanmtate!
I couldn't get web3.eth.sendTransaction to send transactions fast enough to make transactions available in the mempool to process. I'd suggest a load testing tool and confirming that transactions are available in the pending state when blocks are created. If transactions are only available in the mempool at 12tx/sec, you won't see a higher throughput.

[Ryanmtate]

Thanks for the quick response @jpmsam!

I can confirm with txpool.status that I have transactions pending.

{
  pending: 3,
  queued: 0
}

Polling the txpool, the worst I've seen is 5 pending transactions so far.

I'll look into use a load testing tool for better results.

[rtmpnewbie]

so, how to do a TPS benchmark? @bigstar119 , thanks!

[vasa-develop]

Here is a great walkthrough for conducting your own TPS benchmark: https://hackernoon.com/quorum-stress-test-1-140-tps-792f39d0b43f