This repository contains the evaluation tool developed as part of the paper "Analyzing the Performance of the Inter-Blockchain Communication Protocol". This paper has been published at the 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'23).
A full version is available online: https://arxiv.org/abs/2303.10844
The dataset generated through the experiments discussed in the paper are available online: https://drive.google.com/drive/folders/1f3t4Qf_mC2atcgpafTHk4qv9vWesjfwQ?usp=sharing
- Go v1.17.5
- Cosmos Hub (Gaia) v7.0.2
- Rust compiler v1.6.0
- Hermes Relayer v0.15.0 (https://github.com/informalsystems/hermes)
Tip
Those prerequisites can be installed by running the install.sh script.
First, this script sets up two Cosmos Gaia blockchains with funded user accounts. Then, it uses ssh to run a set of validator nodes for each of them on remote machines. After starting the blockchains and synchronizing the nodes it establishes an unordered IBC channel between both blockchains using the Hermes Relayer.
Important
This file needs to be modified to include the address of the remote machines used for validator nodes. The prerequisites for the blockchains must be installed in all remote machines.
Usage:
./setup_chains.sh -n <NUMBER_OF_NODES> -a <FUNDED_ACCOUNTS> -t [TIMEOUT_COMMIT]
Options:
--nodes | -n; Number of consensus nodes to run for each chain.
--accounts | -a; Number of funded accounts initialized in the source chain's genesis.
--timeout-commit | -t; Minimum block interval in seconds (default: 5 seconds).
Example:
./setup_chains.sh -n 5 -a 10 -t 5
This script conducts a performance evaluation of cross-chain communication using the previously established IBC channel. The evaluation workload is composed of cross-chain fungible token transfers (https://github.com/cosmos/ibc/blob/main/spec/app/ics-020-fungible-token-transfer/README.md). This script must be executed in a machine together with one validator node for each blockchain. This is required to retrieve transaction data through RPC.
Usage:
./benchmark.sh -S <SRC_CHAIN_ADDR> -D <DST_CHAIN_ADDR> -u <NUM_USERS> -t <NUM_TRANSACTIONS> -m <NUM_MESSAGES> -o <OUTPUT_DIR>
Options:
-S | --source-addr; RPC address of the source chain.
-D | --destination-addr; RPC address of the destination chain.
-u | --users; Number of users to be used for submitting transactions.
-t | --transactions; Number of transactions submitted per user.
-m | --messages; Number of cross-chain transfer messages inside each transaction (max: 100).
-o | --output-dir; Directory in which to store benchmark working files.
-w | --wait-for-blocks; [Optional] Stop waiting for transactions to complete/timeout and start analyzing data after this many empty blocks have been produced in a row (default: 5).
--tx-timeout; [Optional] Specify how many new blocks can be created before a cross-chain transfer times out (default: 25).
--transaction-analysis; [Optional] Enables analysis of transaction and IBC message sizes (slower).
Note
The option --transaction-analysis is currently unavailable as it requires additional logic that increases the completion time of default benchmarking mode given the current code structure.
Example:
./benchmark.sh -S 'localhost:26657' -D 'localhost:36657' -u 10 -t 25 -m 20 -o 'benchmarking_test'
The tool generates a file called "benchmarking_report.txt" in the specified output directory. This file contains a performance report generated based on the execution of the specified workload.
[+] Benchmark configuration summary:
Source chain: blockchain0
Destination chain: blockchain1
Number of validators in each chain: 5
Number of user accounts: 10
Transactions submitted per user: 50
Total number of transactions: 500
Transfer messages per transaction: 100
Total number of transfer messages: 50000
IBC transfers submission (time elapsed): 5m 18s
Time waiting for empty blocks at the end of benchmark: 0m 1s
Blockchain data collection (time elapsed): 7m 46s
Total time elapsed: 13m 5s
----------------------------------------------------------------------
[+] Transaction distribution analysis for blockchain0:
0 tx(s): 1 block(s)
10 tx(s): 29 block(s)
19 tx(s): 1 block(s)
21 tx(s): 17 block(s)
23 tx(s): 1 block(s)
25 tx(s): 1 block(s)
26 tx(s): 1 block(s)
----------------------------------------------------------------------
[+] Transaction distribution analysis for blockchain1:
0 tx(s): 26 block(s)
2 tx(s): 1 block(s)
9 tx(s): 1 block(s)
11 tx(s): 20 block(s)
16 tx(s): 1 block(s)
17 tx(s): 1 block(s)
----------------------------------------------------------------------
[+] Throughput analysis for chain 'blockchain0':
Blocks finalized: 51
Avg. block time: 6.302 seconds
Number of empty blocks: 1 (1.96%)
Avg. number of txs per block (counting empty blocks): 14.51
Avg. number of txs per second(transfer, recv, ack): 2.35
Avg. number of messages per tx: 97.30
Avg. number of messages per block (counting empty blocks): 1411.76
Avg. number of messages per second (transfer, recv, ack, timeout): 228.50
Avg. number of transfers per second: 158.68
----------------------------------------------------------------------
[+] Throughput analysis for chain 'blockchain1':
Blocks finalized: 50
Avg. block time: 6.322 seconds
Number of empty blocks: 26 (52.00%)
Avg. number of txs per block (counting empty blocks): 5.28
Avg. number of txs per second(transfer, recv, ack): 0.85
Avg. number of messages per tx: 90.91
Avg. number of messages per block (counting empty blocks): 480.00
Avg. number of messages per second (transfer, recv, ack, timeout): 77.47
Avg. number of transfers per second: 0.00
----------------------------------------------------------------------
[+] Round trip time analysis for chains 'blockchain0 -> blockchain1':
Average round trip time: 178.920s
Shortest round trip time: 163.744s
Longest round trip time: 188.840s
----------------------------------------------------------------------
[+] Success rate analysis for channel 'blockchain0 -> blockchain1':
IBC transfers submitted to 'blockchain0': 50000
'Transfer' messages committed to 'blockchain0': 50000
'Receive' messages committed to 'blockchain1': 24000
'Acknowledgement' messages committed to 'blockchain0': 22000
'Timeout' messages committed' to 'blockchain0' (source chain): 0
Transfers completed (transfer, recv, ack): 22000 (44.00%)
Transfers partially completed (transfer, recv): 2000 (4.00%)
Transfers only initiated (transfer): 26000 (52.00%)
Transfers not initiated (submitted but not committed): 0 (0.00%)
Timed out transfers: 0 (0.00%)
----------------------------------------------------------------------
[+] IBC messages confirmation latency analysis for chains 'blockchain0 -> blockchain1':
Avg. transfer message confirmation latency: 6.311s
Shortest transfer latency observed: 5.131s
Longest transfer latency observed: 7.855s
Avg. recv message confirmation latency: 12.190s
Shortest recv message confirmation latency: 6.386s
Longest recv message confirmation latency: 90.453s
Avg. acknowledgement message confirmation latency: 8.431s
Shortest acknowledgement message confirmation latency: 6.208s
Longest acknowledgement message confirmation latency: 10.349s
----------------------------------------------------------------------
[+] Data analysis for chain 'blockchain0':
Collective size of all blocks (excl. empty blocks at the end): 37.24 MB
Avg. block size (excl. empty blocks at the end): 747.81 kB
Number of transactions committed to the blockchain: 740
Number of messages committed in the blockchain: 72000
Number of transactions containing transfer messages on blockchain0: 500
Number of transfer messages on blockchain0: 50000
Number of transactions containing recv messages on blockchain0: 0
Number of recv messages on blockchain0: 0
Number of transactions containing ack messages on blockchain0: 240
Number of ack messages on blockchain0: 22000
Number of transactions containing timeout messages on blockchain0: 0
Number of timeout messages on blockchain0: 0
----------------------------------------------------------------------
[+] Data analysis for chain 'blockchain1':
Collective size of all blocks (excl. empty blocks at the end): 30.07 MB
Avg. block size (excl. empty blocks at the end): 615.79 kB
Number of transactions committed to the blockchain: 264
Number of messages committed in the blockchain: 24000
Number of transactions containing transfer messages on blockchain1: 0
Number of transfer messages on blockchain1: 0
Number of transactions containing recv messages on blockchain1: 264
Number of recv messages on blockchain1: 24000
Number of transactions containing ack messages on blockchain1: 0
Number of ack messages on blockchain1: 0
Number of transactions containing timeout messages on blockchain1: 0
Number of timeout messages on blockchain1: 0
----------------------------------------------------------------------