Benchmark/performances for the executions

[antho1404]

When trying to fix #1537 I realized that with the current configuration we were only capable of processing 4-5 transactions per block.

This can come from 2 problems:

• we reach the block capacity, either with the size of the messages (which I doubt) or the number of gas spent (more likely)
• the processing/validation before adding in the pool of transaction takes a lot of time and we can only sign/broadcast ~4 transactions per seconds (current block time)

We have a huge bottleneck somewhere, I think because we try to send the transactions in a synchronous way. I have high doubts that this is just because of pure processing like hashing functions or so but it might be something to explore too.

---

I tried to increase the block time to 10 seconds and it seems that we can process about 20-25 transactions (with similar executions message) so it seems that the problem doesn't come from the capacity of the block.

[antho1404]

I run a bunch of tests to try to understand the bottleneck issue that we have

Tests

Run the process of this repository https://github.com/antho1404/application-erc20-analytics
You can monitor the metrics with this branch/PR #1580

Results

We always the same kind of results

• Execution created and pre-signed immediately
• Execution signed with a delay (blue curve)
• Execution in progress connected to tx signed until a certain point where it stops
• Execution completed caped by the executions in progress

It seems that there is a correlation between the transaction signed (blue) and the transaction completed (red) but that doesn’t impact the transaction in progress

The signature/broadcast of the transaction is clearly a bottleneck that needs to be fixed

[NicolasMahe]

Source code that should be the source of the problem for the slowness of signing / broadcasting:

engine/cosmos/client.go

Lines 84 to 128 in 47b743e

	// BuildAndBroadcastMsg builds and signs message and broadcast it to node.
	func (c *Client) BuildAndBroadcastMsg(msg sdktypes.Msg) (*abci.ResponseDeliverTx, error) {
	c.broadcastMutex.Lock() // Lock the whole signature + broadcast of the transaction
	signedTx, err := c.sign(msg)
	if err != nil {
	c.broadcastMutex.Unlock()
	return nil, err
	}
	txres, err := c.BroadcastTxSync(signedTx)
	c.broadcastMutex.Unlock()
	if err != nil {
	return nil, err
	}
	if txres.Code != abci.CodeTypeOK {
	return nil, fmt.Errorf("transaction returned with invalid code %d", txres.Code)
	}
	// TODO: 20*time.Second should not be hardcoded here
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	subscriber := "engine"
	query := tenderminttypes.EventQueryTxFor(signedTx).String()
	out, err := c.Subscribe(ctx, subscriber, query)
	if err != nil {
	return nil, err
	}
	defer c.Unsubscribe(ctx, subscriber, query)
	select {
	case result := <-out:
	data, ok := result.Data.(tenderminttypes.EventDataTx)
	if !ok {
	return nil, errors.New("result data is not the right type")
	}
	if data.TxResult.Result.IsErr() {
	return nil, fmt.Errorf("an error occurred in transaction: %s", data.TxResult.Result.Log)
	}
	return &data.TxResult.Result, nil
	case <-ctx.Done():
	return nil, errors.New("i/o timeout")
	}
	}

• TODO: implement the mutex in a much more clever way

Extracting the account and signature logic to a dedicated structure can only help. Started in PR #1573 because of issue #1565

Two attempts to create benchmark on the whole API:

• wip benchmark e2e #1621
• Benchmark #1622