From 07ac3d172ca63826326e6052b35b132bbc8a63fa Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Mon, 1 Jul 2024 13:36:28 -0500 Subject: [PATCH 1/7] initial --- docs/bridge/docs/Services/Submitter.md | 16 +++++++++------- .../Services/img/submitter/submitter_flow.svg | 3 +++ 2 files changed, 12 insertions(+), 7 deletions(-) create mode 100644 docs/bridge/docs/Services/img/submitter/submitter_flow.svg diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index daa6c447a3..0fbcfb69c7 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -8,9 +8,12 @@ This section is still in progress, please see [here](https://pkg.go.dev/github.c The Ethergo Submitter module is designed to submit transactions to an EVM-based blockchain. It handles gas bumping and confirmation checking to ensure that transactions are eventually confirmed. This module is essential because the EVM does not specify transaction submission or consensus, and rate limits can affect transaction submission. +![submitter flow](img/submitter/submitter_flow.svg) + ## Key Features -- **Transaction Submission**: The main function of the module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. +The module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. + - **Gas Bumping**: Automatically adjusts the gas price to ensure timely transaction confirmation. - **Confirmation Checking**: Continuously checks the status of submitted transactions to confirm their inclusion in the blockchain. - **Reaper Functionality**: Flushes old entries in the database that have reached a terminal state. @@ -27,12 +30,11 @@ This section is still in progress, please see [here](https://pkg.go.dev/github.c Submitter exposes metrics for Prometheus. The metrics are: - - `num_pending_txs`: The number of pending transactions. - - `current_nonce`: The current nonce. - - `oldest_pending_tx`: The age of the oldest pending transaction. - - `confirmed_queue`: The number of confirmed transactions. - - `gas_balance`: The current gas balance. - +- `num_pending_txs`: The number of pending transactions. +- `current_nonce`: The current nonce. +- `oldest_pending_tx`: The age of the oldest pending transaction. +- `confirmed_queue`: The number of confirmed transactions. +- `gas_balance`: The current gas balance. The metrics can be used in a dashboard [here](https://raw.githubusercontent.com/synapsecns/sanguine/master/ethergo/dashboard.json). It looks like this: diff --git a/docs/bridge/docs/Services/img/submitter/submitter_flow.svg b/docs/bridge/docs/Services/img/submitter/submitter_flow.svg new file mode 100644 index 0000000000..6ef2bd7eb7 --- /dev/null +++ b/docs/bridge/docs/Services/img/submitter/submitter_flow.svg @@ -0,0 +1,3 @@ + + +
Yes
No
Yes
No
External Fetches
DB
Increment Database Nonce
Get last used nonce in database
RPC
Get last nonce on-chain
Lock Nonce Mutex
Errored?
Unlock Nonce Mutex
Is on-chain nonce > database nonce?
Use database nonce
Store Association Between Database Nonce and Transaction
Use on-chain nonce
 \ No newline at end of file From f8ce0c2de45f553a2a5c25547cd96ea05d6411dd Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Mon, 1 Jul 2024 16:11:36 -0500 Subject: [PATCH 2/7] more --- docs/bridge/docs/Services/Submitter.md | 57 ++++++++++++++++++++++++-- ethergo/submitter/chain_queue.go | 3 +- 2 files changed, 56 insertions(+), 4 deletions(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index 0fbcfb69c7..3f4af7ef32 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -12,7 +12,7 @@ The Ethergo Submitter module is designed to submit transactions to an EVM-based ## Key Features -The module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. +The module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. The nonce may then be used in the `GetSubmissionStatus` method to check the state: `NotFound`, `Pending`, `Confirming`, or `Confirmed`. - **Gas Bumping**: Automatically adjusts the gas price to ensure timely transaction confirmation. - **Confirmation Checking**: Continuously checks the status of submitted transactions to confirm their inclusion in the blockchain. @@ -20,11 +20,62 @@ The module is the `SubmitTransaction` method, which returns a nonce and ensures ### Reaper -The submitter also has "reaper" functionality, which flushes old entries in the database that have reached a terminal state (`Replaced`, `ReplacedOrConfirmed`, `Confirmed`). By default, entries are flushed after a week, but this functionality is configurable by the `MaxRecordAge` config value. +The Submitter also has "reaper" functionality, which flushes old entries in the database that have reached a terminal state (`Replaced`, `ReplacedOrConfirmed`, `Confirmed`). By default, entries are flushed after a week, but this functionality is configurable by the `MaxRecordAge` config value. ### Submitter Config -This section is still in progress, please see [here](https://pkg.go.dev/github.com/synapsecns/sanguine/ethergo@v0.9.0/submitter/config) for details. +Config contains configuration for the Submitter. It can be loaded from a YAML file. +Chain-specific configuration items can be provided via the `Chains` map, which overrides the global config +for each chain. If a chain-specific item is not provided, the global config is used. + +#### Example config + +```yaml +submitter_config: + chains: + 1: + supports_eip_1559: true + gas_estimate: 1000000 + 42161: + gas_estimate: 30000000 + max_gas_price: 10000000000 + supports_eip_1559: true + 534352: + gas_estimate: 600000 + max_gas_price: 30000000000 + gas_bump_percentage: 20 + reaper_interval: 604800000000000 # int64(7 * 24 * time.Hour) + max_record_age: 86400000000000 # int64(1 * 24 * time.Hour) +``` + +Please see [here](https://pkg.go.dev/github.com/synapsecns/sanguine/ethergo@v0.9.0/submitter/config) for details on the configuration. + +### Overview + +`SubmitTransaction` abstracts many of the complexities of on-chain transaction submission such as nonce management and gas bumping. In addition, sent transactions are stored in the database for easy indexing of older transactions. + +#### Example of SubmitTransaction + +Below is an example of how to submit a transaction using Submitter. Note that the actual transaction submission logic takes place in the callback. We use an abigen binding here to send the transaction (`ReceiveMessage`), but any way to send a transaction also works, like our [Ethergo/EVM client](https://pkg.go.dev/github.com/synapsecns/sanguine/ethergo@v0.9.0/client) or geth's `ethclient`. + +```go +nonce, err := c.txSubmitter.SubmitTransaction( + ctx, + big.NewInt(int64(msg.DestChainID)), + func(transactor *bind.TransactOpts) (tx *types.Transaction, err error) { + tx, err = contract.ReceiveMessage( + transactor, + msg.Message, + msg.Attestation, + ) + if err != nil { + return nil, fmt.Errorf("could not submit transaction: %w", err) + } + + return tx, nil + }, +) +``` ### Observability diff --git a/ethergo/submitter/chain_queue.go b/ethergo/submitter/chain_queue.go index 635c061bb5..b0f53a657e 100644 --- a/ethergo/submitter/chain_queue.go +++ b/ethergo/submitter/chain_queue.go @@ -3,12 +3,13 @@ package submitter import ( "context" "fmt" - "github.com/ethereum/go-ethereum/params" "math/big" "sort" "sync" "time" + "github.com/ethereum/go-ethereum/params" + "github.com/synapsecns/sanguine/ethergo/util" "github.com/ethereum/go-ethereum/common" From 4d79bcfdc3c4bca39dfea14adf0fb447098d0ebe Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Mon, 1 Jul 2024 18:22:17 -0500 Subject: [PATCH 3/7] submitter docs --- docs/bridge/docs/Services/Submitter.md | 40 +++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index 3f4af7ef32..69ab03a481 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -12,7 +12,7 @@ The Ethergo Submitter module is designed to submit transactions to an EVM-based ## Key Features -The module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. The nonce may then be used in the `GetSubmissionStatus` method to check the state: `NotFound`, `Pending`, `Confirming`, or `Confirmed`. +The module is the `SubmitTransaction` method, which returns a nonce and ensures that the transaction will eventually be confirmed. The nonce may then be used in the `GetSubmissionStatus` method to check the state: `Pending`, `Stored`, `Submitted`, `FailedSubmit`, `ReplacedOrConfirmed`, `Replaced`, `Confirmed`. - **Gas Bumping**: Automatically adjusts the gas price to ensure timely transaction confirmation. - **Confirmation Checking**: Continuously checks the status of submitted transactions to confirm their inclusion in the blockchain. @@ -77,6 +77,44 @@ nonce, err := c.txSubmitter.SubmitTransaction( ) ``` +### Nonce Management, Database, Internals + +#### Nonce Management and Multichain + +Submitter was designed with multiple chains in mind by keeping track of a thread-safe `map[chainid]nonce`. When we build the transaction opts, we lock on the chainid until we finish firing off the transaction. +We also keep a txHash -> txStatus map with a similar, thread-safe mechanism. + +This allows for a concurrent nature where we're able to concurrently fire off transactions on different chains while ensuring our nonces are correct per chain. The [Chain Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) is the actual implementation of the queue. + +The Chain Queue db interface, [Service](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/service.go), allows a user to customize their transaction db behavior. The base implementation is in [store.go](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/txdb/store.go). + +#### Database + +When sending transactions, we are able to check on the status of them after we fire them off. The schema for a transaction is + +```go +// ETHTX contains a raw evm transaction that is unsigned. +type ETHTX struct { + ID uint64 `gorm:"column:id;primaryKey;autoIncrement:true"` + // UUID is a unique ID for this transaction that will persist across retries. + UUID string `gorm:"column:uuid;index"` + // CreatedAt is the time the transaction was created + CreatedAt time.Time + // TXHash is the hash of the transaction + TXHash string `gorm:"column:tx_hash;uniqueIndex;size:256"` + // From is the sender of the transaction + From string `gorm:"column:from;index"` + // ChainID is the chain id the transaction hash will be sent on + ChainID uint64 `gorm:"column:chain_id;index"` + // Nonce is the nonce of the raw evm tx + Nonce uint64 `gorm:"column:nonce;index"` + // RawTx is the raw serialized transaction + RawTx []byte `gorm:"column:raw_tx"` + // Status is the status of the transaction + Status db.Status `gorm:"column:status;index"` +} +``` + ### Observability Submitter exposes metrics for Prometheus. The metrics are: From e1c4eca5e283321b09c9f2bd34fc8b8aa2c4065b Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Tue, 2 Jul 2024 10:51:28 -0500 Subject: [PATCH 4/7] chain queue and queue --- docs/bridge/docs/Services/Submitter.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index 69ab03a481..d8a7915107 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -77,14 +77,18 @@ nonce, err := c.txSubmitter.SubmitTransaction( ) ``` -### Nonce Management, Database, Internals +### Architecture, Nonce Management, Database, Internals + +#### Architecture: Submitter, Chain Queue, Queue #### Nonce Management and Multichain Submitter was designed with multiple chains in mind by keeping track of a thread-safe `map[chainid]nonce`. When we build the transaction opts, we lock on the chainid until we finish firing off the transaction. We also keep a txHash -> txStatus map with a similar, thread-safe mechanism. -This allows for a concurrent nature where we're able to concurrently fire off transactions on different chains while ensuring our nonces are correct per chain. The [Chain Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) is the actual implementation of the queue. +This allows us to concurrently fire off transactions on different chains while ensuring our nonces are correct. The [Chain Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) is the actual implementation of the queue, while The [Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) actually handles the overall processing of the queue in the `processQueue` method. + +#### Service The Chain Queue db interface, [Service](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/service.go), allows a user to customize their transaction db behavior. The base implementation is in [store.go](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/txdb/store.go). From 48e3b1516e938e61dac9c9b0bcb813dba0aee582 Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Tue, 2 Jul 2024 11:20:10 -0500 Subject: [PATCH 5/7] docsk --- docs/bridge/docs/Services/Submitter.md | 30 ++++++++++++++++++-------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index d8a7915107..b5784abbc4 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -77,24 +77,23 @@ nonce, err := c.txSubmitter.SubmitTransaction( ) ``` -### Architecture, Nonce Management, Database, Internals - -#### Architecture: Submitter, Chain Queue, Queue +### Nonce Management, Database, Internals #### Nonce Management and Multichain Submitter was designed with multiple chains in mind by keeping track of a thread-safe `map[chainid]nonce`. When we build the transaction opts, we lock on the chainid until we finish firing off the transaction. -We also keep a txHash -> txStatus map with a similar, thread-safe mechanism. +We also keep a `map[txHash]txStatus` with the same thread-safe mechanism. +This allows us to concurrently fire off transactions on different chains while ensuring our nonces are correct. -This allows us to concurrently fire off transactions on different chains while ensuring our nonces are correct. The [Chain Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) is the actual implementation of the queue, while The [Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) actually handles the overall processing of the queue in the `processQueue` method. +The [Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) has a selector loop running at all times that calls the `processQueue` method, concurrently processing and storing confirmed txs, or using the [chain queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) to fire off and store pending txs on chain. -#### Service +#### Customizing DB Behavior -The Chain Queue db interface, [Service](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/service.go), allows a user to customize their transaction db behavior. The base implementation is in [store.go](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/txdb/store.go). +The Chain Queue db interface, [Service](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/service.go), allows a user to customize their Transaction DB behavior. The concrete implementation is in [store.go](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/db/txdb/store.go). -#### Database +#### Transaction DB -When sending transactions, we are able to check on the status of them after we fire them off. The schema for a transaction is +The schema for a transaction to be stored in the Transaction DB is: ```go // ETHTX contains a raw evm transaction that is unsigned. @@ -119,6 +118,19 @@ type ETHTX struct { } ``` +Using [GORM.db](https://pkg.go.dev/gorm.io/gorm), you can use whatever database you'd like, MySQL, Sqlite, etc. + +#### MySQL Example + +```go +gdb, err := gorm.Open(mysql.Open(dbURL), &gorm.Config{ + Logger: common_base.GetGormLogger(logger), + FullSaveAssociations: true, + NamingStrategy: NamingStrategy, + NowFunc: time.Now, +}) +``` + ### Observability Submitter exposes metrics for Prometheus. The metrics are: From de7062c4a85f51c3f15e533d5c413570f1b588b7 Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Tue, 2 Jul 2024 11:21:34 -0500 Subject: [PATCH 6/7] change nunbers --- docs/bridge/docs/Services/Submitter.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index b5784abbc4..ebb9d66b76 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -36,13 +36,13 @@ submitter_config: 1: supports_eip_1559: true gas_estimate: 1000000 - 42161: + 43114: gas_estimate: 30000000 - max_gas_price: 10000000000 + max_gas_price: 100000000000 supports_eip_1559: true - 534352: - gas_estimate: 600000 - max_gas_price: 30000000000 + 10: + gas_estimate: 400000 + max_gas_price: 90000000000 gas_bump_percentage: 20 reaper_interval: 604800000000000 # int64(7 * 24 * time.Hour) max_record_age: 86400000000000 # int64(1 * 24 * time.Hour) From c8462de95c425160ad0bdfbf9f2eac8582c4b041 Mon Sep 17 00:00:00 2001 From: golangisfun123 Date: Tue, 2 Jul 2024 11:35:53 -0500 Subject: [PATCH 7/7] typoes --- docs/bridge/docs/Services/Submitter.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/bridge/docs/Services/Submitter.md b/docs/bridge/docs/Services/Submitter.md index ebb9d66b76..8084d5a8bf 100644 --- a/docs/bridge/docs/Services/Submitter.md +++ b/docs/bridge/docs/Services/Submitter.md @@ -84,8 +84,7 @@ nonce, err := c.txSubmitter.SubmitTransaction( Submitter was designed with multiple chains in mind by keeping track of a thread-safe `map[chainid]nonce`. When we build the transaction opts, we lock on the chainid until we finish firing off the transaction. We also keep a `map[txHash]txStatus` with the same thread-safe mechanism. This allows us to concurrently fire off transactions on different chains while ensuring our nonces are correct. - -The [Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) has a selector loop running at all times that calls the `processQueue` method, concurrently processing and storing confirmed txs, or using the [chain queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) to fire off and store pending txs on chain. +The [Queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) has a selector loop running at all times which calls the `processQueue` method, concurrently processing and storing confirmed txs, or using the [chain queue](https://github.com/synapsecns/sanguine/blob/ethergo/v0.9.0/ethergo/submitter/chain_queue.go) to fire off and store pending txs on chain. #### Customizing DB Behavior @@ -124,10 +123,10 @@ Using [GORM.db](https://pkg.go.dev/gorm.io/gorm), you can use whatever database ```go gdb, err := gorm.Open(mysql.Open(dbURL), &gorm.Config{ - Logger: common_base.GetGormLogger(logger), - FullSaveAssociations: true, - NamingStrategy: NamingStrategy, - NowFunc: time.Now, + Logger: common_base.GetGormLogger(logger), + FullSaveAssociations: true, + NamingStrategy: NamingStrategy, + NowFunc: time.Now, }) ``` @@ -135,11 +134,11 @@ gdb, err := gorm.Open(mysql.Open(dbURL), &gorm.Config{ Submitter exposes metrics for Prometheus. The metrics are: -- `num_pending_txs`: The number of pending transactions. -- `current_nonce`: The current nonce. -- `oldest_pending_tx`: The age of the oldest pending transaction. -- `confirmed_queue`: The number of confirmed transactions. -- `gas_balance`: The current gas balance. +- `num_pending_txs`: The number of pending transactions +- `current_nonce`: The current nonce +- `oldest_pending_tx`: The age of the oldest pending transaction +- `confirmed_queue`: The number of confirmed transactions +- `gas_balance`: The current gas balance The metrics can be used in a dashboard [here](https://raw.githubusercontent.com/synapsecns/sanguine/master/ethergo/dashboard.json). It looks like this: