[indexer] merge in indexer breaking change park #18899

emmazzz · 2024-08-03T01:01:22Z

Description

Changes in this uhaul PR includes:

Addition of objects_version table
Addition of a number of transaction and event indices tables, with the removal of tx_calls table.
Enable pruning for the newly added indices tables and make tx sequence number a valid partition range, in addition to cp sequence number.

Test plan

tested locally

Release notes

Protocol:
Nodes (Validators and Full nodes):
Indexer: This PR modifies the indexer database schemas for improving GraphQL query performance. Specifically, an objects_version table along with various transaction and events lookup tables are added. The tx_calls table is replaced by more fine-grained tables: tx_calls_pkg, tx_calls_mod, and tx_calls_fun.
JSON-RPC:
GraphQL:
CLI:
Rust SDK:
REST API:

) ## Description One transaction can call multiple functions in a package so the primary key should include all parts of a function - package, module, func. ## Test plan Existing tests. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into `objects_history` by offering a path for a first look-up to the correct partition in that table for a given object's ID and version. This PR introduces the table, and the logic to populate it in the indexer. ``` sui$ cargo nextest run -p sui-indexer sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration ``` A future PR will make use of this table from GraphQL, which will test it further. - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

) ## Description Recreating #17690 which was accidentally closed. Adding data to the `packages` table, to support the following GraphQL queries: ```graphql type Query { # Fetch all packages created strictly `afterCheckpoint` and strictly # before `beforeCheckpoint`. packages( afterCheckpoint: Int, beforeCheckpoint: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch all packages in the same family as the package at `address` # with versions strictly after `afterVersion` and strictly before # `beforeVersion`. packageVersions( address: SuiAddress!, afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch a package by its address, and optionally supplying a # version. If the version is supplied, returns the package whose # Original ID matches the Original ID of the package at `address`, # but whose version is `version`, otherwise just fetches the package # directly. package(address: SuiAddress!, version: Int): MovePackage } type MovePackage { # Return the package whose Original ID matches this package's but # whose version matches `version`. If `version` is not supplied, # defaults to the latest version. atVersion(version: Int): MovePackage # Fetch all packages in the same family as this package, with # versions strictly after `afterVersion` and strictly before # `beforeVersion`. versions( afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection } ``` These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry. ## Test plan Make sure nothing is broken: ``` sui$ cargo nextest run -p sui-indexer ``` Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: Adds the following fields: `packages.original_id`, `packages.package_version`, `packages.checkpoint_sequence_number` to support queries about package upgrades. - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

…17705) ## Description As titled ## Test plan existing tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

…iles for pg-backed indexer per transactions benchmark findings (#18231) Apply findings from latest transactions benchmarking. Instead of keeping indexes on `transactions`, we offload all filtering work to the lookup tables. Additionally, we add `tx_calls_pkg` and `tx_calls_mod` and dedicate `tx_calls_fun` to solely handle filters on `package`, `package` and `module`, and `package`, `module`, and `func` respectively. This is because it's possible to invoke the same package, mod, and even func in a transaction. Finally, the new setup drops `cp_sequence_number` from the lookup tables and adds `sender` to them to support `sender` + specialized filter. The Rust structs and code to index data to the lookup tables are also updated. The full stack of changes are listed below, but each can stand independently, and will be individually merged into `idx-breaking-change-park` 1. #18224 - handling tables partitioned on epoch's worth of tx_sequence_number 2. #18244 - storing a mapping of checkpoint -> tx_sequence_number range --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

… of tx_sequence_number (#18224) To partition an epoch per `tx_sequence_number`, the last `tx_sequence_number` of an epoch is the `network_total_transactions` from the last checkpoint in the current epoch. The first `tx_sequence_number` is then the `network_total_transactions` from the last checkpoint in the previous epoch + 1, or the `network_total_transactions` of the current epoch - `epoch_total_transactions` of the current epoch. Since we have `network_total_transactions` from the checkpoint to commit, and `epoch_total_transactions` since we fetch current epoch from db, we can derive the range without additional db fetches. However, this approach uncovered a bug of sorts: when we read the network total txn of epoch X - 2 from the db on the indexing side, we `select max(network_total_transactions) from checkpoints where epoch = {epoch}`. However, we may not have actually committed epoch X-2's data on the committer side. This is unlikely to happen in prod because it requires the lag between indexing and committing to be >= one epoch. Since an epoch today consists of ~80k checkpoints, it is quite improbably for this to occur. However, in many of our tests it's possible to have as little as one checkpoint and/ or transaction between epochs, which brought this race condition to light. To resolve, we can place the responsibility on `persist_epoch`. When we get to a point to persist epoch X - 1, epoch X - 2's data must have been indexed to db. Existing tests --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

…_number to checkpoints table (#18244) Per title - add `min_tx_sequence_number` and `max_tx_sequence_number` to `checkpoints` table to support queries mapping `cp` to a `tx` range. This is so that when given a `checkpoint_sequence_number` range, we can map it to the corresponding `tx_sequence_number` range to facilitate transactions queries. The two new columns are nullable, because it is possible for a checkpoint to not contain any transactions. In practice, we'll typically have at least a commit prologue txn. The nullable columns do not impact the queries we intend to run on the table. If through a CTE, a null value will eventually resolve to FALSE when we attempt to use the min or max tx sequence number in a query against the transactions and tx lookup tables. If we make multiple roundtrips, we can stop early if we see that the range is empty for a checkpoint. Existing tests --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

Add event index tables to indexer db for graphql perf. The corresponding script used for investigation is here: https://github.com/MystenLabs/graphql-benchmark/pull/23/files tested locally against devnet --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: --------- Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com>

) If a table is partitioned by not by epochs, we shouldn't do anything to it when advancing epochs. Tested locally. --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

vercel · 2024-08-03T01:01:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 11, 2024 11:32am

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Aug 11, 2024 11:32am
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Aug 11, 2024 11:32am
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Aug 11, 2024 11:32am

emmazzz · 2024-08-03T01:49:55Z

The lates commit ([indexer] add pruning of event indices) was added by me mimicking what Ge did for txns. Please help me check if it looks right @gegaowp.

emmazzz · 2024-08-03T01:52:09Z

crates/sui-indexer/src/models/checkpoints.rs

@@ -42,6 +42,8 @@ pub struct StoredCheckpoint {
    pub checkpoint_commitments: Vec<u8>,
    pub validator_signature: Vec<u8>,
    pub end_of_epoch_data: Option<Vec<u8>>,
+    pub min_tx_sequence_number: Option<i64>,


I remember us having a discussion regarding this under a PR and it was related to some checkpoints containing no txns in testing setup, but why does this need to be Option again? @wlmyng

I think we can privilege @gegaowp 's implementation over the stuff here (iirc his changes were merged into main instead of the breaking changes park)

Actually we'll need to keep these min/max tx_sequence_numbers the way they are because the backfill dbs have uses schema and Ge's implementation for pruner actually didn't add these columns in checkpoints table.

emmazzz · 2024-08-09T10:53:40Z

Hmm we are failing the msql clippy. Is there an easy way to disable it so it's not required for merging? @gegaowp

This reverts commit 212f3de.

Reverts #18899

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging #18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before #18899 - cherry-pick this pr - cherry-pick #18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description title, this is another attempt of #18899 which got reverted as it triggered some CI issues, the issues have been resolved by #18993 ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Emma Zhong <emma@mystenlabs.com> Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com> Co-authored-by: Emma Zhong <emmazhongjy@gmail.com>

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging MystenLabs#18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before MystenLabs#18899 - cherry-pick this pr - cherry-pick MystenLabs#18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging #18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before #18899 - cherry-pick this pr - cherry-pick #18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging #18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before #18899 - cherry-pick this pr - cherry-pick #18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging MystenLabs#18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before MystenLabs#18899 - cherry-pick this pr - cherry-pick MystenLabs#18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

…9005) ## Description title, this is another attempt of MystenLabs#18899 which got reverted as it triggered some CI issues, the issues have been resolved by MystenLabs#18993 ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Emma Zhong <emma@mystenlabs.com> Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com> Co-authored-by: Emma Zhong <emmazhongjy@gmail.com>

## Description Move the release notes validator into its own workflow ## Test plan ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py check 19148 Found issues with release notes in 19148: - 'Protocol' is checked but has no release note. ``` ![Screenshot 2024-08-30 at 9 07 27 AM](https://github.com/user-attachments/assets/7665bad5-4b36-4faf-9e41-8195dc935156) ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py generate releases/sui-v1.31.0-release releases/sui-v1.32.0-release ## Protocol #### Sui Protocol Version in this release: `55` #19031: Enable Move enums in mainnet ## Indexer #18899: .... ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [x] Protocol: - [x] Nodes (Validators and Full nodes): and here - [x] Indexer: and here - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

commit 951980dbb74cd18fea916e0ad162f7f22d6785e0 Merge: 57b22f7697 77b18b45c1 Author: Carlos Baez <c@mevton.com> Date: Fri Sep 13 12:21:56 2024 +0200 finish merge Signed-off-by: Carlos Baez <c@mevton.com> commit 77b18b45c195103e25d62fb6cd35ab812b9e7f12 Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Sun Sep 8 20:42:23 2024 -0700 Cherry pick 19246 (finalize bridge committee on mainnet in protocol version 58) (#19268) ## Description https://github.com/MystenLabs/sui/pull/19246 ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit fafb1429b7afb56431068cfe5a7cce6537d09784 Author: Eugene Boguslavsky <eugene@mystenlabs.com> Date: Fri Sep 6 11:03:01 2024 -0700 Add debug builds back for all platforms (#19250) ## Description Add debug builds back for all platforms ## Test plan https://github.com/MystenLabs/sui/actions/runs/10741384417/job/29791637040 commit 6c732bf42ca374e9f1c4fa1aa9b43e79c463de7b Author: Eugene Boguslavsky <eugene@mystenlabs.com> Date: Thu Sep 5 18:16:51 2024 -0700 Sui v1.33.0 Bytecode Framework Snapshot (#19240) ## Description Sui v1.33.0 Bytecode Framework Snapshot ## Test plan 👀 commit 0c02e0735a8d6ec26f95f2f5d62d0c29f20b59ba Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Thu Sep 5 18:12:53 2024 -0700 [CI] do not ignore failures from Split Cluster Check in PRs (#19218) ## Description Currently when the check runs in PR CI, the failure is ignored and bisect does not run. So the workflow succeeds when it shouldn't. For example, https://github.com/MystenLabs/sui/actions/runs/10641580676/job/29502904956#step:3:2268 ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c86143bbfecc0a4cdec48e5917c358562f83450e Author: Bridgerz <bridgerzoske@gmail.com> Date: Thu Sep 5 17:52:53 2024 -0700 Add storage gap variable to the CommitteeUpgradeable contract (#19228) ## Description In the case the `CommitteeUpgradeable` contract needs to include more storage variables, a storage gap variable is needed to reserve storage slots so the child contract state is not overwritten. ## Test plan Contract upgrade unit tests commit 611056816a1e01f482e329b46ac9b3fd8a28a39c Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Thu Sep 5 17:18:15 2024 -0700 [bridge-indexer] batch processing checkpoints and indexing progress metrics (#19179) ## Description This PR does two things: 1. pull checkpoint batches to processed when there are multiple, rather than one-by-one. This improves the efficiency especially for write_process 2. add metrics to check indexing progress. ## Test plan Deployed in production. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit fa776a03b4d94079aec74e200d275c340e3325fb Author: Adam Welc <adam@mystenlabs.com> Date: Thu Sep 5 17:12:42 2024 -0700 [trace-view] Added support for stepping back through the trace (#19215) ## Description This PR adds the ability to step back through the trace. The main limitation is that stepping back into or over (previously executed) function calls is not supported. We will add this support after support for viewing variables is added as it will requires snapshotting variable state (which we do not have at the moment). ## Test plan Tested manually that: - viewer correctly steps back within the function and from inside the callee to the caller - viewer correctly stops stepping back at the beginning of the trace - viewer correctly stops stepping back upon encountering previously executed function call commit 2b3991a841ce655bd790abfcbbdc5d2f27e14448 Author: Xun Li <lxfind@gmail.com> Date: Thu Sep 5 16:28:12 2024 -0700 [indexer] Compatibility check using migration records (#19156) ## Description Previously we check DB compatibility by making sure that we could make select query on all columns to the DB based on the locally known schema. This doesn't cover all cases, for instance, there could be tables in the DB that does not exist in the local schema. This PR changes how we do the compatibility check by fully leveraging the migration records. It checks that the migration records in the DB must fully match with the locally known one. It also moves the code to sui-indexer crate, so that we could do this check on the startup of both indexer and graphql server. This does require that from now on we fully respect the migration scripts, and don't do adhoc modifications on the existing migration. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 89737f7ca52a6be1047a02a3ffb9ca319098017b Author: wlmyng <127570466+wlmyng@users.noreply.github.com> Date: Thu Sep 5 14:39:23 2024 -0700 [indexer] Objects Snapshot Processor continuously reads from stream (#19232) ## Description In ci, we saw that the `objects_snapshot_processor` stalls if there are unprocessed checkpoint data that we cannot commit unless we have a continuous stream of checkpoints from `[start_cp, max_allowed_cp]`. To address this, the processor should continue to read from stream whether `unprocessed` is empty or not. ## Test plan Ran indexer manually to observe that `objects_snapshot_processor` doesn't stall --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 369d44baab028fd3c685fb95ce4a02196e1324eb Author: Todd Nowacki <tmn@mystenlabs.com> Date: Thu Sep 5 14:24:02 2024 -0700 [move stdlib] Add `fun bitwise_not` and `macro fun max_value` (#19126) ## Description - Added a bitwise not function, `fun bitwise_not` to `u*` modules - Added a maximum value macro, `macro fun max_value` to `u*` modules ## Test plan - New tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [X] CLI: Move integer modules now have a `bitwise_not` function and a `max_value` macro function. - [ ] Rust SDK: - [ ] REST API: commit 7d0ebb9d8039cf4a5d592ebe842f49770117ca37 Author: Ge Gao <106119108+gegaowp@users.noreply.github.com> Date: Thu Sep 5 17:22:47 2024 -0400 indexer: clean up legacy snapshot codes (#19078) ## Description the legacy method of updating `objects_snapshot` is no longer needed, thus this pr cleans it up ## Test plan ci --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 3b3603cc73a6d9ba28ed9ba3f4f00e0682b509d2 Author: phoenix <51927076+phoenix-o@users.noreply.github.com> Date: Thu Sep 5 17:16:56 2024 -0400 [data ingestion] handle termination in worker pool (#19192) ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 262390b27ca7befe9bd3a537aae6ebef9f569348 Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Thu Sep 5 13:41:06 2024 -0700 [rocksdb] add metric for num level 0 files (#19230) ## Description Too many level 0 files can be a trigger to write stall & stop as well ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit d275762040247747ef429eda6309ffbef005c28b Author: jk jensen <jk@mystenlabs.com> Date: Thu Sep 5 11:45:36 2024 -0700 [suiop] add interactive incident selection (#19186) ## Description Allow interactive incident review selection with suiop ## Test plan Successful e2e test in #test-notifications --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit a831de6cd80001dd33e6f7686cb1f29f00dd35b4 Author: Eugene Boguslavsky <eugene@mystenlabs.com> Date: Thu Sep 5 11:43:11 2024 -0700 Do not build sui in debug mode for macos-arm64 and delete unused software. (#19210) ## Description Do not build sui in debug mode for `macos-arm64` and delete unused software. ``` Filesystem Size Used Avail Capacity Mounted on /dev/disk3s1s1 295Gi 9.6Gi 17Gi 37% / Filesystem Size Used Avail Capacity Mounted on /dev/disk3s1s1 295Gi 9.6Gi 89Gi 10% / ``` ## Test plan https://github.com/MystenLabs/sui/actions/runs/10725603432/job/29743813335 commit 9a3a08502ba0c840b47295a6cc332c93c74343b1 Author: wlmyng <127570466+wlmyng@users.noreply.github.com> Date: Thu Sep 5 10:41:06 2024 -0700 [graphql] Remove usage of legacy objects snapshot processing (#19175) ## Description A separate task is solely responsible for updating `objects_snapshot`, which means the graphql e2e tests, which depend on forcing `objects_snapshot` changes through an update query, also need to obey this. In lieu of the current behavior, we instead configure the snapshot lag per test scenario, and ``` //# advance-clock --duration-ns 1 //# create-checkpoint ``` to get the desired `objects_snapshot` state ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 28feff4c36806869f2d34d86db46536cda107e0c Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 10:15:44 2024 -0500 indexer: use async connection for SystemPackageTask commit 5a331b4f1f214b9375ce65ce43bed649db68805a Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 10:04:29 2024 -0500 indexer: use async connection for get_object commit 277479e483fbf23c074d656784db782fd03203a0 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:57:44 2024 -0500 indexer: remove impl ObjectStore for IndexerReader commit 0348913d6b08660e3027179f6cce4b9d7079f5a1 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:22:23 2024 -0500 indexer: use async connection for get_object_read commit 8d7bcb2e95d717a06d0f7dfd5b0285ef24ba1645 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:18:39 2024 -0500 indexer: use async connection for get_owned_objects commit 21a20469be7e5dc2a0008ebaf7b07abd63e0690e Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:15:27 2024 -0500 indexer: use async connection for multi_get_objects commit a0e193ba62eb2f90e44c5beaf877d6a2945e6733 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:12:27 2024 -0500 indexer: use async connection for query_transaction_blocks commit dc6c4b4569722dde4d4380a6fe3919de6c65ce6d Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:06:39 2024 -0500 indexer: use async connection for query_events commit 47778a2bd281a4aff635f1b51dda3ef49b9dc65e Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 09:00:21 2024 -0500 indexer: use async connection for get_transaction_events commit 76f45f5c0ea63b5dc0389f54aaeed25fa30fe8e3 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 08:56:04 2024 -0500 indexer: use async connection for multi_get_transactions_with_sequence_numbers commit 88ca95177906d5978f48c8aec74732c4cb294a8f Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 08:51:36 2024 -0500 indexer: use async connection for multi_get_transactions commit c1c0d255baaa8d5959484404d58830d99cdc94cd Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 08:48:41 2024 -0500 indexer: use async connection for get_dynamic_fields commit 4b2777d78560611cd21fa0b53c4a2a45600bb629 Author: Brandon Williams <brandon@mystenlabs.com> Date: Thu Sep 5 08:41:20 2024 -0500 indexer: use async connection for get_object_refs commit 3a30a4d5c17e1b5f2bfce425a8f5360fefcf2235 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 19:37:51 2024 -0500 indexer: use async connection for get_coin_balances commit beec6c0315a6379232af7d97664a281d78edcb62 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 19:35:16 2024 -0500 indexer: use async connection for get_owned_coins commit ba937ca3f87905be3c08af1eb675aab0a08de1be Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 17:11:30 2024 -0500 indexer: use async connection for get_display_object_by_type commit 58eabc52a3fac08c340f5b8840588724fff7c12c Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 17:05:18 2024 -0500 indexer: use async connection for get_coin_metadata and get_total_supply commit 4eb829150aeef3deb91f4f85d6e8e87faa61918b Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 16:21:06 2024 -0500 indexer: use async connection for get_checkpoints commit 7b45e7aeb14002e27f6c42cce9c48419bd2c1dcf Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 16:18:40 2024 -0500 indexer: use async connection for get_latest_checkpoint commit 373c0ab28ce08ee08aa1c94a3422cfca8371c084 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 16:14:09 2024 -0500 indexer: use async connection for get_epochs commit 69563730855e2c92c64a191520b6565bbda2eaf0 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 16:07:08 2024 -0500 indexer: remove unused get_consistent_read_range method commit fb1fcb0653b04b2307d57df6b4387d87b06265db Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 16:03:57 2024 -0500 indexer: use async connection for get_epoch_info commit a95f48deb0bb1ba695dc764b88b6b9638164bc9d Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 15:39:24 2024 -0500 indexer: use async connection for get_checkpoint commit 07248c9f9b092cadf48b626f69cb93979f7efe19 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 15:09:57 2024 -0500 indexer: use async connection for package resolver commit 728accdb857540d1735a85279c2990b700a61c44 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 15:09:51 2024 -0500 indexer-writer: instantiate async connection pool commit bd96deea1e0a142b862e96f3c9befb0146ef8670 Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 13:23:29 2024 -0500 indexer-reader: instantiate async connection pool commit 829d03ad52170dd1d833aeb8369e9c74fd28bc2e Author: Brandon Williams <brandon@mystenlabs.com> Date: Wed Sep 4 12:36:48 2024 -0500 indexer: perform database reset via async connection commit 0e78656655529c130e6b680e0962ff6204306713 Author: Andrew Schran <aschran@mystenlabs.com> Date: Thu Sep 5 17:10:00 2024 +0100 Add metric for active time of monitored futures (#19226) commit bb778828e36d53a7d91a27e55109f2f45621badc Author: phoenix <51927076+phoenix-o@users.noreply.github.com> Date: Thu Sep 5 10:43:54 2024 -0400 repair coin index (#19142) temporary PR. This PR introduces a background task to repair the `coin_index` and remove any dangling entries. The PR will be active for one release and will be reverted afterward. The background task works by iterating over a snapshot of the `coin_index`, identifying coins that no longer belong to their respective owners, and populating a list of candidates for removal(some entries might be benign) Once the candidate list is populated, the task makes a second pass over the candidates list. This time it locks the corresponding entries to prevent potential races with concurrent writes. The task then reverifies the eligibility criteria and removes the dangling entries commit 22844ae53bb2f59fdf790fadaa9ad1b7e5a7d94e Author: Jordan Gensler <jordan@mystenlabs.com> Date: Thu Sep 5 08:36:47 2024 -0400 Update zkSend docs (#19170) commit 2d019963d9bfebfcf09940e623ecafd0b6bcfcb8 Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Wed Sep 4 22:24:41 2024 -0700 Revert "Remove RandomnessStateUpdate" from ConsensusTransactionKind (#19217) ## Description It turns out BCS serializes enum purely based on the variant order. The discriminant / tag does not matter. https://github.com/diem/bcs/blob/master/README.md#externally-tagged-enumerations Also revert tagging of the enum since it is no longer useful. ## Test plan SUI_PROTOCOL_CONFIG_CHAIN_OVERRIDE=testnet scripts/compatibility/split-cluster-check.sh origin/testnet a6336e6390b31379c7d41ef0d6fba4f966fad00c --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 1ebf5f8a395ca00f7cdcfa19e988975d8ccc15a3 Author: Ashok Menon <ashok@mystenlabs.com> Date: Thu Sep 5 01:04:20 2024 +0100 test(graphql): Remove explicit cleanup_resources (#19201) ## Description These were originally added to try and deal with a deadlock issue, which was due to an issue with `inotify` on macOS, fixed by #19195. These calls are safe to remove (also note that if the test failed, they would never be hit, because the assert would cause a panic before they ran). ## Test plan Rebase on top of #19195, run the following on macOS: ``` sui$ cargo nextest run \ -j 1 -p sui-graphql-rpc \ --test e2e_tests \ --features pg_integration ``` This used to intermittently hang, but now succeeds. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit e1abe0963b579d69471ce40bd465d63fb80f0f41 Author: Zhe Wu <halfprice@users.noreply.github.com> Date: Wed Sep 4 16:31:30 2024 -0700 Add a metric to track the number of managed files per column family in RocksDb (#19214) We don't add more detailed level to control the cardinality of the metrics. Manually tested in local cluster: <img width="1971" alt="Screenshot 2024-09-04 at 3 11 22 PM" src="https://github.com/user-attachments/assets/968bb0ac-321f-4749-bc13-e5f852c93ea1"> ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit bf303ab96cb7c8baeea1a6a78fdb75e32fb99a6b Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Wed Sep 4 15:59:56 2024 -0400 Version Packages (#19212) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/deepbook-v3@0.5.0 ### Minor Changes - c53baf2: Redeploy packages Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit c53baf24d4759ea88f7f5aaf5a11495047ef058b Author: Tony Lee <tony.lee@mystenlabs.com> Date: Wed Sep 4 15:48:55 2024 -0400 Testnet update after campaign (#19147) ## Description Package updates with latest changes ## Test plan How did you test the new or updated feature? Testnet ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit bf8a3ae6387ec3e8dc3abb15d4f4f217bf948130 Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Wed Sep 4 12:11:00 2024 -0700 [Narwhal] remove Narwhal crypto usages from Sui (#19205) ## Description This is another step in removing Narwhal integration with Sui. Narwhal crypto types used in Sui are replaced with the corresponding Sui types. There should be no difference in logic or serialized formats. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 6a0adb5de7f3fb87d3aa23364f200b542e3e8b87 Author: Sadhan Sood <106645797+sadhansood@users.noreply.github.com> Date: Wed Sep 4 11:51:33 2024 -0700 Delete sst files in key range in rocksdb (#19211) ## Description This PR adds a rocksdb endpoint to delete .sst files in key range in rocksdb which is useful to prune data (in certain scenarios) without compaction. commit a22b71b70d6330f33c165198c194e4b716bc953b Author: Andrew Schran <aschran@mystenlabs.com> Date: Wed Sep 4 19:51:19 2024 +0100 add monitored scopes to CheckpointExecutor (#19209) commit aa9dd4221020dc59d7576d938d8e2954671927ae Author: phoenix <51927076+phoenix-o@users.noreply.github.com> Date: Wed Sep 4 09:11:37 2024 -0400 [data ingestion] disable inotify for macos (#19195) ## Description prevents local tests from hanging ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit e7437194b6e634660b9c27e7a86b2eb6f2d34d29 Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Tue Sep 3 21:03:41 2024 -0700 [bridge] enable bridge on mainnet (#19200) ## Description enable bridge creation on mainnet by setting bridge is true. Before this change, the value true for testnet and devnet, but not mainnet. So this only applies to mainnet. ## Test plan ### Mainnet * expect 56, 57, 58 all have bridge = true * [old 57](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_57.snap) v.s. [new 58](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_58.snap): https://www.diffchecker.com/AbcQx7DQ/ * [old 56](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_56.snap) v.s. [new 57](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_57.snap): https://www.diffchecker.com/E3b9mlHd/, notably random_beacon_reduction_lower_bound: 800 in new 57, which was in old 56 * [old 56](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_56.snap) v.s [new 56](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Mainnet_version_56.snap): https://www.diffchecker.com/EkWIOCeQ/ this reverts [@Andrew Schran](https://mysten-labs.slack.com/team/U03TDESBNR0) ’s change of random_beacon_reduction_lower_bound , now back to 1000. ### Testnet * unlike mainnet, bridge has been set to true months ago, so expect no change for this parameter. * [old 57](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_57.snap) v.s. [new 58](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_58.snap): https://www.diffchecker.com/bc7A5fVn/, no change * [old 56](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_56.snap) vs [new 57](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_57.snap): https://www.diffchecker.com/XaktzhpC/ no change. notably random_beacon_reduction_lower_bound: 800 in new 57, which was in old 56 * [old 56 ](https://github.com/MystenLabs/sui/blob/main/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_56.snap)v.s [new 56:](https://github.com/MystenLabs/sui/blob/bb25a072dac4848337c111d4fefeb0256fa36cd8/crates/sui-protocol-config/src/snapshots/sui_protocol_config__test__Testnet_version_56.snap) https://www.diffchecker.com/SeNewiDN/ this reverts [@Andrew Schran](https://mysten-labs.slack.com/team/U03TDESBNR0) ’s change of random_beacon_reduction_lower_bound , now back to 1000. * effectively version 56 is a no-op for testnet --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 4b0d129d0548a7e36dc549c499529087c8999a0a Author: Andrew Schran <aschran@mystenlabs.com> Date: Wed Sep 4 01:30:52 2024 +0100 Add monitored scope for single threaded checkpoint builder (#19197) commit 76995edaa85ea9f41d403da76954a8e047013056 Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Tue Sep 3 17:18:25 2024 -0700 [Docs] update references and metrics post Mysticeti launch (#19198) ## Description Checkpoint rate is now ~4/s post Mysticeti launch. Narwhal and Bullshark are no longer running in Sui and references to them need to be migrated to Mysticeti. ## Test plan CI --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit dfe05bf6411c969327945b41103be49d22b2162d Author: Pika <pzm16@tsinghua.org.cn> Date: Wed Sep 4 07:45:34 2024 +0800 Update swaps.mdx (#19187) fix typo `coin::zer()` --> `coin::zero()` ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 04e6b0f3e4de536e68fe757d628ab66fc30aa582 Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com> Date: Tue Sep 3 16:40:23 2024 -0700 Version Packages (#19168) This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @mysten/sui@1.8.0 ### Minor Changes - 569511a: Add data to result of executeTransaction methods on Transaction executor classes ## @mysten/zksend@0.11.0 ### Minor Changes - 4bdef4a: Add support for testnet in Stashed and zkSend SDKs. ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 - @mysten/wallet-standard@0.13.3 ## @mysten/create-dapp@0.3.18 ### Patch Changes - Updated dependencies [569511a] - Updated dependencies [012aefe] - @mysten/sui@1.8.0 - @mysten/dapp-kit@0.14.18 ## @mysten/dapp-kit@0.14.18 ### Patch Changes - 012aefe: Support passing network param through to stashed wallet - Updated dependencies [4bdef4a] - Updated dependencies [569511a] - @mysten/zksend@0.11.0 - @mysten/sui@1.8.0 - @mysten/wallet-standard@0.13.3 ## @mysten/deepbook@0.8.17 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/deepbook-v3@0.4.3 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/enoki@0.4.1 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 - @mysten/zklogin@0.7.18 ## @mysten/graphql-transport@0.2.17 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/kiosk@0.9.17 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/suins-toolkit@0.5.17 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/wallet-standard@0.13.3 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 ## @mysten/zklogin@0.7.18 ### Patch Changes - Updated dependencies [569511a] - @mysten/sui@1.8.0 Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 569511aceb344ad31dea30084938bf9ccffa5bc9 Author: hayes-mysten <135670682+hayes-mysten@users.noreply.github.com> Date: Tue Sep 3 16:20:17 2024 -0700 Add data to result of executeTransaction methods on Transaction execu… (#19202) …tor classes ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit f351cba7898f7a18ad9dc5a27d0fbefe6eb7dead Author: Adam Welc <adam@mystenlabs.com> Date: Tue Sep 3 16:03:49 2024 -0700 [move-ide] Fixes auto-completion for fields (#19176) ## Description This PR fixes field auto-completion in two ways: - adds support for struct fields (previously only variant fields were being auto-completed) - named fields are now listed in their definition order (which should typically be what a developer wants) - finesses auto-completion formatting (previously all fields where inserted on a single line, and now named fields are inserted on separate lines if there are more than two fields) - adds a no-field auto-completion option (for when a struct or variant is used as a type and not in pack/unpack context) ## Test plan All tests must pass commit e03a8abd7b4a4e3ef22eb405246361349e3cce49 Author: Cam Swords <cameronswords@gmail.com> Date: Tue Sep 3 15:58:14 2024 -0700 [move][move-vm] Add more benchmarks, slightly reorganize the code (#18864) ## Description This adds a few more benchmarks to the current VM benchmark suite, plus reorganizes it slightly ## Test plan `cargo bench` in the directory --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 8e713e26c9b013955ce19544c345d0626e8fce96 Author: Pankaj Jangid <pankaj.jangid@gmail.com> Date: Wed Sep 4 03:51:36 2024 +0530 Fixed incomplete sentence in the docs (#19160) - Follow the instructions here to run your own Sui Full. + Follow the instructions here to run your own Sui Full Node. --------- Included some commit chatter likely due to rebase Co-authored-by: ronny-mysten <118224482+ronny-mysten@users.noreply.github.com> commit 32c7828da902476ff344effe8467b7331bfda508 Author: Pika <pzm16@tsinghua.org.cn> Date: Wed Sep 4 06:20:09 2024 +0800 fix typo in deepbookv3-sdk.mdx doc (#19181) commit fa7419bfcfeffaa816e3232c5d6af5db67dc7ca8 Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Tue Sep 3 15:14:36 2024 -0700 Refactor ConsensusTransactionKind (#19177) ## Description A few small refactors: - Remove the deprecated RandomnessStateUpdate variant - Rename UserTransaction to CertifiedTransaction. UserTransaction will be used for the non-certified variant later. - Tag variants so serialization is unaffected when variants are reordered. Using `repr(u8)` should be compatible with the varint serialization of default tag type. ## Test plan CI. Upgrade tests in simulation and PT. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 87ec25b6563bb38c80d8ea378ca824dd4dae9889 Author: Cam Swords <cameronswords@gmail.com> Date: Tue Sep 3 14:03:21 2024 -0700 [move][move-2024] Boolean binop optimizations (#18943) ## Description Optimize boolean binop groups to avoid extra locals during HLIR lowering, helping with metering prices. ## Test plan All tests still pass --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 82bf44e58df032e5a8dd5e452c51a1aa6959f15e Author: Andrew Schran <aschran@mystenlabs.com> Date: Tue Sep 3 19:24:56 2024 +0100 Migrate users of mysten_metrics Histogram to prometheus Histogram (#19124) mysten_metrics variant is difficult to use in Grafana because its precomputation of percentiles makes it impossible to aggregate streams from multiple hosts in a statistically valid way. This keeps a few of the mysten_metrics Histogram versions around to help with the transition of exsiting users. commit dd8e82ccfc6950831a32d9644518e3e0c83ef1e1 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 16:34:51 2024 -0500 graphql: convert remaining tests to use ephemeral postgres db Convert remaining graphql tests to use an ephemeral pstrgres db enabling them all to be run in parallel. After this patch the graphql-test workflow takes ~7 minutes compared to ~19 minutes it used to take. commit 12890bd598786d8dd53bc1cd10bb0629e13b5bc8 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 16:03:05 2024 -0500 graphql: convert examples-validation_tests to use temporary db commit 4205ea4115d4149b47d9c3cfed61fe839385de56 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 15:56:34 2024 -0500 sui-cluster-test: convert tests to use temporary postgres db commit c412c82f07bc7fe6792ae5d0fc22a5acee40b064 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 15:03:17 2024 -0500 sui-indexer, sui-graphql-e2e-tests: convert tests to use TempDb commit 347c9da11e0dc0b9a62a7ae4bd2150da5b4fe4e8 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 14:59:05 2024 -0500 indexer: introduce TempDb and LocalDatabase Introduce LocalDatabase, a local instance of a postgres server, as well as TempDb, a way to create ephemeral postgres databases for use in testing enviornments. commit f11d10319305621b3a7200166a2db5c181169c76 Author: Brandon Williams <brandon@mystenlabs.com> Date: Mon Sep 2 14:57:35 2024 -0500 indexer: introduce async postgres connection helpers commit e66bdb5a2e2ec0e935778c76c128a9b881856b86 Author: Ashok Menon <ashok@mystenlabs.com> Date: Tue Sep 3 18:07:57 2024 +0100 feat(graphql): Remove versioning support (#19191) ## Description Removes support for multiple versions from GraphQL, including: - Version parsing - Routing by version - Associated tests - `ServiceConfig.availableVersions` in schema - `[versions].versions` in TOML ServiceConfig This change also removes the only use of some error APIs which have also been cleaned up. The Service now quotes the same version as `sui-node`, etc, so to prevent churn during tests, it has been mocked to a fixed value. ## Test plan ``` sui-graphql-rpc$ cargo nextest run sui-graphql-e2e-tests$ cargo nextest run --features pg_integration ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Removes support for multiple versions. GraphQL's version now aligns with the `sui-node` version, and it drops support for the `ServiceConfig.availableVersions` query. - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit efc47d1ae5b6a9ee2bbf6a535f9478f0dda3808b Author: omahs <73983677+omahs@users.noreply.github.com> Date: Tue Sep 3 18:46:39 2024 +0200 [docs] Fix typos (#19190) [docs] Fix typos commit ceeb4c60687daf7376df8119fb49bc14826f3617 Author: Bridgerz <bridgerzoske@gmail.com> Date: Tue Sep 3 09:40:32 2024 -0700 Reintroduce soldeer dependency management and pin foundry version (#19131) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit fdc8325abfd62013c83fdee06034609ae60f4c7c Author: stefan-mysten <135084671+stefan-mysten@users.noreply.github.com> Date: Sat Aug 31 04:05:56 2024 -0700 [GraphQL] Add a mutation payload size (#18017) ## Description The mutation payload can be a lot higher than a query payload, due to the possibility of passing a large transaction data (e.g., publishing a package). This PR adds a different check for when a mutation is requested, and adds a `max_tx_payload_size` variable that holds the max bytes that can be sent through a GraphQL mutation request. The total sum of `txBytes + signatures` of all GraphQL mutation or `txBytes` in a `dryRunTransactionBlock` query have to be below the `max_tx_payload_size`. The `max_tx_payload_size` is computed based on the `protocol_version -> max_tx_bytes` and a Base64 overhead as follows: `max_tx_bytes * 4 / 3` ## Test plan Added several tests. `cd crates/sui-graphql-rpc` `cargo nextest run --features pg_integration -- test_query test_mutation test_dry_run_transaction test_transaction_dry_run` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Added a `max_tx_payload_size` variable to protect against large transaction queries. The sum of `txBytes + signatures` in all GraphQL mutation `executeTransactionBlock` nodes or `txBytes` in `dryRunTransactionBlock` nodes from a query have to be below the `max_tx_payload_size`. The `max_tx_payload_size` is computed based on the `protocol_version -> max_tx_bytes` and a Base64 overhead as follows: `max_tx_bytes * 4 / 3` Added also a check that the overall query size is not larger than `max_tx_payload_size` + `max_query_payload_size`, where `max_query_payload_size` is the `read` part of the query. - [ ] CLI: - [ ] Rust SDK: --------- Co-authored-by: Ashok Menon <ashok@mystenlabs.com> commit 44ab1f5e7e9f6bccea1ff244c09888fdd19fd02c Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Fri Aug 30 21:52:07 2024 -0700 [bridge-indexer] change live task start point (#19174) ## Description 1. now the live task's starting height is always `get_live_task_starting_checkpoint`. See the comment for more consideration. Previously we used a value in config to determine. 2. add `fn get_live_task_starting_checkpoint` and `fn get_genesis_height` to `DataSource` trait. Therefore each datasource implements their own method to pick these values to determine task ranges, as opposed to we do it on `main.rs` today. 3. clean up `fn build` for `IndexerBuilder` by moving existing parameters to elsewhere. 4. remove unused parameters in indexer config ## Test plan tests and more tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit b9dd8837bbb837229cdc9c6e6d1d1e22901c987f Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Fri Aug 30 15:52:35 2024 -0700 [bridge-indexer] fix duplicated task creation (#19171) ## Description In function `tasks()` we return only incomplete tasks. As a result, when the latest backfill tasks are done and ignored, we mistakenly use an intermediate value as the `latest target checkpoint` and use it to fill gaps. This causes duplicated tasks. This PR fixes it: 1. rename `tasks` to `get_ongoing_tasks` for semantics 2. add `get_largest_backfill_task_target_checkpoint` and use that to determine whether there is a gap 3. simplify `update_tasks` for backfill task creation 4. add some utils functions for testing. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit c661699d3511facf92e30af288c062d3ba39f210 Author: Brandon Williams <brandon@mystenlabs.com> Date: Fri Aug 30 16:45:58 2024 -0500 indexer: rework config and cli arguments (#19162) This patch reworks a number of the indexer configs and cli arguments to try and centralize where the configuration for the indexer is loaded. This also cleans up the cli arguments into well defined subcommands instead of using boolean flags for deciding if a jsonrpc or indexer writer service should be started. commit 1f9fbfba5bb728286100a3f9d4d3aa0acb5e9a35 Author: Lu Zhang <8418040+longbowlu@users.noreply.github.com> Date: Fri Aug 30 14:10:02 2024 -0700 [bridge-indexer] some logs and todos (#19154) ## Description as title ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit f64c6ffe4aba85549eb77dac55dfe8657cbc3216 Author: Eugene Boguslavsky <eugene@mystenlabs.com> Date: Fri Aug 30 13:38:30 2024 -0700 Move release notes validator into its own workflow (#19148) ## Description Move the release notes validator into its own workflow ## Test plan ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py check 19148 Found issues with release notes in 19148: - 'Protocol' is checked but has no release note. ``` ![Screenshot 2024-08-30 at 9 07 27 AM](https://github.com/user-attachments/assets/7665bad5-4b36-4faf-9e41-8195dc935156) ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py generate releases/sui-v1.31.0-release releases/sui-v1.32.0-release ## Protocol #### Sui Protocol Version in this release: `55` https://github.com/MystenLabs/sui/pull/19031: Enable Move enums in mainnet ## Indexer https://github.com/MystenLabs/sui/pull/18899: .... ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [x] Protocol: - [x] Nodes (Validators and Full nodes): and here - [x] Indexer: and here - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 012aefe3c5b9b2bae6c33643932a6bff246c3ad7 Author: Jordan Gensler <jordan@mystenlabs.com> Date: Fri Aug 30 15:46:26 2024 -0400 Stashed wallet dapp-kit (#19169) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 42dbb6253959e1deda9afe4f34c0d717b9d67ccc Author: mwtian <81660174+mwtian@users.noreply.github.com> Date: Fri Aug 30 12:14:33 2024 -0700 [Store] disable write stall on fullnodes perpetual db (#19134) ## Description For fullnodes that do not prune the `perpetual` DB, especially `transactions` and `effects` cfs, they can run into write stalls occasionally that makes the fullnode non operational. Since fullnodes have to accept all writes from checkpoints, throttling writes do not seem to make much sense. Write stalling on validators is left enabled. ## Test plan CI Deploy to a few fullnodes. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit ddeffaabe0cf0ced3a235912831c62296c73d434 Author: Mark Logan <103447440+mystenmark@users.noreply.github.com> Date: Fri Aug 30 19:51:41 2024 +0100 Add decode-key command to http_kv_tool (#18990) e.g. `http_kv_tool decode-key -u https://transactions.sui.io/mainnet/cP1pPYHRXroEHhXrsD_uy-kbAcH5lZguUEPfocX0zXsIsR0GAAAAAA/ob` commit 724b54cbb8bcb6046d1ee63ba23823261d594cfb Author: Andrew Schran <aschran@mystenlabs.com> Date: Fri Aug 30 19:26:03 2024 +0100 reduce minimum random beacon shares to 800 (#19165) commit 346775946303e0e432685a7aebbe4ae108ed5e88 Author: Andrew Schran <aschran@mystenlabs.com> Date: Fri Aug 30 19:25:21 2024 +0100 Put back HandleConsensusOutput scope removed in PR #19089 (#19166) commit 4bdef4a070b590f366730a8aba77251b9a867331 Author: Jordan Gensler <jordan@mystenlabs.com> Date: Fri Aug 30 14:13:00 2024 -0400 Add testnet support to stashed (#19167) ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 59fe76678b288807f4e54bcd2da13cf777fb0da8 Author: wlmyng <127570466+wlmyng@users.noreply.github.com> Date: Fri Aug 30 10:38:05 2024 -0700 [indexer][graphql] Pruner prunes only epoch-partitioned tables (#19164) ## Description Currently, the pruner assumes that all partitioned tables are partitioned on epoch, which is an issue since `objects_version` is not partitioned by epoch. Modify the pruner so that it will filter out non-epoch-partitioned tables, and otherwise do the same thing. Change `EPOCHS_TO_KEEP` from an env variable to a config, so we can pass in test values through the transactional test runner, and a prune.move test to validate that we prune epoch data without any trouble. ## Test plan prune.move --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 131004734a2bdd8bf626eee8e7c9e2147f69eb89 Author: Cam Swords <cameronswords@gmail.com> Date: Thu Aug 29 21:22:43 2024 -0700 [move][move-2024] Add match fix for typing around literal switches, plus tests (#19133) ## Description This addresses a bug where `abort` was causing mistyped literal arm binders in match compilation. It also addresses some false-positive dead code complaints that I discovered while fixing the bug up. Longer-term, it would be nice to eliminate temp binders of the form `#%1: _|_ = unreachable` from HLIR generation so that CFGIR can invariantly ensure none exist, catching these sorts of issues, but due to multiple-binding forms `(x, y, z) = (abort 0, abort 1, abort 2)` and the current structure of the pass, that is work left for another day. ## Test plan Several more tests to cover these cases, though still never enough. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 070a2c38134a4ff4afaca150218289a3c8927603 Author: Mark Logan <103447440+mystenmark@users.noreply.github.com> Date: Fri Aug 30 01:58:44 2024 +0100 Improvements to thread stall monitor (#19151) Use a watchdog thread for monitoring thread stalls. This allows us to detect a thread stall while it is still occurring, rather than only after the fact. commit 07f8b22d5615ad2bcb355d32b02317e024987da2 Author: Mark Logan <103447440+mystenmark@users.noreply.github.com> Date: Fri Aug 30 01:39:57 2024 +0100 Caching resolver for Tonic. (#19152) On some platforms (GKE) the local resolver does not seem to cache aggressively, which results in high variance for requests commit 89f3a3a86719c5f7c64352072cc777f6e7e585b8 Author: Tim Zakian <2895723+tzakian@users.noreply.github.com> Date: Thu Aug 29 17:09:17 2024 -0700 [execution] Update how verifier constants for signing are set (#19094) ## Description Since verifier constants for limits and metering for signing do not need to be protocol versioned, a previous PR moved some of these constants out of the protocol config. This PR takes that one step further by allowing it to be set by the node config instead of being hardcoded into the binary. The default is that these should remain unset in the node config, but this gives the ability to easily change them later on. ## Test plan Make sure existing tests pass. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: commit 63f7e47af0765ca7c6f318bb03e90c2ca6eb399a Author: Adam Welc <adam@mystenlabs.com> Date: Thu Aug 29 17:01:54 2024 -0700 [move-ide] A fix to empty variant autocompletion (#19150) ## Description This fixes a discrepancy in auto-completion behavior between when an identifier starts with a lower-case character and upper-case character. Prior to this PR, the in the following code fragment, auto-completion in `foo` would offer `bar` as a suggestion (even though `bar` starts with lower case and `B` is upper case, but that's VSCode not filtering on capital letters): ``` module test::M1 { fun bar() :u64 { 42 } fun foo(): u64 { B } } ``` It would however not offer the same suggestion in the following code fragment: ``` module test::M1 { fun bar() :u64 { 42 } fun foo(): u64 { b } } ``` The reason for it was that I…

## Description Changes in this uhaul PR includes: - Addition of `objects_version` table - Addition of a number of transaction and event indices tables, with the removal of `tx_calls` table. - Enable pruning for the newly added indices tables and make tx sequence number a valid partition range, in addition to cp sequence number. ## Test plan tested locally --- ## Release notes This PR modifies the indexer db schemas for improving GraphQL query performance. Specifically, an `objects_version` table along with various transaction and events lookup tables are added. `tx_calls` table is removed by more fine-grained tables `tx_calls_pkg`, `tx_calls_mod` and `tx_calls_fun`. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com>

Reverts #18899

## Description the issue was that, prev indexer db reset was done via dropping all tables, which is problematic when we change a PG PROCEDURE parameter, see this slack message. https://mysten-labs.slack.com/archives/C03TCGDF45N/p1723507055114959 this caused issues on CI after merging #18899 and it got reverted, this pr changes it to reverting all migrations and cleans up the table dropping codes ## Test plan locally - reset DB before #18899 - cherry-pick this pr - cherry-pick #18899 run cmd below, which was the cmd on CI that ran into issue ``` DB_POOL_SIZE=10 cargo run --bin sui-indexer -- --db-url "postgres://postgres:postgres@localhost/gegao" --reset-db ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

## Description title, this is another attempt of #18899 which got reverted as it triggered some CI issues, the issues have been resolved by #18993 ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API: --------- Co-authored-by: Emma Zhong <emma@mystenlabs.com> Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com> Co-authored-by: Emma Zhong <emmazhongjy@gmail.com>

## Description Move the release notes validator into its own workflow ## Test plan ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py check 19148 Found issues with release notes in 19148: - 'Protocol' is checked but has no release note. ``` ![Screenshot 2024-08-30 at 9 07 27 AM](https://github.com/user-attachments/assets/7665bad5-4b36-4faf-9e41-8195dc935156) ``` eugene@eugene-dev ~/code/sui/ (ebmifa/fix_release_notes) $ ./scripts/release_notes.py generate releases/sui-v1.31.0-release releases/sui-v1.32.0-release ## Protocol #### Sui Protocol Version in this release: `55` #19031: Enable Move enums in mainnet ## Indexer #18899: .... ``` --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [x] Protocol: - [x] Nodes (Validators and Full nodes): and here - [x] Indexer: and here - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:

emmazzz and others added 9 commits August 2, 2024 10:09

[indexer] add pruning of event indices

2345023

vercel bot deployed to Preview – sui-docs August 3, 2024 01:22 View deployment

emmazzz marked this pull request as ready for review August 3, 2024 01:43

emmazzz requested review from amnn, wlmyng, stefan-mysten and suiwombat as code owners August 3, 2024 01:43

emmazzz commented Aug 3, 2024

View reviewed changes

emmazzz requested a review from gegaowp August 3, 2024 01:54

gegaowp approved these changes Aug 9, 2024

View reviewed changes

[indexer] fix mysql clippy issues

cec31d3

vercel bot deployed to Preview – sui-docs August 11, 2024 11:32 View deployment

gegaowp merged commit 212f3de into main Aug 12, 2024
48 checks passed

gegaowp deleted the idx-breaking-change-park branch August 12, 2024 20:06

ebmifa added a commit that referenced this pull request Aug 13, 2024

Revert "[indexer] merge in indexer breaking change park (#18899)"

f7c0775

This reverts commit 212f3de.

ebmifa mentioned this pull request Aug 13, 2024

Revert "[indexer] merge in indexer breaking change park" #18975

Merged

ebmifa added a commit that referenced this pull request Aug 13, 2024

Revert "[indexer] merge in indexer breaking change park" (#18975)

fe922b7

Reverts #18899

gegaowp restored the idx-breaking-change-park branch August 14, 2024 19:07

gegaowp mentioned this pull request Aug 14, 2024

indexer fix: reset db via reverting migrations #18993

Merged

8 tasks

gegaowp mentioned this pull request Aug 15, 2024

indexer 2024-08: merge idx-breaking-change-park to main #19005

Merged

8 tasks

gegaowp mentioned this pull request Aug 16, 2024

indexer fix: reset db via reverting migrations (#18993) #19011

Closed

16 tasks

gegaowp mentioned this pull request Aug 16, 2024

indexer fix: reset db via reverting migrations (#18993) #19012

Merged

16 tasks

suiwombat pushed a commit that referenced this pull request Sep 16, 2024

Revert "[indexer] merge in indexer breaking change park" (#18975)

9d18302

Reverts #18899

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[indexer] merge in indexer breaking change park #18899

[indexer] merge in indexer breaking change park #18899

emmazzz commented Aug 3, 2024 •

edited by ronny-mysten

Loading

vercel bot commented Aug 3, 2024 •

edited

Loading

emmazzz commented Aug 3, 2024

emmazzz Aug 3, 2024

wlmyng Aug 3, 2024

emmazzz Aug 9, 2024

emmazzz commented Aug 9, 2024

[indexer] merge in indexer breaking change park #18899

[indexer] merge in indexer breaking change park #18899

Conversation

emmazzz commented Aug 3, 2024 • edited by ronny-mysten Loading

Description

Test plan

Release notes

vercel bot commented Aug 3, 2024 • edited Loading

emmazzz commented Aug 3, 2024

emmazzz Aug 3, 2024

Choose a reason for hiding this comment

wlmyng Aug 3, 2024

Choose a reason for hiding this comment

emmazzz Aug 9, 2024

Choose a reason for hiding this comment

emmazzz commented Aug 9, 2024

emmazzz commented Aug 3, 2024 •

edited by ronny-mysten

Loading

vercel bot commented Aug 3, 2024 •

edited

Loading