-
Notifications
You must be signed in to change notification settings - Fork 11.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[indexer] merge in indexer breaking change park #18899
Commits on Aug 2, 2024
-
[indexer] primary key of tx_calls should include module and func (#17602
) ## Description One transaction can call multiple functions in a package so the primary key should include all parts of a function - package, module, func. ## Test plan Existing tests. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for bc5a802 - Browse repository at this point
Copy the full SHA bc5a802View commit details -
[Indexer] Create
objects_version
table. (#17542)This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into `objects_history` by offering a path for a first look-up to the correct partition in that table for a given object's ID and version. This PR introduces the table, and the logic to populate it in the indexer. ``` sui$ cargo nextest run -p sui-indexer sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration ``` A future PR will make use of this table from GraphQL, which will test it further. - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for 634790a - Browse repository at this point
Copy the full SHA 634790aView commit details -
[Indexer] Index Package original ID, version, cp sequence number (#17726
) ## Description Recreating #17690 which was accidentally closed. Adding data to the `packages` table, to support the following GraphQL queries: ```graphql type Query { # Fetch all packages created strictly `afterCheckpoint` and strictly # before `beforeCheckpoint`. packages( afterCheckpoint: Int, beforeCheckpoint: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch all packages in the same family as the package at `address` # with versions strictly after `afterVersion` and strictly before # `beforeVersion`. packageVersions( address: SuiAddress!, afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch a package by its address, and optionally supplying a # version. If the version is supplied, returns the package whose # Original ID matches the Original ID of the package at `address`, # but whose version is `version`, otherwise just fetches the package # directly. package(address: SuiAddress!, version: Int): MovePackage } type MovePackage { # Return the package whose Original ID matches this package's but # whose version matches `version`. If `version` is not supplied, # defaults to the latest version. atVersion(version: Int): MovePackage # Fetch all packages in the same family as this package, with # versions strictly after `afterVersion` and strictly before # `beforeVersion`. versions( afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection } ``` These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry. ## Test plan Make sure nothing is broken: ``` sui$ cargo nextest run -p sui-indexer ``` Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: Adds the following fields: `packages.original_id`, `packages.package_version`, `packages.checkpoint_sequence_number` to support queries about package upgrades. - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for 96b9f13 - Browse repository at this point
Copy the full SHA 96b9f13View commit details -
[indexer] fix backfilling of objects snapshot to include all columns (#…
…17705) ## Description As titled ## Test plan existing tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for ea96b9c - Browse repository at this point
Copy the full SHA ea96b9cView commit details -
[1+2/n][gql-performance][pg] Update tx lookup tables-related schema f…
…iles for pg-backed indexer per transactions benchmark findings (#18231) Apply findings from latest transactions benchmarking. Instead of keeping indexes on `transactions`, we offload all filtering work to the lookup tables. Additionally, we add `tx_calls_pkg` and `tx_calls_mod` and dedicate `tx_calls_fun` to solely handle filters on `package`, `package` and `module`, and `package`, `module`, and `func` respectively. This is because it's possible to invoke the same package, mod, and even func in a transaction. Finally, the new setup drops `cp_sequence_number` from the lookup tables and adds `sender` to them to support `sender` + specialized filter. The Rust structs and code to index data to the lookup tables are also updated. The full stack of changes are listed below, but each can stand independently, and will be individually merged into `idx-breaking-change-park` 1. #18224 - handling tables partitioned on epoch's worth of tx_sequence_number 2. #18244 - storing a mapping of checkpoint -> tx_sequence_number range --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for dc5cc59 - Browse repository at this point
Copy the full SHA dc5cc59View commit details -
[4/n][gql-performance] Indexer handles partitioning per epoch's worth…
… of tx_sequence_number (#18224) To partition an epoch per `tx_sequence_number`, the last `tx_sequence_number` of an epoch is the `network_total_transactions` from the last checkpoint in the current epoch. The first `tx_sequence_number` is then the `network_total_transactions` from the last checkpoint in the previous epoch + 1, or the `network_total_transactions` of the current epoch - `epoch_total_transactions` of the current epoch. Since we have `network_total_transactions` from the checkpoint to commit, and `epoch_total_transactions` since we fetch current epoch from db, we can derive the range without additional db fetches. However, this approach uncovered a bug of sorts: when we read the network total txn of epoch X - 2 from the db on the indexing side, we `select max(network_total_transactions) from checkpoints where epoch = {epoch}`. However, we may not have actually committed epoch X-2's data on the committer side. This is unlikely to happen in prod because it requires the lag between indexing and committing to be >= one epoch. Since an epoch today consists of ~80k checkpoints, it is quite improbably for this to occur. However, in many of our tests it's possible to have as little as one checkpoint and/ or transaction between epochs, which brought this race condition to light. To resolve, we can place the responsibility on `persist_epoch`. When we get to a point to persist epoch X - 1, epoch X - 2's data must have been indexed to db. Existing tests --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for e4246e5 - Browse repository at this point
Copy the full SHA e4246e5View commit details
Commits on Aug 3, 2024
-
[3/n][gql-performance] add min_tx_sequence_number and max_tx_sequence…
…_number to checkpoints table (#18244) Per title - add `min_tx_sequence_number` and `max_tx_sequence_number` to `checkpoints` table to support queries mapping `cp` to a `tx` range. This is so that when given a `checkpoint_sequence_number` range, we can map it to the corresponding `tx_sequence_number` range to facilitate transactions queries. The two new columns are nullable, because it is possible for a checkpoint to not contain any transactions. In practice, we'll typically have at least a commit prologue txn. The nullable columns do not impact the queries we intend to run on the table. If through a CTE, a null value will eventually resolve to FALSE when we attempt to use the min or max tx sequence number in a query against the transactions and tx lookup tables. If we make multiple roundtrips, we can stop early if we see that the range is empty for a checkpoint. Existing tests --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for e498889 - Browse repository at this point
Copy the full SHA e498889View commit details -
[indexer][gql-perf] add event indices (#18352)
Add event index tables to indexer db for graphql perf. The corresponding script used for investigation is here: https://github.com/MystenLabs/graphql-benchmark/pull/23/files tested locally against devnet --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: --------- Co-authored-by: Ashok Menon <ashok@mystenlabs.com> Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 1d6d789 - Browse repository at this point
Copy the full SHA 1d6d789View commit details -
[indexer] ignore tables not partitioned by epoch at epoch change (#18410
) If a table is partitioned by not by epochs, we shouldn't do anything to it when advancing epochs. Tested locally. --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:
Configuration menu - View commit details
-
Copy full SHA for aa76e71 - Browse repository at this point
Copy the full SHA aa76e71View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2345023 - Browse repository at this point
Copy the full SHA 2345023View commit details
Commits on Aug 11, 2024
-
Configuration menu - View commit details
-
Copy full SHA for cec31d3 - Browse repository at this point
Copy the full SHA cec31d3View commit details