[GraphQL/MovePackage] Query for latest version #17693

amnn · 2024-05-13T15:35:41Z

Description

Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another MovePackage). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs.

Test plan

New transactional tests:

sui$ cargo nextest run -p sui-graphql-e2e-tests \
  --features pg_integration                     \
  -- packages/versioning

Stack

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2024-05-13T15:35:45Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 19, 2024 10:43am

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:43am
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:43am
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Aug 19, 2024 10:43am

stefan-mysten

Looks good!

wlmyng · 2024-05-13T17:54:56Z

crates/sui-graphql-rpc/src/types/move_package.rs

+        let futures = ids_by_cursor
+            .into_iter()
+            .map(|(checkpoint_viewed_at, ids)| {
+                self.execute(move |conn| {
+                    let results: Vec<(Vec<u8>, Vec<u8>)> = conn.results(|| {


not unique to this particular implementation, but these concurrent reads in our dataloaders may exhaust the database connection pool; it's possible for someone then to cook up some request that will hog up all connections. By default dataloader is configured to coalesce 1k individual loads, so I don't think it'll be too hard to figure out a query that would exceed the pool size configured for our graphql service

This is a good call-out -- it's worth batching these so we only issue 10 reads at a time per data loader (and I think it's also worth upping the default connection limit to like 50 or something).

I'll do this in a follow-up PR that handles all the cases where we do this kind of grouping.

Looked into this. We only ever send out parallel requests when we group requests that could have distinct checkpoint viewed at cursors. This means that we're not introducing more connection parallelism than is already present in the underlying query (the DataLoader sends out at most as many parallel requests as an unoptimised query would).

This means that even if we limited the fan-out within data-loader enabled queries, someone could create the same issue with some other kind of query, which suggests that the right thing to do would be to apply a more generic limit (which we already do -- the input node limit) and tweak that to preserve the connection pool.

wlmyng · 2024-05-13T17:56:23Z

crates/sui-graphql-rpc/src/types/move_package.rs

+                            .select((dsl::package_id, o_package_id))
+                            .filter(dsl::package_id.eq_any(ids.iter().cloned()))
+                            .filter(o_cp_seq_num.le(checkpoint_viewed_at as i64))
+                            .order_by((dsl::package_id, dsl::original_id, o_version.desc()))


i assume this is pretty performant already, but what if we order all 3 in descending order? would that further leverage the index?

I can give it a try!

This doesn't make a big difference, but this comment did lead me to change the primary key of this table to (package_id, original_id, package_version) and add a separate uniqueness constraint on package_id, and that does reduce the cost slightly.

## Description Add a command for generating a config TOML file for the GraphQL service with all its parameters set to their default values. (We used to have a similar command for the YAML file which we weren't using, but we still use the TOML file). ## Test plan ``` cargo run --bin sui-graphql-rpc -- generate-config /tmp/config.toml ``` ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: New sub-command for `sui-graphql-rpc`, `generate-config` for creating a TOML config with all default values set. - [ ] CLI: - [ ] Rust SDK:

## Description Replace the original implementation of the dump-packages command (which requires access to an indexer database) with an implementation that reads from a GraphQL service. The former is not readily accessible, but the latter should be. The new tool is also able to run incrementally: Fetching only packages created before a certain checkpoint, or pick up where a previous invocation took off to fetch new packages that were introduced since. ## Test plan Ran a test invocation, on our experimental read replica. With a max page size of 200, I was able to fetch 17000 packages (all the packages at the time the read replica was created) in 3 minutes. ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 - #18288 - #18336 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17690 - #17543 - #17692 - #17693 - #17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK:

## Description Remove commands to generate examples, markdown and schema from the main binary as we do not use them: - Instead of generating examples, we have hand-crafted examples in our docs. Removing this code also removes a test that forces regeneration of a markdown file from these docs (which we also were not using). - We also never used the output from the `generate-schema` sub-command, because the schema was always available as a file, or via introspection commands from the running service. - Logic for gathering examples to test has been moved into the test file, to avoid including test-only code in the main library. ## Test plan CI ## Description Describe the changes or additions included in this PR. ## Test plan How did you test the new or updated feature? ## Stack - #17543 - #17692 - #17693 - #17696 - #17697 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The GraphQL binary no longer supports generating examples, or exporting its own schema as these commands have been unused for some time. - [ ] CLI: - [ ] Rust SDK:

## Description Remove `draft_target_schema.graphql` and promote `current_progress_schema.graphql` to be the canonical schema for the service -- move it to the top-level of the `sui-graphql-rpc` crate to make it easier to find. This is to avoid confusion about source of truth for the GraphQL schema. Because the TS SDK references the schema at multiple GraphQL versions, we will need to cherry-pick this change to release branches when it lands. ## Test plan CI ## Stack - #17543 - #17692 - #17693 - #17696 - #18287 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: The schema file has been moved from `crates/sui-graphql-rpc/schemas/current_progress_schema.graphql` to `crates/sui-graphql-rpc/schema.graphql`. - [ ] CLI: - [ ] Rust SDK: