[Indexer] Index Package original ID, version, cp sequence number #17726

amnn · 2024-05-14T11:43:33Z

Description

Recreating #17690 which was accidentally closed.

Adding data to the packages table, to support the following GraphQL queries:

type Query {
  # Fetch all packages created strictly `afterCheckpoint` and strictly
  # before `beforeCheckpoint`.
  packages(
    afterCheckpoint: Int,
    beforeCheckpoint: Int,
    first: Int,
    after: String,
    last: Int,
    before: String,
  ): MovePackageConnection

  # Fetch all packages in the same family as the package at `address`
  # with versions strictly after `afterVersion` and strictly before 
  # `beforeVersion`.
  packageVersions(
    address: SuiAddress!,
    afterVersion: Int,
    beforeVersion: Int,
    first: Int,
    after: String,
    last: Int,
    before: String,
  ): MovePackageConnection

  # Fetch a package by its address, and optionally supplying a
  # version. If the version is supplied, returns the package whose
  # Original ID matches the Original ID of the package at `address`,
  # but whose version is `version`, otherwise just fetches the package
  # directly.
  package(address: SuiAddress!, version: Int): MovePackage
}

type MovePackage {
  # Return the package whose Original ID matches this package's but
  # whose version matches `version`. If `version` is not supplied,
  # defaults to the latest version.
  atVersion(version: Int): MovePackage

  # Fetch all packages in the same family as this package, with
  # versions strictly after `afterVersion` and strictly before
  # `beforeVersion`.
  versions(
    afterVersion: Int,
    beforeVersion: Int,
    first: Int,
    after: String,
    last: Int,
    before: String,
  ): MovePackageConnection
}

These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry.

Test plan

Make sure nothing is broken:

sui$ cargo nextest run -p sui-indexer

Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL.

Stack

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

Protocol:
Nodes (Validators and Full nodes):
Indexer: Adds the following fields: packages.original_id, packages.package_version, packages.checkpoint_sequence_number to support queries about package upgrades.
JSON-RPC:
GraphQL:
CLI:
Rust SDK:

vercel · 2024-05-14T11:43:37Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 14, 2024 10:26pm

3 Ignored Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:26pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:26pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:26pm

) ## Description Recreating #17690 which was accidentally closed. Adding data to the `packages` table, to support the following GraphQL queries: ```graphql type Query { # Fetch all packages created strictly `afterCheckpoint` and strictly # before `beforeCheckpoint`. packages( afterCheckpoint: Int, beforeCheckpoint: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch all packages in the same family as the package at `address` # with versions strictly after `afterVersion` and strictly before # `beforeVersion`. packageVersions( address: SuiAddress!, afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch a package by its address, and optionally supplying a # version. If the version is supplied, returns the package whose # Original ID matches the Original ID of the package at `address`, # but whose version is `version`, otherwise just fetches the package # directly. package(address: SuiAddress!, version: Int): MovePackage } type MovePackage { # Return the package whose Original ID matches this package's but # whose version matches `version`. If `version` is not supplied, # defaults to the latest version. atVersion(version: Int): MovePackage # Fetch all packages in the same family as this package, with # versions strictly after `afterVersion` and strictly before # `beforeVersion`. versions( afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection } ``` These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry. ## Test plan Make sure nothing is broken: ``` sui$ cargo nextest run -p sui-indexer ``` Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: Adds the following fields: `packages.original_id`, `packages.package_version`, `packages.checkpoint_sequence_number` to support queries about package upgrades. - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into `objects_history` by offering a path for a first look-up to the correct partition in that table for a given object's ID and version. This PR introduces the table, and the logic to populate it in the indexer. ## Test plan ``` sui$ cargo nextest run -p sui-indexer sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration ``` A future PR will make use of this table from GraphQL, which will test it further. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

) ## Description Recreating #17690 which was accidentally closed. Adding data to the `packages` table, to support the following GraphQL queries: ```graphql type Query { # Fetch all packages created strictly `afterCheckpoint` and strictly # before `beforeCheckpoint`. packages( afterCheckpoint: Int, beforeCheckpoint: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch all packages in the same family as the package at `address` # with versions strictly after `afterVersion` and strictly before # `beforeVersion`. packageVersions( address: SuiAddress!, afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch a package by its address, and optionally supplying a # version. If the version is supplied, returns the package whose # Original ID matches the Original ID of the package at `address`, # but whose version is `version`, otherwise just fetches the package # directly. package(address: SuiAddress!, version: Int): MovePackage } type MovePackage { # Return the package whose Original ID matches this package's but # whose version matches `version`. If `version` is not supplied, # defaults to the latest version. atVersion(version: Int): MovePackage # Fetch all packages in the same family as this package, with # versions strictly after `afterVersion` and strictly before # `beforeVersion`. versions( afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection } ``` These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry. ## Test plan Make sure nothing is broken: ``` sui$ cargo nextest run -p sui-indexer ``` Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: Adds the following fields: `packages.original_id`, `packages.package_version`, `packages.checkpoint_sequence_number` to support queries about package upgrades. - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Use the `objects_version` table to speed up point look-ups (via data loaders) for historical objects (ID + version), and dynamic fields (object look-up bounding version by parent ID). With this change, the restriction of accessing dynamic fields only within the available range is dropped. ## Test plan ``` sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration. ``` Perform a query that involves fetching a large number of dynamic fields, which should now be fast. The following example, fetching dynamic fields on a deepbook pool loads 50 dynamic fields in about 5s from cold (which also requires loading packages for resolution), and then 2s from there: ``` query { owner( address: "0x029170bfa0a1677054263424fe4f9960c7cf05d359f6241333994c8830772bdb" ) { dynamicFields { pageInfo { hasNextPage endCursor } nodes { name { type { repr } json } value { ... on MoveValue { type { repr } json } ... on MoveObject { contents { json type { repr } } } } } } } } ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Dynamic fields can now be looked up on any historical object (not just objects in the available range). - [ ] CLI: - [ ] Rust SDK:

## Description Implement `Query.package` and `MovePackage.atVersion` to query a package at a specific version, using the new fields added to the `packages` table, exposed via some new data loaders. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduce `Query.package` and `MovePackage.atVersion` to query packages at specific versions. - [ ] CLI: - [ ] Rust SDK:

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK: