[Indexer] Create `objects_version` table. #17542

amnn · 2024-05-07T13:25:10Z

Description

This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into objects_history by offering a path for a first look-up to the correct partition in that table for a given object's ID and version.

This PR introduces the table, and the logic to populate it in the indexer.

Test plan

sui$ cargo nextest run -p sui-indexer
sui$ cargo nextest run -p sui-graphql-rpc
sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration

A future PR will make use of this table from GraphQL, which will test it further.

Stack

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2024-05-07T13:25:13Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 14, 2024 10:25pm

3 Ignored Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:25pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:25pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	May 14, 2024 10:25pm

amnn

@gegaowp -- what's the standard procedure for where we land schema changes and how we ensure the backfill for them happens?

amnn · 2024-05-07T13:26:36Z

crates/sui-indexer/migrations/mysql/2024-05-05-155158_obj_indices/down.sql

@@ -0,0 +1,2 @@
+-- This file should undo anything in `up.sql`
+-- TODO: objects_version


I left these blank for now, but let me know what the best way to handle this is @sadhansood (whether I should fill this out or leave it blank, what the best way is to test it, etc).

I noticed that the Indexer tests don't run these migrations (at least on the Postgres side, I have to run the GraphQL E2E tests to test this kind of code), so I'm nervous to touch this without knowing how to make sure it's working.

amnn · 2024-05-07T13:27:06Z

crates/sui-indexer/migrations/pg/2024-05-05-155158_obj_indices/up.sql

This is very ...manual. @gegaowp, let me know if there is a more sensible way of setting up these partitions.

we prob can write a script to create all the partitions, maybe except the last one for max value

DO $$ BEGIN WHILE i < 256 LOOP start_value := ... end_value := ... EXECUTE FORMAT('CREATE TABLE objects_version_%s PARTITION OF objects_version FOR VALUES FROM (%L) TO (%L);', i, start_value, end_value); i := i + 1; END LOOP; END $$;

crates/sui-indexer/src/models/obj_indices.rs

gegaowp

overall LG, however please do not merge to main just yet, as this is a breaking change, and the steps for indexer breaking change is listed here
https://mysten-labs.slack.com/archives/C0578KFD9D2/p1714510858042569

gegaowp · 2024-05-08T22:11:51Z

crates/sui-indexer/migrations/pg/2024-05-05-155158_obj_indices/up.sql

we prob can write a script to create all the partitions, maybe except the last one for max value

DO $$ BEGIN WHILE i < 256 LOOP start_value := ... end_value := ... EXECUTE FORMAT('CREATE TABLE objects_version_%s PARTITION OF objects_version FOR VALUES FROM (%L) TO (%L);', i, start_value, end_value); i := i + 1; END LOOP; END $$;

amnn · 2024-05-09T10:44:33Z

Thanks @gegaowp ! So should I land this to idx-breaking-change-park instead? Couple of follow up questions:

I noticed the instructions suggest that the parking branch will be reset to main when main has been updated with its changes. IIRC main is now updated, but the parking branch is not up-to-date yet. Is it safe to update it now?
Is there some standard/regular process for picking up changes to be backfilled, (i.e. is it enough for me to lang changes into this branch, or is there more coordination required to make sure it gets into the indexer release).
I noticed that the uhaulPR is not currently part of the 2024.4 GraphQL release -- I'm planning on cherry-picking it there, so that the Tx Digest changes can go out with the next release -- do you foresee any issues?

amnn · 2024-05-09T11:10:07Z

I noticed that the uhaulPR is not currently part of the 2024.4 GraphQL release -- I'm planning on cherry-picking it there, so that the Tx Digest changes can go out with the next release -- do you foresee any issues?

#17605

gegaowp · 2024-05-09T13:45:48Z

IIRC main is now updated, but the parking branch is not up-to-date yet. Is it safe to update it now?

@amnn sry and yes, I updated the idx-breaking-change-park branch to the latest main.

Is there some standard/regular process for picking up changes to be backfilled

not really for now, as backfill will take about a week, so if we know we have some coming schema changes too, we can batch them together, I saw Emma and Will have some incoming schema changes.

Tx Digest changes can go out with the next release -- do you foresee any issues?

it's more likely to fix things rather than breaking afaict.

amnn · 2024-05-09T14:19:09Z

Okay, great! I also have some more schema changes to go in, related to the package table, so I'll move this to target the new parking branch, thanks @gegaowp ! (And can I assume that it's safe to land into the parking branch?)

Tx Digest changes can go out with the next release -- do you foresee any issues?

it's more likely to fix things rather than breaking afaict.

😅, do we already know of issues related to this being stale? (I guess if we landed the optimisations on the GraphQL side that use the new per-type tables into 2024.4 then we might have an issue, but if we haven't yet, then we probably should cc @emmazzz) Hopefully it shouldn't cause any issues not to land it, because that would imply that the legacy release will also face issues (it would be a pain to pick the uhaul there because we don't have the changes related to removing the v1 schema, etc).

This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into `objects_history` by offering a path for a first look-up to the correct partition in that table for a given object's ID and version. This PR introduces the table, and the logic to populate it in the indexer. ``` sui$ cargo nextest run -p sui-indexer sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration ``` A future PR will make use of this table from GraphQL, which will test it further. - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

) ## Description Recreating #17690 which was accidentally closed. Adding data to the `packages` table, to support the following GraphQL queries: ```graphql type Query { # Fetch all packages created strictly `afterCheckpoint` and strictly # before `beforeCheckpoint`. packages( afterCheckpoint: Int, beforeCheckpoint: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch all packages in the same family as the package at `address` # with versions strictly after `afterVersion` and strictly before # `beforeVersion`. packageVersions( address: SuiAddress!, afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection # Fetch a package by its address, and optionally supplying a # version. If the version is supplied, returns the package whose # Original ID matches the Original ID of the package at `address`, # but whose version is `version`, otherwise just fetches the package # directly. package(address: SuiAddress!, version: Int): MovePackage } type MovePackage { # Return the package whose Original ID matches this package's but # whose version matches `version`. If `version` is not supplied, # defaults to the latest version. atVersion(version: Int): MovePackage # Fetch all packages in the same family as this package, with # versions strictly after `afterVersion` and strictly before # `beforeVersion`. versions( afterVersion: Int, beforeVersion: Int, first: Int, after: String, last: Int, before: String, ): MovePackageConnection } ``` These queries are important for writing tools that perform whole-chain package analyses, and also for the .move Registry. ## Test plan Make sure nothing is broken: ``` sui$ cargo nextest run -p sui-indexer ``` Tests of new features will be included in a stacked change that uses these tables and indices in GraphQL. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [x] Indexer: Adds the following fields: `packages.original_id`, `packages.package_version`, `packages.checkpoint_sequence_number` to support queries about package upgrades. - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

This table maps an object's ID and version to a checkpoint sequence number, in a table partitioned by the first byte of the object ID. This speeds up look ups into `objects_history` by offering a path for a first look-up to the correct partition in that table for a given object's ID and version. This PR introduces the table, and the logic to populate it in the indexer. ``` sui$ cargo nextest run -p sui-indexer sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration ``` A future PR will make use of this table from GraphQL, which will test it further. - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 --- Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

## Description Use the `objects_version` table to speed up point look-ups (via data loaders) for historical objects (ID + version), and dynamic fields (object look-up bounding version by parent ID). With this change, the restriction of accessing dynamic fields only within the available range is dropped. ## Test plan ``` sui$ cargo nextest run -p sui-graphql-rpc sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration. ``` Perform a query that involves fetching a large number of dynamic fields, which should now be fast. The following example, fetching dynamic fields on a deepbook pool loads 50 dynamic fields in about 5s from cold (which also requires loading packages for resolution), and then 2s from there: ``` query { owner( address: "0x029170bfa0a1677054263424fe4f9960c7cf05d359f6241333994c8830772bdb" ) { dynamicFields { pageInfo { hasNextPage endCursor } nodes { name { type { repr } json } value { ... on MoveValue { type { repr } json } ... on MoveObject { contents { json type { repr } } } } } } } } ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Dynamic fields can now be looked up on any historical object (not just objects in the available range). - [ ] CLI: - [ ] Rust SDK:

## Description Implement `Query.package` and `MovePackage.atVersion` to query a package at a specific version, using the new fields added to the `packages` table, exposed via some new data loaders. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduce `Query.package` and `MovePackage.atVersion` to query packages at specific versions. - [ ] CLI: - [ ] Rust SDK:

## Description Add a new kind of package point look-up to get the latest version of the package at a given ID (or from another `MovePackage`). For system packages, this is analogous to getting the latest version of the object at that ID, but the versions of other packages all exist at different IDs. ## Test plan New transactional tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Add `Query.latestPackage` and `MovePackage.latest` for fetching the latest version of a package. - [ ] CLI: - [ ] Rust SDK:

## Description Adds a query, `Query.packages` for fetching all packages that were introduced within a given checkpoint range. Useful for fetching package contents in bulk, to do local analyses. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` Also tested for performance against a large read replica (the query planner quotes a high estimate for the query but the actual results do not take very long to run because queries on many sub-partitions are eliminated). ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17726 - #17543 - #17692 - #17693 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packages` for paginating through all packages (optionally bounding by the checkpoint the package was introduced in). - [ ] CLI: - [ ] Rust SDK:

## Description Introduce two new queries: `Query.packageVersions` and `MovePackage.versions` for iterating over all the different versions of a given package. This kind of query is useful for understanding package history. These were introduced as a separate query, instead of having a single query for iterating over packages that could optionally take a checkpoint bounds or version bounds because of how system packages interact with the `packages` table: Because system packages are updated in-place, they only have one row in the `packages` table. This makes sense for paginating packages in bulk (e.g. by checkpoint) where the primary aim is to get a snapshot of the packages available at a certain point in time, but doesn't work for answering package version queries for system packages, and it prevents us from creating a combined query. A combined query would also allow someone to create a filter that bounds checkpoints and versions, but doesn't bound the package itself (or would require us to prevent that combination), which is complicated to implement efficiently and not particularly useful. ## Test plan New E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests \ --features pg_integration \ -- packages/versioning ``` & Testing against a read replica to make sure system package tests work well, and performance is reasonable. ## Stack - #17686 - #17687 - #17688 - #17689 - #17691 - #17694 - #17695 - #17542 - #17690 - #17543 - #17692 - #17693 - #17696 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [x] GraphQL: Introduces `Query.packageVersions` and `MovePackage.versions` for paginating over the versions of a particular package. - [ ] CLI: - [ ] Rust SDK: