Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[indexer] merge in indexer breaking change park #18899

Merged
merged 11 commits into from
Aug 12, 2024
Merged

Commits on Aug 2, 2024

  1. [indexer] primary key of tx_calls should include module and func (#17602

    )
    
    ## Description 
    
    One transaction can call multiple functions in a package so the primary
    key should include all parts of a function - package, module, func.
    
    ## Test plan 
    
    Existing tests.
    
    ---
    
    ## Release notes
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol: 
    - [ ] Nodes (Validators and Full nodes): 
    - [x] Indexer: 
    - [ ] JSON-RPC: 
    - [ ] GraphQL: 
    - [ ] CLI: 
    - [ ] Rust SDK:
    emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    bc5a802 View commit details
    Browse the repository at this point in the history
  2. [Indexer] Create objects_version table. (#17542)

    This table maps an object's ID and version to a checkpoint sequence
    number, in a table partitioned by the first byte of the object ID. This
    speeds up look ups into `objects_history` by offering a path for a first
    look-up to the correct partition in that table for a given object's ID
    and version.
    
    This PR introduces the table, and the logic to populate it in the
    indexer.
    
    ```
    sui$ cargo nextest run -p sui-indexer
    sui$ cargo nextest run -p sui-graphql-rpc
    sui$ cargo nextest run -p sui-graphql-e2e-tests --features pg_integration
    ```
    
    A future PR will make use of this table from GraphQL, which will test it
    further.
    
    - #17686
    - #17687
    - #17688
    - #17689
    - #17691
    - #17694
    - #17695
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    amnn authored and emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    634790a View commit details
    Browse the repository at this point in the history
  3. [Indexer] Index Package original ID, version, cp sequence number (#17726

    )
    
    ## Description
    
    Recreating #17690 which was accidentally closed.
    
    Adding data to the `packages` table, to support the following GraphQL
    queries:
    
    ```graphql
    type Query {
      # Fetch all packages created strictly `afterCheckpoint` and strictly
      # before `beforeCheckpoint`.
      packages(
        afterCheckpoint: Int,
        beforeCheckpoint: Int,
        first: Int,
        after: String,
        last: Int,
        before: String,
      ): MovePackageConnection
    
      # Fetch all packages in the same family as the package at `address`
      # with versions strictly after `afterVersion` and strictly before 
      # `beforeVersion`.
      packageVersions(
        address: SuiAddress!,
        afterVersion: Int,
        beforeVersion: Int,
        first: Int,
        after: String,
        last: Int,
        before: String,
      ): MovePackageConnection
    
      # Fetch a package by its address, and optionally supplying a
      # version. If the version is supplied, returns the package whose
      # Original ID matches the Original ID of the package at `address`,
      # but whose version is `version`, otherwise just fetches the package
      # directly.
      package(address: SuiAddress!, version: Int): MovePackage
    }
    
    type MovePackage {
      # Return the package whose Original ID matches this package's but
      # whose version matches `version`. If `version` is not supplied,
      # defaults to the latest version.
      atVersion(version: Int): MovePackage
    
      # Fetch all packages in the same family as this package, with
      # versions strictly after `afterVersion` and strictly before
      # `beforeVersion`.
      versions(
        afterVersion: Int,
        beforeVersion: Int,
        first: Int,
        after: String,
        last: Int,
        before: String,
      ): MovePackageConnection
    }
    ```
    
    These queries are important for writing tools that perform whole-chain
    package analyses, and also for the .move Registry.
    
    ## Test plan
    
    Make sure nothing is broken:
    
    ```
    sui$ cargo nextest run -p sui-indexer
    ```
    
    Tests of new features will be included in a stacked change that uses
    these tables and indices in GraphQL.
    
    ## Stack
    
    - #17686 
    - #17687 
    - #17688 
    - #17689 
    - #17691
    - #17694 
    - #17695 
    - #17542  
    
    ---
    
    ## Release notes
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol: 
    - [ ] Nodes (Validators and Full nodes): 
    - [x] Indexer: Adds the following fields: `packages.original_id`,
    `packages.package_version`, `packages.checkpoint_sequence_number` to
    support queries about package upgrades.
    - [ ] JSON-RPC: 
    - [ ] GraphQL: 
    - [ ] CLI: 
    - [ ] Rust SDK:
    amnn authored and emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    96b9f13 View commit details
    Browse the repository at this point in the history
  4. [indexer] fix backfilling of objects snapshot to include all columns (#…

    …17705)
    
    ## Description 
    
    As titled
    
    ## Test plan 
    
    existing tests
    ---
    
    ## Release notes
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol: 
    - [ ] Nodes (Validators and Full nodes): 
    - [ ] Indexer: 
    - [ ] JSON-RPC: 
    - [ ] GraphQL: 
    - [ ] CLI: 
    - [ ] Rust SDK:
    emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    ea96b9c View commit details
    Browse the repository at this point in the history
  5. [1+2/n][gql-performance][pg] Update tx lookup tables-related schema f…

    …iles for pg-backed indexer per transactions benchmark findings (#18231)
    
    Apply findings from latest transactions benchmarking. Instead of keeping
    indexes on `transactions`, we offload all filtering work to the lookup
    tables. Additionally, we add `tx_calls_pkg` and `tx_calls_mod` and
    dedicate `tx_calls_fun` to solely handle filters on `package`, `package`
    and `module`, and `package`, `module`, and `func` respectively. This is
    because it's possible to invoke the same package, mod, and even func in
    a transaction.
    
    Finally, the new setup drops `cp_sequence_number` from the lookup tables
    and adds `sender` to them to support `sender` + specialized filter. The
    Rust structs and code to index data to the lookup tables are also
    updated.
    
    The full stack of changes are listed below, but each can stand
    independently, and will be individually merged into
    `idx-breaking-change-park`
    1. #18224 - handling tables
    partitioned on epoch's worth of tx_sequence_number
    2. #18244 - storing a mapping of
    checkpoint -> tx_sequence_number range
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    wlmyng authored and emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    dc5cc59 View commit details
    Browse the repository at this point in the history
  6. [4/n][gql-performance] Indexer handles partitioning per epoch's worth…

    … of tx_sequence_number (#18224)
    
    To partition an epoch per `tx_sequence_number`, the last
    `tx_sequence_number` of an epoch is the `network_total_transactions`
    from the last checkpoint in the current epoch. The first
    `tx_sequence_number` is then the `network_total_transactions` from the
    last checkpoint in the previous epoch + 1, or the
    `network_total_transactions` of the current epoch -
    `epoch_total_transactions` of the current epoch.
    
    Since we have `network_total_transactions` from the checkpoint to
    commit, and `epoch_total_transactions` since we fetch current epoch from
    db, we can derive the range without additional db fetches.
    
    However, this approach uncovered a bug of sorts: when we read the
    network total txn of epoch X - 2 from the db on the indexing side, we
    `select max(network_total_transactions) from checkpoints where epoch =
    {epoch}`. However, we may not have actually committed epoch X-2's data
    on the committer side. This is unlikely to happen in prod because it
    requires the lag between indexing and committing to be >= one epoch.
    Since an epoch today consists of ~80k checkpoints, it is quite
    improbably for this to occur. However, in many of our tests it's
    possible to have as little as one checkpoint and/ or transaction between
    epochs, which brought this race condition to light.
    
    To resolve, we can place the responsibility on `persist_epoch`. When we
    get to a point to persist epoch X - 1, epoch X - 2's data must have been
    indexed to db.
    
    Existing tests
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    wlmyng authored and emmazzz committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    e4246e5 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2024

  1. [3/n][gql-performance] add min_tx_sequence_number and max_tx_sequence…

    …_number to checkpoints table (#18244)
    
    Per title - add `min_tx_sequence_number` and `max_tx_sequence_number` to
    `checkpoints` table to support queries mapping `cp` to a `tx` range.
    This is so that when given a `checkpoint_sequence_number` range, we can
    map it to the corresponding `tx_sequence_number` range to facilitate
    transactions queries.
    
    The two new columns are nullable, because it is possible for a
    checkpoint to not contain any transactions. In practice, we'll typically
    have at least a commit prologue txn. The nullable columns do not impact
    the queries we intend to run on the table. If through a CTE, a null
    value will eventually resolve to FALSE when we attempt to use the min or
    max tx sequence number in a query against the transactions and tx lookup
    tables. If we make multiple roundtrips, we can stop early if we see that
    the range is empty for a checkpoint.
    
    Existing tests
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    wlmyng authored and emmazzz committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    e498889 View commit details
    Browse the repository at this point in the history
  2. [indexer][gql-perf] add event indices (#18352)

    Add event index tables to indexer db for graphql perf. The corresponding
    script used for investigation is here:
    https://github.com/MystenLabs/graphql-benchmark/pull/23/files
    
    tested locally against devnet
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    
    ---------
    
    Co-authored-by: Ashok Menon <ashok@mystenlabs.com>
    Co-authored-by: wlmyng <127570466+wlmyng@users.noreply.github.com>
    3 people committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    1d6d789 View commit details
    Browse the repository at this point in the history
  3. [indexer] ignore tables not partitioned by epoch at epoch change (#18410

    )
    
    If a table is partitioned by not by epochs, we shouldn't do anything to
    it when advancing epochs.
    
    Tested locally.
    
    ---
    
    Check each box that your changes affect. If none of the boxes relate to
    your changes, release notes aren't required.
    
    For each box you select, include information after the relevant heading
    that describes the impact of your changes that a user might notice and
    any actions they must take to implement updates.
    
    - [ ] Protocol:
    - [ ] Nodes (Validators and Full nodes):
    - [ ] Indexer:
    - [ ] JSON-RPC:
    - [ ] GraphQL:
    - [ ] CLI:
    - [ ] Rust SDK:
    emmazzz committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    aa76e71 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2345023 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2024

  1. Configuration menu
    Copy the full SHA
    cec31d3 View commit details
    Browse the repository at this point in the history