docs(graphql): explain scan limits rationale

crates/sui-graphql-rpc/src/types/transaction_block/tx_lookups.rs

@@ -1,6 +1,49 @@
 // Copyright (c) Mysten Labs, Inc.
 // SPDX-License-Identifier: Apache-2.0
 
+//! # Transaction Filter Lookup Tables
+//!
+//! ## Schemas
+//!
+//! Tables backing Transaction filters in GraphQL all follow the same rough shape:
+//!
+//! 1. They each get their own table, mapping the filter value to the transaction sequence number.
+//!
+//! 2. They also include a `sender` columns, and a secondary index over the sender, filter values
+//!    and the transaction sequence number.
+//!
+//! 3. They also include a secondary index over the transaction sequence number.
+//!
+//! ## Query construction
+//!
+//! Queries that filter transactions work in two phases: Identify the transaction sequence numbers
+//! to fetch, and then fetch their contents. Filtering all happens in the first phase:
+//!
+//! - Firstly filters are broken down into individual queries targeting the appropriate lookup
+//!   table. Each constituent query is expected to return a sorted run of transaction sequence
+//!   numbers.
+//!
+//! - If a `sender` filter is included, then it is incorporated into each constituent query,
+//!   leveraging their secondary indices (2), otherwise each constituent query filters only based on
+//!   its filter value using the primary index (1).
+//!
+//! - The fact that both the primary and secondary indices contain the transaction sequence number
+//!   help to ensure that the output from an index scan is already sorted, which avoids a
+//!   potentially expensive materialize and sort operation.
+//!
+//! - If there are multiple constituent queries, they are intersected using inner joins. Postgres
+//!   can occasionally pick a poor query plan for this merge, so we require that filters resulting in
+//!   such merges also use a "scan limit" (see below).
+//!
+//!
+//!
+//! By following this pattern, we are able to give people a simple rule to follow: You can apply
+//! any single filter without worrying, and you can optionally include a sender filter without
+//! worrying, but anything else requires using scan limits. Otherwise it is very difficult for
+//! users to figure out what they can do, based on the schema we advertise, very much like things
+//! are today in JSON-RPC, where the schema implies we support any number of queries that we don't,
+//! for a variety of reasons.
+
 use super::{Cursor, TransactionBlockFilter};
 use crate::{
     data::{pg::bytea_literal, Conn, DbConnection},