Writing in batches #4606

lutter · 2023-05-09T14:19:03Z

This PR changes graph-node so that subgraphs write their changes in batches instead of at every block. During syncing, a batch of changes is only written if it is bigger than a certain size or older than a certain amount of time; while the batch is held, calls to transact_block_operations will append their changes to the current batch. Batching is controlled by the environment variables GRAPH_STORE_WRITE_BATCH_DURATION and GRAPH_STORE_WRITE_BATCH_SIZE. Setting either to 0 will turn off batching and force writes at every block.

The two most difficult parts of the PR are probably the logic in RowGroup.append_row, coupled with how entities are looked up in RowGroup.last_op and RowGroup.effective_ops (I am working on unit tests for that). The other complicated part of this PR is the locking logic in graph::store::postgres::writable::Queue, especially the interplay between start_writer and push_write.

It's strongly recommended to look at this PR commit by commit; the first few commits, up to 'Introduce a data structure to capture a write' just lay some groundwork to make what comes after simpler/possible. After that, commits up to 'Attempt to extend existing write requests' prepare the code base for batches that span multiple blocks; the remaining commits implement the actual logic and control for combining multiple transact_block_operations into one batch.

lutter · 2023-05-11T10:27:53Z

Rebased to latest master

lutter · 2023-05-11T14:15:53Z

Rebased once more

leoyvens

Wonderful ✨

leoyvens · 2023-05-11T14:46:36Z

graph/src/components/store/write.rs

+            .rows
+            .last()
+            .map(|emod| emod.block() <= block)
+            .unwrap_or(true));


I'd make this a hard assert! since it's cheap to run.

I don't like panics in production, but here it was easy to make that a constraint_violation!

leoyvens · 2023-05-13T15:07:08Z

store/postgres/src/writable.rs

-/// The tracker relies on `update` being called in the order newest request
-/// in the queue to oldest request so that reverts are seen before the
-/// writes that they revert.
+/// The best way to use the trtacker is to use the `fold_map` and `find`


Suggested change

/// The best way to use the trtacker is to use the `fold_map` and `find`

/// The best way to use the tracker is to use the `fold_map` and `find`

leoyvens · 2023-05-16T09:55:37Z

graph/src/components/store/write.rs

+                key,
+                end: Some(end),
+                ..
+            } if at < *end => EntityOp::Write { key, entity: data },


This looks correct, but it would be good to have a test that would fail if this were <=.

The unit tests in last_op also fail when this is turned into a <=

leoyvens · 2023-05-16T09:56:02Z

graph/src/components/store/write.rs

-            .filter(move |emod| seen.insert(emod.id()))
-            .map(EntityOp::from)
+            .filter(move |emod| {
+                if emod.block() <= at {


Same here, a test that fails if this were <.

I added a commit with a bunch of unit tests, and checked tha changing to a < here makes them fail

lutter · 2023-05-17T19:40:19Z

Rebased onto latest master

leoyvens

The test is very readable, thanks for adding that

- Columns are now returned in the order in which they are defined, not a random order - In the common case where a column is present in all entities, we don't have to iterate over all entities

The mapping never changes, so it can be set once for the WritableStore, and doesn't need to be passed in for every transact_block_operations

The contents of graph::components::store::write are somewhat provisional, mostly to plumb using a `Batch` through the rest of the code. A later commit will change this quite a bit.

We group all pending updates into runs by block.

When we start combining batches, we will want to mutate some changes in place to avoid the data copies that we would need if we kept them separated by the kind of change that is needed.

When changes to entities get combined, inserting a row can either create a new entity, or update an existing version, and the number of rows inserted is no longer an accurate count of new entities. Instead of relying on data coming from the database, we now use the in-memory representation of changes to determine how applying a batch changes the number of entities in a deployment.

The result of the append is a batch that, when written to the database, has the same effect as writing the two batches in two separate transactions.

When a new write request gets queued, try to append it to an existing one whenever possible, instead of creating a new request.

So far, new write requests were appended to existing ones only opportunistically; this commit makes it so that we actually hold back writing to allow processing to append to an existing write request. Only write a batch if its size is beyond WRITE_BATCH_SIZE or if it is older than WRITE_BATCH_DURATION. Since we don't have a way to wait on the batch growing big enough, poll for that every 2s This also requires that stopping or flushing the WritableStore sets `batch_writes` to `false`, mostly for tests, to ensure that all pending writes get written out before the background writer shuts down.

lutter requested review from leoyvens and mangas May 9, 2023 14:19

lutter force-pushed the lutter/batch-write branch from 67a935f to f653157 Compare May 11, 2023 10:27

lutter force-pushed the lutter/batch-write branch from f653157 to 7985569 Compare May 11, 2023 14:13

leoyvens approved these changes May 16, 2023

View reviewed changes

lutter mentioned this pull request May 17, 2023

Update nonFatalErrors in subgraphs.subgraph_deployment table #4615

Merged

lutter force-pushed the lutter/batch-write branch from 499564f to 249287c Compare May 17, 2023 19:37

leoyvens approved these changes May 18, 2023

View reviewed changes

lutter force-pushed the lutter/batch-write branch 2 times, most recently from 2f7f76a to 5992b54 Compare May 18, 2023 17:19

lutter added 17 commits May 18, 2023 10:46

graph: Make debug output for Entity, EntityKey/Type more readable

eb9c355

graph: Implement StoreError.clone()

0f7cd9d

store: Do not mutate entities for InsertQuery

ad1c6ea

store: Improve InsertQuery::unique_columns

6c3b401

- Columns are now returned in the order in which they are defined, not a random order - In the common case where a column is present in all entities, we don't have to iterate over all entities

store: Remove mutability from entities in various places

1cf11f5

all: Set manifest_idx_and_name when creating WritableStore

379851c

The mapping never changes, so it can be set once for the WritableStore, and doesn't need to be passed in for every transact_block_operations

graph, store: Introduce a data structure to capture a write

c5fde60

The contents of graph::components::store::write are somewhat provisional, mostly to plumb using a `Batch` through the rest of the code. A later commit will change this quite a bit.

store: Allow using a different block for each entity on insert

3803ae7

graph, store: Allow using multiple blocks for updating entities

2f899af

We group all pending updates into runs by block.

graph, store: Allow using multiple blocks for removing entities

6893f18

store: Allow persisting data sources at different blocks

9f3c5aa

graph, store: Keep all pending changes in one large list

12427fd

When we start combining batches, we will want to mutate some changes in place to avoid the data copies that we would need if we kept them separated by the kind of change that is needed.

store: Streamline logic in DeploymentStore.apply_entity_modifications

0f59591

graph, store: Use a struct, not a tuple, to pass around writes

ea7305e

graph, store: Allow inserting entities with an end to their block_range

9e10c9b

store: Encapsulate logic to iterate over visible part of queue better

43fafec

lutter added 10 commits May 18, 2023 10:46

graph: Add appending one batch to another

1848d5e

The result of the append is a batch that, when written to the database, has the same effect as writing the two batches in two separate transactions.

graph, store: Attempt to extend existing write requests

b629a0a

When a new write request gets queued, try to append it to an existing one whenever possible, instead of creating a new request.

graph, store: Limit the size of a write batch by cache weight

e24f285

docs, graph, store: Make write batching configurable

d0cdd43

graph, store: Make RowGroup.rows private

743e189

ci: Reduce batch duration to avoid timeouts

e4d7563

graph: Always check that RowGroup::push does not go backwards

f5b7cb5

graph: Add unit tests for some RowGroup functionality

1df5a66

store: Fix typo in comment

76619a6

lutter force-pushed the lutter/batch-write branch from 5992b54 to 76619a6 Compare May 18, 2023 17:47

lutter merged commit 76619a6 into master May 18, 2023

lutter deleted the lutter/batch-write branch May 18, 2023 18:10

neysofu mentioned this pull request Jun 1, 2023

[Feature] Batched writes during subgraph indexing #4538

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Writing in batches #4606

Writing in batches #4606

lutter commented May 9, 2023

lutter commented May 11, 2023

lutter commented May 11, 2023

leoyvens left a comment

leoyvens May 11, 2023

lutter May 17, 2023

leoyvens May 13, 2023

lutter May 17, 2023

leoyvens May 16, 2023

lutter May 17, 2023

leoyvens May 16, 2023

lutter May 17, 2023

lutter commented May 17, 2023

leoyvens left a comment

	/// The best way to use the trtacker is to use the `fold_map` and `find`
	/// The best way to use the tracker is to use the `fold_map` and `find`

Writing in batches #4606

Writing in batches #4606

Conversation

lutter commented May 9, 2023

lutter commented May 11, 2023

lutter commented May 11, 2023

leoyvens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lutter commented May 17, 2023

leoyvens left a comment

Choose a reason for hiding this comment