process_inherent_data: use an in memory overlay to minimize reads and writes to storage #3644

sandreim · 2024-03-11T12:58:39Z

Changes on top of #3479

Introduce a genericStorageMapOverlay and use it to efficiently mutate PendingAvailability storage in memory. Early in process_inherent_data we create this overlay and only write back what was changed before returning from the function.

Runtime APIs must still access the storage directly.

This changeset doesn't attempt to solve any logic issues that have been identified in the review of #3479, but it does try to help with allowing to change values while iterating the in-memory map, currently at the cost of an extra clone.

Signed-off-by: alindima <alin@parity.io>

also no need to sort by core index any more

…. optimise process_candidates to be O(N)

…aling-runtime

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim · 2024-03-13T14:09:55Z

bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-13T14:10:00Z

"$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent was queued.

Comment bot cancel 9-bb67ca25-f727-4aec-a7b8-4c43a48363db to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2024-03-13T14:10:35Z

@sandreim Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result:

ValidationError: "id" is required
ValidationError: "id" is required
{"message":{"base":["Reference not found"]}}

sandreim · 2024-03-13T14:18:00Z

bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-13T14:18:05Z

"$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent was queued.

Comment bot cancel 10-64f6bfcf-a5ef-4548-be6c-b2b52deb992a to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2024-03-13T14:18:42Z

@sandreim Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result:

ValidationError: "id" is required
ValidationError: "id" is required
{"message":{"base":["Reference not found"]}}

ggwpez · 2024-03-13T14:31:16Z

bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-13T14:31:21Z

@ggwpez https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5517794 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 11-6a538673-4134-4af3-a596-3a70c41f3ad0 to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2024-03-13T15:30:46Z

@ggwpez Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5517794 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5517794/artifacts/download.

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim · 2024-03-13T16:07:27Z

bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-13T16:07:32Z

@sandreim https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5520978 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 15-af575d13-ff31-4256-9a48-ef838166b0eb to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2024-03-13T17:07:46Z

@sandreim Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5520978 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5520978/artifacts/download.

paritytech-cicd-pr · 2024-03-13T17:14:00Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5523172

eskimor · 2024-03-14T11:41:00Z

polkadot/runtime/parachains/src/paras_inherent/mod.rs

@@ -667,6 +691,11 @@ impl<T: Config> Pallet<T> {
 			disputes,
 			parent_header,
 		};
+
+		// Write back to storage only keys that have been updated or deleted.
+		for (para_id, candidates) in pending_availability_overlay.into_iter() {


What about using a Drop instance like here?

yes, good idea, I'll have to see how it should work in the case of this generic overlay.

eskimor · 2024-03-14T11:43:11Z

polkadot/runtime/parachains/src/util.rs

+{
+	// Construct a new overlay instance.
+	pub fn new() -> Self {
+		let data = P::populate();


Given that we are doubling the memory footprint here, it would be really good if this improved performance by more than just a few percent.

Right now it improves it by around 12% . I'd say it is not great, but that's what is currently possible. What kind of improvement are you expecting for the tradeoff to be worth it ?

Good question. Depends on how tight we are on those axes. Time wise we are pretty good already. No idea about memory. Probably worth it.

But I just noticed something different: What this gains us, as far as I can see, is that it conflates two writes into one. At a quick glance it seems, that this should also easily be achievable by merging this mutate with the previous one. We are traversing them in order anyways. Instead of pushing to candidates_made_available, we could just enact immediately, if there were no holes (unavailable candidates in the loop before).

This should give us the same performance gain, but with half the memory consumption. @alindima what do you say?

I'll think about it a bit. In the first mutate, we update the availability votes. But when doing the enactments, we may enact more candidates than the ones that have been just made available in this block.
If for example we have A B C backed. B and C are already available. A is just being made available now. So we have to enact all candidates (A B and C).

That's why I chose to iterate twice there. I'll think more about it to see if we can call mutate only once for each para

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim · 2024-03-14T13:27:18Z

bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::paras_inherent -v UPSTREAM_MERGE=n

command-bot · 2024-03-14T13:27:23Z

@sandreim https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5532841 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-73ab14b9-7697-44c3-af04-731a5ad89026 to cancel this command or bot cancel to cancel all commands in this pull request.

…=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-14T14:26:14Z

@sandreim Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5532841 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5532841/artifacts/download.

Extracted Benchbuilder enhancements used in #3644 . Might still require some work to fully support all scenarios when disputing elastic scaling parachains, but it should be useful in writing elastic scaling runtime tests. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…3690) Extracted Benchbuilder enhancements used in paritytech#3644 . Might still require some work to fully support all scenarios when disputing elastic scaling parachains, but it should be useful in writing elastic scaling runtime tests. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alindima added 23 commits February 26, 2024 12:34

Initial draft changes

442185e

bugfixes

799fabe

Signed-off-by: alindima <alin@parity.io>

filter descendants of disputed candidates

9f3ba62

also no need to sort by core index any more

some simplifications

5b13eb7

assert that candidates of a para are sorted in chain dependency order…

ecc5088

…. optimise process_candidates to be O(N)

deduplicate some of the logic for freeing cores

0fb7b8c

update some comments

0ed6fc3

unify dropped candidates errors

0a08fa0

add more logs

9e51e20

remove some todos

4e154e6

review comments

ccef35b

some more nits

3f67898

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

c4fedd7

…aling-runtime

add runtime migration to inclusion storage

58b4129

add migration tests

23a8d34

don't allow candidate cycles

e0d9dff

fix bug

0198fb3

make tests compile and make paras_inherent tests pass

267fa05

fix inclusion tests

7733b25

fix some more tests

c32f925

clippy

40e2933

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

e8f5d2c

…aling-runtime

some review comments

630261e

sandreim added the R0-silent Changes should not be mentioned in any release notes label Mar 11, 2024

alindima added 6 commits March 11, 2024 15:07

simplify update_pending_availability_and_get_freed_cores

7dc5358

fix clippy

3793f20

add prdoc

6c0d062

fix availability_cores runtime API

74acc46

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

4dd28a2

…aling-runtime

remove some unused errors and reintroduce ValidationDataHashMismatch

5afb901

bench only big inherent

2bf9605

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added 2 commits March 13, 2024 18:05

add overlay get_mut

a3878ec

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

less cloning

728aa29

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim force-pushed the sandreim/storage_map_overlay branch from dd1e8f9 to 728aa29 Compare March 13, 2024 16:07

sandreim mentioned this pull request Mar 14, 2024

Add elastic scaling support in ParaInherent BenchBuilder #3690

Merged

eskimor reviewed Mar 14, 2024

View reviewed changes

add 3 code upgrades

03841e1

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim force-pushed the sandreim/storage_map_overlay branch from 6a43427 to 03841e1 Compare March 14, 2024 13:24

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

186d082

…=westend --target_dir=polkadot --pallet=runtime_parachains::paras_inherent

Base automatically changed from alindima/elastic-scaling-runtime to master March 21, 2024 10:44

sandreim closed this Aug 28, 2024

sandreim deleted the sandreim/storage_map_overlay branch August 28, 2024 16:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

process_inherent_data: use an in memory overlay to minimize reads and writes to storage #3644

process_inherent_data: use an in memory overlay to minimize reads and writes to storage #3644

sandreim commented Mar 11, 2024 •

edited

Loading

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024

ggwpez commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024

paritytech-cicd-pr commented Mar 13, 2024

eskimor Mar 14, 2024

sandreim Mar 14, 2024

eskimor Mar 14, 2024

sandreim Mar 14, 2024

eskimor Mar 14, 2024

alindima Mar 14, 2024

sandreim commented Mar 14, 2024

command-bot bot commented Mar 14, 2024 •

edited

Loading

command-bot bot commented Mar 14, 2024

process_inherent_data: use an in memory overlay to minimize reads and writes to storage #3644

process_inherent_data: use an in memory overlay to minimize reads and writes to storage #3644

Conversation

sandreim commented Mar 11, 2024 • edited Loading

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 • edited Loading

command-bot bot commented Mar 13, 2024

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 • edited Loading

command-bot bot commented Mar 13, 2024

ggwpez commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 • edited Loading

command-bot bot commented Mar 13, 2024

sandreim commented Mar 13, 2024

command-bot bot commented Mar 13, 2024 • edited Loading

command-bot bot commented Mar 13, 2024

paritytech-cicd-pr commented Mar 13, 2024

eskimor Mar 14, 2024

Choose a reason for hiding this comment

sandreim Mar 14, 2024

Choose a reason for hiding this comment

eskimor Mar 14, 2024

Choose a reason for hiding this comment

sandreim Mar 14, 2024

Choose a reason for hiding this comment

eskimor Mar 14, 2024

Choose a reason for hiding this comment

alindima Mar 14, 2024

Choose a reason for hiding this comment

sandreim commented Mar 14, 2024

command-bot bot commented Mar 14, 2024 • edited Loading

command-bot bot commented Mar 14, 2024

sandreim commented Mar 11, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 13, 2024 •

edited

Loading

command-bot bot commented Mar 14, 2024 •

edited

Loading