This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
State migration rpc #10981
Merged
paritytech-processbot
merged 252 commits into
paritytech:master
from
cheme:rpc-state-migration
Mar 16, 2022
Merged
State migration rpc #10981
Changes from 245 commits
Commits
Show all changes
252 commits
Select commit
Hold shift + click to select a range
7e98291
setting flag
cheme 5d3a1b2
flag in storage struct
cheme 4256a40
fix flagging to access and insert.
cheme 5bb0b68
added todo to fix
cheme 7081864
also missing serialize meta to storage proof
cheme 3f1ee83
extract meta.
cheme 1bbdf35
Isolate old trie layout.
cheme e5feabf
failing test that requires storing in meta when old hash scheme is used.
cheme aa78ed6
old hash compatibility
cheme 045b097
Db migrate.
cheme 79a43dd
runing tests with both states when interesting.
cheme aa2e5c3
fix chain spec test with serde default.
cheme 91bd0aa
export state (missing trie function).
cheme 183b7b5
Pending using new branch, lacking genericity on layout resolution.
cheme 8511a40
extract and set global meta
cheme 270ed28
Update to branch 4
cheme b838e7e
fix iterator with root flag (no longer insert node).
cheme 30742b7
fix trie root hashing of root
cheme c620869
complete basic backend.
cheme a3b1039
Remove old_hash meta from proof that do not use inner_hashing.
cheme 693e38b
fix trie test for empty (force layout on empty deltas).
cheme 6e89b3a
Root update fix.
cheme c14e11c
Merge branch 'master' into state-update4
cheme f568794
debug on meta
cheme fb0ea03
Use trie key iteration that do not include value in proofs.
cheme cd98390
switch default test ext to use inner hash.
cheme 5ab0c01
small integration test, and fix tx cache mgmt in ext.
cheme 99ef85c
Proof scenario at state-machine level.
cheme 4d6aee8
trace for db upgrade
cheme 791acae
try different param
cheme b51eaa4
act more like iter_from.
cheme 715b1a5
Bigger batches.
cheme 3cdb0ed
Update trie dependency.
cheme ae1454f
drafting codec changes and refact
cheme c807f44
before removing unused branch no value alt hashing.
cheme a70ce70
alt hashing only for branch with value.
cheme 11f0d5a
fix trie tests
cheme cf0f0df
Hash of value include the encoded size.
cheme b721733
removing fields(broken)
cheme 2f8360f
fix trie_stream to also include value length in inner hash.
cheme 76b6d7a
triedbmut only using alt type if inner hashing.
cheme cfa0ecd
trie_stream to also only use alt hashing type when actually alt hashing.
cheme 60f7012
Refactor meta state, logic should work with change of trie treshold.
cheme 40ec2f7
Remove NoMeta variant.
cheme 204f51f
Remove state_hashed trigger specific functions.
cheme b62cdf5
pending switching to using threshold, new storage root api does not
cheme 74fb9c8
refactoring to use state from backend (not possible payload changes).
cheme 97a1aa6
Applying from previous state
cheme f4849a2
Remove default from storage, genesis need a special build.
cheme 4eabbde
rem empty space
cheme 18abb6e
Merge branch 'master' into state-update4-refact
cheme ae627f6
Catch problem: when using triedb with default: we should not revert
cheme 4706814
fix compilation
cheme 09d5cf6
Right logic to avoid switch on reencode when default layout.
cheme 906cd48
Clean up some todos
cheme 662eabe
remove trie meta from root upstream
cheme 17ac7cb
Merge branch 'master' into state-update4-refact
cheme ac0e019
update upstream and fix benches.
cheme 7e73a70
split some long lines.
cheme 3498d17
Merge branch 'master' into state-update4-refact
cheme 934021c
UPdate trie crate to work with new design.
cheme 1e40e6d
Finish update to refactored upstream.
cheme 602723e
update to latest triedb changes.
cheme 66ee72d
Clean up.
cheme 23c5db3
fix executor test.
cheme 03d93b2
rust fmt from master.
cheme b13ebd0
Merge branch 'master' into state-update4
cheme 93aaa4c
rust format.
cheme 66c832e
Merge branch 'master' into state-update4
cheme 89eba2b
Merge branch 'master' into state-update4
cheme 1e02c01
rustfmt
cheme 7898e49
Merge branch 'master' into state-update4
cheme a48970f
fix
cheme 92b9fcf
start host function driven versioning
cheme 145406a
update state-machine part
cheme 00460da
still need access to state version from runtime
cheme bad0723
state hash in mem: wrong
cheme 537c04f
direction likely correct, but passing call to code exec for genesis
cheme b537440
state version serialize in runtime, wrong approach, just initialize it
cheme b717611
stateversion from runtime version (core api >= 4).
cheme b6a6e83
Merge branch 'master' into state-update4-host2
cheme 846fef0
update trie, fix tests
cheme 0a6dc15
unused import
cheme 4e10a68
clean some TODOs
cheme bf31362
Require RuntimeVersionOf for executor
cheme 8aba5dd
use RuntimeVersionOf to resolve genesis state version.
cheme 499ab5b
update runtime version test
cheme 27cf136
fix state-machine tests
cheme ac4f7dc
TODO
cheme 1c817d6
Use runtime version from storage wasm with fast sync.
cheme f0d9326
rustfmt
cheme 6d930fa
Merge branch 'master' into state-update4-host2
cheme 877c36e
fmt
cheme 8514d5f
fix test
cheme 3385eae
revert useless changes.
cheme 3be7538
clean some unused changes
cheme c858c28
fmt
cheme 63265eb
removing useless trait function.
cheme 4142b8d
remove remaining reference to state_hash
cheme b3c8812
Merge branch 'state-update4-host2' of github.com:cheme/substrate into…
cheme aaca0f8
Merge branch 'master' into state-update4-host2
cheme 5ed824a
fix some imports
cheme b0b8a85
Follow chain state version management.
cheme e7267c0
trie update, fix and constant threshold for trie layouts.
cheme 599714f
update deps
cheme 6045952
Update to latest trie pr changes.
cheme fb8ff8e
fix benches
cheme af86f39
Verify proof requires right layout.
cheme 66daec0
update trie_root
cheme 299fea6
Merge branch 'master' into state-update4-host2
cheme 4ff9009
Update trie deps to latest
cheme fa7ebaf
Merge branch 'master' into state-update4-host2
cheme a343138
Update to latest trie versioning
cheme 3432668
Removing patch
cheme 932eee7
update lock
cheme c86af90
Merge branch 'master' into state-update4-host2
cheme 3299b7f
extrinsic for sc-service-test using layout v0.
cheme e66e40d
Merge branch 'master' into state-update4-host2
cheme 4328973
Adding RuntimeVersionOf to CallExecutor works.
cheme c342bdf
fmt
cheme d8a9b50
error when resolving version and no wasm in storage.
cheme 898bc13
use existing utils to instantiate runtime code.
cheme 1b1092c
migration pallet
kianenigma d35f273
Patch to delay runtime switch.
cheme 90ceb5e
Revert "Patch to delay runtime switch."
cheme eb272e6
fix test
kianenigma a327e5c
Merge branch 'master' into state-update4-host2
cheme e41881c
fix child migration calls.
cheme ac1a80c
useless closure
cheme b06a90c
remove remaining state_hash variables.
cheme 724b26f
Fix and add more tests
kianenigma 84e3ab9
Remove outdated comment
cheme 213d648
useless inner hash
cheme 24309cd
fmt
cheme 6133b5a
remote tests
kianenigma 4eb1d42
finally ksm works
kianenigma c5ae528
Merge branch 'master' of github.com:paritytech/substrate into kiz-inn…
kianenigma 2a98610
batches are broken
kianenigma 0d6590d
Merge branch 'master' of github.com:paritytech/substrate into ecr-sta…
kianenigma 0572949
Merge branch 'ecr-state-update-host' of github.com:paritytech/substra…
kianenigma c95cd1e
clean the benchmarks
kianenigma d9d09fd
Apply suggestions from code review
kianenigma 3eaa235
Apply suggestions from code review
kianenigma ef556f0
Update frame/state-trie-migration/src/lib.rs
kianenigma 9ff5ee9
Update frame/state-trie-migration/src/lib.rs
kianenigma a0e5c58
brand new version
kianenigma c953ff0
merged
kianenigma b6aad51
fix build
kianenigma a980e40
Update frame/state-trie-migration/src/lib.rs
kianenigma 013b389
Update frame/state-trie-migration/src/lib.rs
kianenigma 182ec74
Merge branch 'master' into state-update4-host2
cheme 3b267bf
Merge branch 'state-update4-host2' into ecr-state-update-host
cheme d6bd51d
Update primitives/storage/src/lib.rs
kianenigma 82c8cb2
Update frame/state-trie-migration/src/lib.rs
kianenigma d75b23e
Update frame/state-trie-migration/src/lib.rs
kianenigma 37041fc
merge master and some review comments
kianenigma d46b029
Upstream.into()
kianenigma 98c8467
Merge branch 'master' into ecr-state-update-host
cheme d69bce2
Merge branch 'master' into kiz-inner-hashing-runtime-migration
cheme 9b4837f
fmt and opt-in feature to apply state change.
cheme b3d8f62
feature gate core version, use new test feature for node and test node
cheme dd20a8d
Use a 'State' api version instead of Core one.
cheme a0ac11f
Merge branch 'master' into state-update4-host2
cheme c75d148
fix merge of test function
cheme aba8632
use blake macro.
cheme 205db5e
Fix state api (require declaring the api in runtime).
cheme d493547
Merge branch 'master' into state-update4-host2
cheme 6126ed2
Opt out feature, fix macro for io to select a given version
cheme 610bc23
run test nodes on new state.
cheme c0af4f1
fix
cheme 89b28ac
Merge branch 'master' into state-update4-host2
cheme 8e962f2
Merge branch 'state-update4-host2' into ecr-state-update-host
cheme 2506dd9
Merge branch 'ecr-state-update-host' into kiz-inner-hashing-runtime-m…
cheme 9db1b95
new test structure
kianenigma bb6b9a7
Merge branch 'kiz-inner-hashing-runtime-migration' of github.com:pari…
kianenigma 13e6844
new testing stuff from emeric
kianenigma ab79457
Add commit_all, still not working
kianenigma 07e495b
Fix all tests
kianenigma ba54fd5
add comment
kianenigma 64e4fba
we have PoV tracking baby
kianenigma b367aac
document stuff, but proof size is still wrong
kianenigma a5a2a27
FUCK YEAH
kianenigma 8a5cc10
a big batch of review comments
kianenigma 14d9f42
add more tests
kianenigma 4d392a1
tweak test
kianenigma d886baa
update config
kianenigma 2df11e4
some remote-ext stuff
kianenigma 363cbe1
Master.into()
kianenigma 14cf05d
and upstream.into()
kianenigma 8080e4c
delete some of the old stuff
kianenigma 56d40e5
sync more files with master to minimize the diff
kianenigma fc689fc
Fix all tests
kianenigma bb0b00f
make signed migration a bit more relaxed
kianenigma 6b83426
add witness check to signed submissions
kianenigma 1626d19
allow custom migration to also go above limit
kianenigma 0b3ea2d
Fix these pesky tests
kianenigma c2df41b
==== removal of the unsigned stuff ====
kianenigma 8102e15
Make all tests work again
kianenigma de17ba0
separate the tests from the logic so it can be reused easier
kianenigma 30c42cb
fix overall build
kianenigma 0304711
Update frame/state-trie-migration/src/lib.rs
kianenigma 0dcf25c
Update frame/state-trie-migration/src/lib.rs
kianenigma d822852
Master.into()
kianenigma b910012
Slightly better termination
kianenigma e883516
Master.into()
kianenigma 6e9106a
some final tweaks
kianenigma 4c119a0
Fix tests
kianenigma 9a4e316
Restrict access to signed migrations
kianenigma fa388fa
mig rpc
cheme a912083
fix
cheme 4af0ebc
Merge branch 'master' into kiz-inner-hashing-runtime-migration-check-rpc
cheme ada5916
better rpc name
cheme 67f020e
Make rpc unsafe
cheme 7c6ce54
address most of the review comments
kianenigma b33d8b8
fix defensive
kianenigma dfed611
New simplified code
kianenigma c0fe24d
Master.into()
kianenigma 91b19d1
Fix weights
kianenigma 80bb935
Master.into()
kianenigma 756140b
fmt
kianenigma b7555fc
Update frame/state-trie-migration/src/lib.rs
kianenigma 15f831d
make the tests correctly fail
kianenigma 28bae65
Merge branch 'kiz-inner-hashing-runtime-migration' of github.com:pari…
kianenigma ec1ace7
Fix build
kianenigma 4344530
Fix build
kianenigma 7f60fb3
try and fix the benchmarks
kianenigma c3ae56b
fix build
kianenigma a956f97
Fix cargo file
kianenigma 47654c2
Fix runtime deposit
kianenigma 6b162bb
make rustdoc happy
kianenigma 7294b02
Merge branch 'master' of https://github.com/paritytech/substrate into…
3b9f530
cargo run --quiet --profile=production --features=runtime-benchmarks…
7e6f57e
Merge branch 'kiz-inner-hashing-runtime-migration' into kiz-inner-has…
cheme 8142f35
update rpc deps, try to process empty keys.
cheme c9df2f7
Merge branch 'master' into rpc-state-migration
cheme 40d7f8e
move rpc crate
cheme 3c84337
move check backend out of state machine
cheme 0a5f3d9
Add primitive crate.
cheme 92f378b
module code
cheme 5632b53
fix runtime test
cheme f85752a
StateMigrationStatusProvider
cheme b1836a3
Pass backend to rpc.
cheme 993d14e
fmt
cheme be61414
review changes
cheme 53f1574
move rpc crate
cheme aa703c2
try remove primitive crate
cheme 581b1ca
Update utils/frame/rpc/state-trie-migration-rpc/Cargo.toml
cheme d85c890
review changes.
cheme bdf7cc5
Merge branch 'rpc-state-migration' of github.com:cheme/substrate into…
cheme 78a89bb
Merge branch 'master' into rpc-state-migration
cheme File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
[package] | ||
name = "sc-state-trie-migration-rpc" | ||
version = "4.0.0-dev" | ||
authors = ["Parity Technologies <admin@parity.io>"] | ||
edition = "2021" | ||
license = "Apache-2.0" | ||
homepage = "https://substrate.io" | ||
repository = "https://github.com/paritytech/substrate/" | ||
description = "Node-specific RPC methods for interaction with state trie migration." | ||
readme = "README.md" | ||
|
||
[package.metadata.docs.rs] | ||
targets = ["x86_64-unknown-linux-gnu"] | ||
|
||
[dependencies] | ||
codec = { package = "parity-scale-codec", version = "3" } | ||
jsonrpc-core = "18.0.0" | ||
jsonrpc-core-client = "18.0.0" | ||
jsonrpc-derive = "18.0.0" | ||
serde = { version = "1", features = ["derive"] } | ||
sp-state-trie-migration = { version = "4.0.0-dev", path = "../../primitives/state-trie-migration" } | ||
|
||
# Substrate Dependencies | ||
sc-client-api = { version = "4.0.0-dev", path = "../api" } | ||
sc-rpc-api = { version = "0.10.0-dev", path = "../rpc-api" } | ||
sp-runtime = { version = "6.0.0", path = "../../primitives/runtime" } | ||
|
||
[dev-dependencies] | ||
serde_json = "1" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Node-specific RPC methods for interaction with trie migration. | ||
|
||
License: Apache-2.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
// This file is part of Substrate. | ||
|
||
// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd. | ||
// SPDX-License-Identifier: Apache-2.0 | ||
|
||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
//! Rpc for state migration. | ||
|
||
use jsonrpc_core::{Error, ErrorCode, Result}; | ||
use jsonrpc_derive::rpc; | ||
use sc_rpc_api::DenyUnsafe; | ||
use serde::{Deserialize, Serialize}; | ||
use sp_runtime::{generic::BlockId, traits::Block as BlockT}; | ||
use std::sync::Arc; | ||
|
||
#[derive(Serialize, Deserialize)] | ||
#[serde(rename_all = "camelCase")] | ||
#[serde(deny_unknown_fields)] | ||
pub struct MigrationStatusResult { | ||
top_remaining_to_migrate: u64, | ||
child_remaining_to_migrate: u64, | ||
} | ||
|
||
/// Migration RPC methods. | ||
#[rpc] | ||
pub trait StateMigrationApi<BlockHash> { | ||
/// Check current migration state. | ||
/// | ||
/// This call is performed locally without submitting any transactions. Thus executing this | ||
/// won't change any state. Nonetheless it is a VERY costy call that should be | ||
/// only exposed to trusted peers. | ||
#[rpc(name = "state_trieMigrationStatus")] | ||
fn call(&self, at: Option<BlockHash>) -> Result<MigrationStatusResult>; | ||
} | ||
|
||
/// An implementation of state migration specific RPC methods. | ||
pub struct MigrationRpc<C, B, BA> { | ||
client: Arc<C>, | ||
backend: Arc<BA>, | ||
deny_unsafe: DenyUnsafe, | ||
_marker: std::marker::PhantomData<(B, BA)>, | ||
} | ||
|
||
impl<C, B, BA> MigrationRpc<C, B, BA> { | ||
/// Create new state migration rpc for the given reference to the client. | ||
pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self { | ||
MigrationRpc { client, backend, deny_unsafe, _marker: Default::default() } | ||
} | ||
} | ||
|
||
impl<C, B, BA> StateMigrationApi<<B as BlockT>::Hash> for MigrationRpc<C, B, BA> | ||
where | ||
B: BlockT, | ||
C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>, | ||
BA: 'static + sc_client_api::backend::Backend<B>, | ||
{ | ||
fn call(&self, at: Option<<B as BlockT>::Hash>) -> Result<MigrationStatusResult> { | ||
if let Err(err) = self.deny_unsafe.check_if_safe() { | ||
return Err(err.into()) | ||
} | ||
|
||
let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash)); | ||
let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?; | ||
let (top, child) = | ||
sp_state_trie_migration::migration_status(&state).map_err(error_into_rpc_err)?; | ||
|
||
Ok(MigrationStatusResult { | ||
top_remaining_to_migrate: top, | ||
child_remaining_to_migrate: child, | ||
}) | ||
} | ||
} | ||
|
||
fn error_into_rpc_err(err: impl std::fmt::Display) -> Error { | ||
Error { | ||
code: ErrorCode::ServerError(0), // TODO check errors code | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TODO |
||
message: "Error while checking migration state".into(), | ||
data: Some(err.to_string().into()), | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This crate should also be moved under the pallet. We already have multiple of these pallet rpc crates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had it under pallet at first but the dependencies from client crate sc-service to pallet was rejected.
Wait, I don't have this dependency anymore, think I can move all under pallet now (even move primitive to rpc as suggested) 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, actually rpc requires dependency to clients crate, which is not allowed (other frame crate can do without because they are using the RuntimeApi interface).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then please move it to the
utils/frame/rpc
folderThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's cheating, but yes it results in one less crate, I'm happy with it.