Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Expose the overlay from ApiExt #10922

Closed

Conversation

liuchengxu
Copy link
Contributor

@liuchengxu liuchengxu commented Feb 25, 2022

Expose the necessary API overlay from ApiExt for supporting the extrinsic execution proof.

Providing a way to get the overlay so that it can be passed into the execution proof later.
@liuchengxu liuchengxu changed the title Support generating the extrinsic execution proof Support generating and verifying the extrinsic execution proof Mar 6, 2022
@liuchengxu
Copy link
Contributor Author

@bkchr Any thoughts on the direction of this PR? #10987 could be also related.

Copy link
Contributor

@cheme cheme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just read #10783. IIUC, checking a extrinsic proof will require:

  • block B hash, state root and extrinsic proof for for this block
  • block B+1 hash and state root
  • state root at extrinsic N (probably stored in state for every extrinsic), so need proof for it in block B+1, same for state root at N +1
  • proof of changed value at N to be able to build and verify the initial overlay change: this is not optimal, ideally it will be only value that the extrinsic need to access, but with current code building root at N+1 will access all values (build from scratch, not iteratively).
    Then run and check that the produced root at N+1 is right.

And here to build the proof of the initial change set (changed value at N), looks a bit tricky.
I think it would need extract the change set for 0 to N-1 extrinsic.
Then build a partially in memory trie state (triedbmut with a db that write in an inmemory layer and read in block b state).
Then create the proof for the initial change set by by querying all changes of 0 to N-1 on this temporary trie.

In the end, if I compare to just using a proof of extrinsic 0 to N, we would remove read access of unmodified value for 0 to N-1. Not sure how relevant it is for your runtime (maybe proof of extrinsic 0 to N is just easier).

I may have misunderstood some points, anyway, I don't think these will requires any other substrate changes (except execution verification).

client/service/src/client/call_executor.rs Outdated Show resolved Hide resolved
client/api/src/call_executor.rs Outdated Show resolved Hide resolved
@liuchengxu
Copy link
Contributor Author

liuchengxu commented Mar 8, 2022

Thank you for taking the time to read the background! @cheme

checking a extrinsic proof will require:

  • block B hash, state root and extrinsic proof for for this block
  • block B+1 hash and state root
  • state root at extrinsic N (probably stored in state for every extrinsic), so need proof for it in block B+1, same for state root at N +1
  • proof of changed value at N to be able to build and verify the initial overlay change: this is not optimal, ideally it will be only >value that the extrinsic need to access, but with current code building root at N+1 will access all values (build from scratch, not iteratively).
    Then run and check that the produced root at N+1 is right.

Hmm, not sure if we're thinking about the same problem, I don't understand why block B+1 is involved, the execution proof of an extrinsic in block B is only related to the block B.

First of all, we need to know the verifier of the extrinsic execution proof does not own the whole state so he needs the proof to restore the essential state/trie_backend for executing the extrinsic. Checking the execution proof of an extrinsic then requires:

  • the state_root before executing the extrinsic pre_state_root
  • the state root after executing the extrinsic post_state_root
  • the extrinsic itself
  • the runtime code for executing the method BlockBuilder_apply_extrinsic with the call data(extrinsic)

Given an execution proof, we first restore a trie backend that contains all the storage values necessary for executing this extrinsic, apply the extrinsic by running the executor with the method BlockBuilder_apply_extrinsic and call data, and then calculate the post state root. If the locally produced post_state_root matches the one received along with the proof, we can say the proof is correct.

probably stored in state for every extrinsic

Yes, that's what we're doing right now(we might optimize this later by not storing the intermediate storage root in the state). Now we forked the executive pallet to store the storage root after applying each extrinsic, #10711 is related.

Based on my current understanding(may change with more in-depth experiment), the required substrate changes could be:

@cheme
Copy link
Contributor

cheme commented Mar 8, 2022

I don't understand why block B+1 is involved

To verify value for pre_state_root and post_state_root (if they are in state).

I did think a bit about it and, what would be the perfect situation would be a proof from state B-N to state B-N+1 with B being block number and N the extrinsic number.
Alas that would be a heck of a hacking, I don't think it is easily doable (the way we calculate storage root in state machine is always from B-0).
So we have a proof from B-0 with a verified execution state at B-N (state checked against pre_state_root) to B-N+1, then checked against post_state_root. Note that is is a bit implementation dependant (the way storage_root accesses state between two different calls may not be well specified enough).

@liuchengxu
Copy link
Contributor Author

what would be the perfect situation would be a proof from state B-N to state B-N+1 with B being block number and N the extrinsic number.

That's what we want to achieve in Subspace at present. In our case, all the intermediate storage roots are collected and stored in each block, so you don't get have to calculate the storage root while generating the extrinsic execution proof, making the entire thing doable.

Only the execution proof is missing when we want to form a proof for the extrinsic state transition. The initial state for executing the extrinsic N+1 is parent_block_state + overlay(0..N). In order to generate the execution proof N+1, we rerun the extrinsics 0..N to accumulate the overlayed changes, and then invoke proof_provider.execution_proof to generate the proof. So the proof for N+1 is not from B0, but from BN.

Note that is is a bit implementation dependant (the way storage_root accesses state between two different calls may not be well specified enough).

Hmm, I don't get why it's not deterministic. Anyway, it's fine as long as it produces the same post_state_root for the given initial state, method and call data.

@cheme
Copy link
Contributor

cheme commented Mar 8, 2022

we rerun the extrinsics 0..N to accumulate the overlayed changes, and then invoke proof_provider.execution_proof to generate the proof. So the proof for N+1 is not from B0, but from BN.

What is executed is executed over state at B0, and a proved state (overlay) that matches execution state at BN, but it is not querying BN state.
And the proof registered will be nodes from B0.
When storage_proof at the end of the extrinsic processing will be called, it will calculate BN+1 root over all deltas from B0: expect proof size to grow with extrinsic indices.
So the optimisation make sense if there is a lot of read_only accesses in the extrinsic, so depending on your runtime, the overhead could be negative (overhead being verifying the change set over BN state and verifying pre_state_root and post_state_root against B+1).

Edit: actually B0 is a bit misleading, it is really B (since at extrinsic 0 start there will already be some changes).

@cheme
Copy link
Contributor

cheme commented Mar 8, 2022

What you could do is extract the trie node changes after applying state transition from block initialization and extrinsic 0 to N (transaction field of StorageChange after calling into_storage_changes).
Then you prove your execution over a trie backend that use as a storage the storage from B and an in memory overlay of your extracted changes.

@liuchengxu
Copy link
Contributor Author

actually B0 is a bit misleading, it is really B (since at extrinsic 0 start there will already be some changes).

True, so we are actually storing the intermediate storage roots for the hooks and extrinsics:

  • after the initialize_block
  • after applying each extrinsic
  • after the finalize_block (which is actually the current state_root field in the header)

Thanks for all the info! @cheme Overall I think we're on the same page, just some implementation details. In order to generate the execution proof for the extrinsic position at N+1 in a block BN+1, the core part is to prepare the state BN(the state after applying the extrinsic N), which is B0 + Overlay0..N. As long as we have the state BN, we can precisely derive the storage proof for executing extrinsic N+1, so the deltas will be only from BN, not from B0.

The problem is that the proof genetrated this way will also require the Overlay to be passed into the underlying exeution_proof_check, which is a thing we'd like to get rid of since the verifier of this proof does not have to know the concept of overlay, he only needs a full storage proof to restore a trie backend to execute this extrinsic.

Therefore, we are thinking about a custom overlay which is also able to record the storage accessed during the extrinsic execution(replace it with the custom overlay here). At the end of the day, the verify can check the execution proof with only a full complete storage proof including the necessary overlay changeset. Any suggestion for building a custom overlay which is able to record the storage just like what the proof recorder does?

@cheme
Copy link
Contributor

cheme commented Mar 9, 2022

Don't get me wrong, I got no issue with exposing the overlay as this PR propose.

Just pointing out the design is difficult to put in place.

after applying each extrinsic

Storing the root after extrinsic, is the same as calculating at the end of the extrinsic: it needs to be in the execution to be able to check BN+1, so it will still include all changes from B0 to BN.

The problem is that the proof genetrated this way will also require the Overlay to be passed into the underlying exeution_proof_check, which is a thing we'd like to get rid of since the verifier of this proof does not have to know the concept of overlay, he only needs a full storage proof to restore a trie backend to execute this extrinsic.

In this PR it seems right to use the overlay, as it is the actual state of execution and in a more general context it makes sense to chain multiple executions.
In your api, it should be a key option map (and n child trie key option map) that is basically the changeset at BN.
I think there is a struct for it in state_machine (StorageChange maybe).
This changeset obviously need to be verified.

Using these two proofs, will verify extrinsic N. Just it is a double proof: a proof of state machine state at BN (overlay) and a proof of execution at B0.
In the end it can be a lot smaller than a proof from B0 to BN because the first proof uses a registered root. So it removes all read access from B0 to BN.

The proof for the change set is a difficult story (I describe it in my first comment) and may prove pretty redundant with the proof of execution (could probably be optimize, but all this layers of things do not feel really safe).

So the design with two proofs only make sense if in the extrinsic 0 to N-1 you got lot of read accesses and few write accesses.

Generally my last comment suggest an approach that looks safer to produce a single proof (no need for a changeset).

In my last comment I suggest that the execution proof creation uses a TrieBackendStorage that is the trie encoded nodes from state B (usually rocksdb) completed from the trie encoded nodes from execution of block of size N-1 without finalization. This way you got a proof that register node from state BN and no content needed in overlay.

Of course when I say 2 or 1 proof, I omit the proof to check the roots of BN and BN+1.

Therefore, we are thinking about a custom overlay which is also able to record the storage accessed during the extrinsic execution

Maybe it is related to the way you plan to merge the two proofs, or do some deltas of accesses between 0 to N and N.

I actually updated some old code doing it in some experimental branch yesterday (trying to lookup for access conflict between extrinsic), also previous change trie did it for write access.
But I would not be positive about such code getting merge.

the core part is to prepare the state BN(the state after applying the extrinsic N), which is B0 + Overlay0..N. As long as we have the state BN, we can precisely derive the storage proof for executing extrinsic N+1, so the deltas will be only from BN, not from B0

What I did suggest: using registered node (execution proof) at BN, apply them on the trie storage (by using a in memory overlay and implement trait TrieBackendStorage), is exactly 'prepare the state BN'. Then create execution proof on this prepared state to get a single proof without overlay.

Deriving a BN proof from B0 + Overay0..N seems quite tricky.

@cheme
Copy link
Contributor

cheme commented Mar 9, 2022

Another advantage of extranction execution proof by running on a true state BN (build from state B and overlay of changed trie node from 0 to BN) is that the call to storage root will only run on delta from BN to BN+1 (not 0 to BN+1).

@liuchengxu
Copy link
Contributor Author

Yes, creating a true state BN also looks more straightforward to me! I can be dumb, could you please elaborate on how to convert the B0+overlay to a trie backend for the state BN? Convert the state of block B to a trie backend and conver the overlay a trie backend, and then merge the two into one trie backend for the state BN? @cheme

Additionally, I believe we'll still need to add an optional argument Option<TrieBackend> so that we could pass in the state BN.

@cheme
Copy link
Contributor

cheme commented Mar 9, 2022

It's not trivial, I don't have all the code in mind, but basically in

pub fn prove_execution_on_trie_backend<S, H, Exec, Spawn>(
we want to use a TrieBackend for BN.
That could be build from the trie backend at B by replacing in
pub struct TrieBackendEssence<S: TrieBackendStorage<H>, H: Hasher> {
field root by root at BN and replacing storage by one with the diff.
something like:

struct NodeStorageWithDelta<S> {
  inner: S,
  delta: HashMap<Hash, Option<EncodedNode>>,
}
impl TrieBackendStorage for NodeStorageWithDelta {
  fn get {
// check delta first, if undefined call inner
}
}

with Hash = Vec (it is actually prefixed) and EncodedNode = Vec
Where delta contains the encoded node change from the processing of initialize_block and all extrinsic before N.
Actually delta does not need to contain removed node.
And inner is S at B (from the trie backend at B).

Then the question is how to get the root at BN and the encoded node delta at BN, here I would just call into_storage_changes on the overlay after execution, the fields to use on the result are:

	pub transaction: Transaction,
	/// The storage root after applying the transaction.
	pub transaction_storage_root: H::Out,

@liuchengxu
Copy link
Contributor Author

@cheme Thanks for the hint! Not sure I fully understand them, I have to first have some experiments.

@cheme
Copy link
Contributor

cheme commented Mar 9, 2022

Sometime, my explanation are just not very good, don't hesitate to ask.

@liuchengxu
Copy link
Contributor Author

@cheme Can be great if you could take a quick look if this is on the right track autonomys/substrate@8ce691d...subspace:3f76ce1ac913855ba0c427a32b1807bec80fa310, it compiles but runs with panic, something is definitely wrong, I'm still on the debugging :D

@cheme
Copy link
Contributor

cheme commented Mar 10, 2022

putting delta: Option<(sp_api::OverlayedChanges, Block::Hash)>, in call_executor make sense, it should content your BN hash an the nodes delta (not OverlayedChanges).
Even if node delta can be extracted from overlayed change, this is not the place to do it, extracting the node delta should be done when evaluating "block initialization + extrinsic 0 to N, and into_storage_changes requires to run on the backend of B0 (if there is no call to storage root, it does, even if you should have one at the end of your extrinsic).
Running on a dummy backend like you do could work only if it got the cache from latest call to storage_root, that is very specific.
(the approach could work for testing though, you just need to make sure your extrinsic do not change anything after its call to storage_root).

Your line TODO support BN does not make sense to me, the BN case is when you attach the delta, the other case is normal case (may be some code artifact from previous change, I always have some in my drafts).

@cheme
Copy link
Contributor

cheme commented Mar 10, 2022

The logic for the trie backend change looks good to me (can be a single function).

@cheme
Copy link
Contributor

cheme commented Mar 10, 2022

another small thing I notice, in :

			let current_state_backend = sp_state_machine::CurrentStateBackend::new(
				trie_backend,
				storage_changes.transaction,
				post_delta_root,
			);

it should be

			let current_state_backend = sp_state_machine::CurrentStateBackend::new(
				trie_backend,
				storage_changes.transaction,
				storage_changes.trasaction_storage_root,
			);

Actually for delta, maybe just passing StorageChanges struct could make sense (can be seen as previous transition result).

but the value should be identical between both: you can probably check equality of your post_delta_root and storage_changes.transaction_storage_root (both are root at BN).

@liuchengxu
Copy link
Contributor Author

@cheme Thanks for the comments! I'll look into them.

@liuchengxu
Copy link
Contributor Author

One thing I noticed is parent_hash in into_storage_changes is unused in fact,

parent_hash: H::Out,
mut cache: StorageTransactionCache<B::Transaction, H>,
state_version: StateVersion,
) -> Result<StorageChanges<B::Transaction, H>, DefaultError>
where
H::Out: Ord + Encode + 'static,
{
self.drain_storage_changes(backend, parent_hash, &mut cache, state_version)
}
/// Drain all changes into a [`StorageChanges`] instance. Leave empty overlay in place.
pub fn drain_storage_changes<B: Backend<H>, H: Hasher>(
&mut self,
backend: &B,
_parent_hash: H::Out,
can we remove this argument in into_storage_changes then?

@cheme
Copy link
Contributor

cheme commented Mar 10, 2022

Indeed, hash is already known by the backend. Could be an idea to remove it indeed.

@liuchengxu
Copy link
Contributor Author

liuchengxu commented Mar 11, 2022

If I understand correctly, instead of using a dummy backend, I should just use the state at B https://github.com/subspace/substrate/blob/7da1cf8348/client/service/src/client/call_executor.rs#L318

error[E0308]: mismatched types
   --> client/service/src/client/call_executor.rs:323:5
    |
323 |                 storage_changes.transaction,
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `memory_db::MemoryDB`, found associated type
    |
    = note:       expected struct `memory_db::MemoryDB<<<Block as BlockT>::Header as HeaderT>::Hashing, memory_db::HashKey<<<Block as BlockT>::Header as HeaderT>::Hashing>, Vec<u8>, memory_db::malloc_size_of::NoopTracker<Vec<u8>>>`
            found associated type `<<B as sc_client_api::Backend<Block>>::State as sc_client_api::StateBackend<<<Block as BlockT>::Header as HeaderT>::Hashing>>::Transaction`
    = help: consider constraining the associated type `<<B as sc_client_api::Backend<Block>>::State as sc_client_api::StateBackend<<<Block as BlockT>::Header as HeaderT>::Hashing>>::Transaction` to `memory_db::MemoryDB<<<Block as BlockT>::Header as HeaderT>::Hashing, memory_db::HashKey<<<Block as BlockT>::Header as HeaderT>::Hashing>, Vec<u8>, memory_db::malloc_size_of::NoopTracker<Vec<u8>>>`
    = note: for more information, visit https://doc.rust-lang.org/book/ch19-03-advanced-traits.html

error[E0308]: mismatched types
   --> client/service/src/client/call_executor.rs:323:5
    |
323 |                 storage_changes.transaction,
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `memory_db::MemoryDB`, found associated type
    |
    = note:       expected struct `memory_db::MemoryDB<<<Block as BlockT>::Header as HeaderT>::Hashing, memory_db::HashKey<<<Block as BlockT>::Header as HeaderT>::Hashing>, Vec<u8>>`
            found associated type `<<B as sc_client_api::Backend<Block>>::State as sc_client_api::StateBackend<<<Block as BlockT>::Header as HeaderT>::Hashing>>::Transaction`
    = help: consider constraining the associated type `<<B as sc_client_api::Backend<Block>>::State as sc_client_api::StateBackend<<<Block as BlockT>::Header as HeaderT>::Hashing>>::Transaction` to `memory_db::MemoryDB<<<Block as BlockT>::Header as HeaderT>::Hashing, memory_db::HashKey<<<Block as BlockT>::Header as HeaderT>::Hashing>, Vec<u8>>`

Then how can I convert StateBackend::Transaction to something that implements HashDB or a HashMap of trie nodes so that I can implement TrieBackendStorage? Use this branch if you'd like to compile it. @cheme

@cheme
Copy link
Contributor

cheme commented Mar 11, 2022

If I understand correctly, instead of using a dummy backend, I should just use the state at B

yes, you can, even if I would make the call to into_storage_change out of prove_execution, and pass an optional memorydb and a root as param, but that's a detail.
Not just a detail as converting transaction to memorydb is certainly easier out of this function where you will probably have access to the type of Transaction (MemoryDb) instead of just its trait.

Other parametr input type for prove function than MemoryDb could be just a HashMap<H::Out, Vec> (prefix can be ignored).

@liuchengxu
Copy link
Contributor Author

Not just a detail as converting transaction to memorydb is certainly easier out of this function where you will probably have access to the type of Transaction (MemoryDb) instead of just its trait.

👍 This way works.

@cheme Please have a look at the latest changes where I changed the input to PrefixedMemoryDB autonomys/substrate@8ce691d...subspace:142eeeb5545a03edd69d99cae0d39fc0e8a64973 I have tested the execution proof locally, seemingly works well.

If the changes look good to you, I'd love to split this PR into multiple ones to make some progress incrementally:

  1. Add an extra argument delta_changes to execution_proof for generating an execution proof for any extrinsic.
  2. Support verify_execution_proof in ProofProvider.
  3. Make execution_proof more flexible, e.g., the runtime code will be fetched another way in Subspace.
  4. Check if any features missing for generating the execution proof for any stages(initialize_block, extrinsic, fianlize_block).

@bkchr bkchr added B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit A2-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). labels Mar 18, 2022
@bkchr
Copy link
Member

bkchr commented Mar 18, 2022

@liuchengxu could you please merge master to make the CI happy?

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

@liuchengxu , I lost track of where exposing the overlay is needed, no problem with it, but I just can't remember.

@liuchengxu
Copy link
Contributor Author

@cheme I need the overlay to rebuild the state to re-execute a certain extrinsic precisely, which is basically parent_state + overlay. Specifically, the overlay will be converted to a memory DB(something that implements HashDB) and used to build a DeltaBackend in #11019.

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

@cheme I need the overlay to rebuild the state to re-execute a certain extrinsic precisely, which is basically parent_state + overlay. Specifically, the overlay will be converted to a memory DB(something that implements HashDB) and used to build a DeltaBackend in #11019.

Isn't it build from 'transaction' and 'root' field of the result of 'into_storage_changes'?
Or in case of call to prove_on_trie_backend it is a mutable reference as input parameter, so we already got it.
(I think I am missing something)

@liuchengxu
Copy link
Contributor Author

@cheme Yes, transaction and transaction_storage_root is built by overlay.into_storage_changes(), but the overlay is built by partially executing a custom block builder, so we need to access the overlay some time during the block execution, this piece of code might be helpful https://github.com/subspace/subspace/blob/0adb26f311/cumulus/client/block-builder/src/lib.rs#L233

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

Why do you need to prepare an overlay with content? I mean everything should be executed on empty overlay (since we now use full states, eg BN being build over trie storage that contains the changes from the overlay, so using empty overlay is fine)

@liuchengxu
Copy link
Contributor Author

Proving the execution prove_execution_on_trie_backend will be run on an empty overlay, in order to do that, we need a true state of BN which is achieved by using a DeltaBackend that wraps parent_state and overlay.

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

Proving the execution prove_execution_on_trie_backend will be run on an empty overlay, in order to do that, we need a true state of BN which is achieved by using a DeltaBackend that wraps parent_state and overlay.

Isn't DeltaBackend built from the resul to into_storage_root (transaction and root)? (like https://github.com/subspace/subspace/blob/ad7649bbcc9c2d103f98437eb72b92a6896fa20d/cumulus/client/cirrus-executor/src/lib.rs#L330)

@liuchengxu
Copy link
Contributor Author

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

So I don't understand where we need to access the overlay with the new method :(

@liuchengxu
Copy link
Contributor Author

https://github.com/subspace/subspace/blob/ad7649bbcc/cumulus/client/block-builder/src/lib.rs#L233 This line requires this new overlay API, otherwise, we can't get the overlayed changes before executing the extrinsic N, thus we can't build the state BN later. :P

@cheme
Copy link
Contributor

cheme commented Mar 18, 2022

Brings me back to #10922 (comment)

we can't get the overlayed changes before executing the extrinsic N, thus we can't build the state BN later. :P

but state bn is build from executing from state B the block up to extrinsic N. So it run on empty overlay. And then we extract BN by using into_storage_change.

@liuchengxu
Copy link
Contributor Author

but state bn is build from executing from state B the block up to extrinsic N. So it run on empty overlay. And then we extract BN by using into_storage_change.

Oh, get your point finally! Yes, this PR is unnecessary! I was solving the problem indirectly. This can be useful to understand what happens here for these interested in this issue.

@cheme Really appreciated about all the amazing help on this topic! Hope I can figure out the last issue in #10922 (comment) soon :P

@liuchengxu liuchengxu closed this Mar 19, 2022
@cheme
Copy link
Contributor

cheme commented Mar 19, 2022

No problem, I think I still got email alert on closed PR, don't hesitate to continue the conversation here if needed, it is an interesting topic.

driemworks added a commit to ideal-lab5/substrate-offchain-ipfs that referenced this pull request May 9, 2022
* Make some UI test expectations crisper (#10791)

* Make some UI test expectations crisper

* Update frame/support/test/tests/construct_runtime_ui/undefined_genesis_config_part.rs

* Update UI test expectations

* Missing newlines

* More whitespace issues

* contracts: `is_contract(address)` and `caller_is_origin()` are added to API (#10789)

* is_contract() and caller_is_origin() added to Ext API

* is_contract() exposed in wasm runtime.rs

* + test for is_contract()

* + seal_is_contract benchmark

* caller_is_origin() exposed to wasm/runtime.rs and covered by a test

* + seal_caller_is_origin benchmark

* Update frame/contracts/src/exec.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/exec.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/exec.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/wasm/runtime.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/wasm/runtime.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/wasm/runtime.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Update frame/contracts/src/exec.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* identation fix for benchmark macroses; test cosmetic improvement

* benchmark fix

* + is_contract() wasm test

* + caller_is_origin() wasm test

* Apply suggestions from code review

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* is_contract() to borrow param instead of taking ownership

* phrasing improved

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* fixed wasm tests according to @athei feedback

* dead code warnings suppressed by unstable-interface attributes

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>

* maybe_total_supply in pallet-assets (#10799)

* maybe_total_supply in pallet-assets

* Update frame/assets/src/functions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Referenda and Conviction Voting pallets (#10195)

* Initial draft of new referendum state machine.

* Docs

* Fixes

* Fixes

* Add conviction-voting pallet

* Basic build

* Building

* Some TODOs

* Tests building

* Add missing file

* Basic lifecycle test

* Add couple of tests

* Another test

* More tests

* Fixes

* Fixes

* Formatting

* Fixes

* Tests

* Fixes

* Fixes

* More tests

* Formatting

* First few benchmarks

* First few benchmarks

* Defered queue servicing

* More testing

* Benchmarks

* Fiddly benchmark

* Final nudge benchmarks

* Formatting

* Formatting

* Finished up benchmarks

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_referenda --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/referenda/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Events finished

* Missing file

* No GenesisConfig for Referenda

* Formatting

* Docs

* Docs

* Docs

* Per-class conviction voting

* New test & mock utils

* More tests

* Tests

* Tests finished 🎉

* Benchmarking stuff

* Fixes

* Test harness

* Test harness

* Benchmarks for Conviction=Voting

* Benchmarking pipeline complete

* Docs

* Formatting

* Remove unneeded warning

* Fix UI tests

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_conviction_voting --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/conviction-voting/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Docs

* Update frame/conviction-voting/src/vote.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* update sp-runtime version

* MEL Fixes for Referenda and Conviction Voting (#10725)

* free maxencodedlen

* more maxencodedlen

* more MEL

* more mel

* disable storage info

* More Referenda Patches (#10760)

* basic fixes

* fix benchmarking

* fix license

* prevent panic in curve math

* fmt

* bump crate versions

* Update mock.rs

Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* More efficient WASM instance memory decommit on macos (#10801)

* More efficient WASM instance memory decommit on macos

* Apply suggestions from code review

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Updated error message

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* make inner field of IdentityFields pub (#10773)

* Corrected description of clear_attribute extrinsic (#10729)

* block validators after X sessions, update mock runtime for test

* sp-maybe-compressed-blob: reduce boilerplate code (#10814)

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* `seal_delegate_call` api function (support for library contracts)  (#10617)

* seal_call_code implementation

- tests
- benchmark

* Addressing @xgreenx's comments

* Fix test-linux-stable-int

* Rename seal_call_code to seal_delegate_call

* Pass value unchanged into lib contract

* Address @athei's comments

- whitespace .wat issues
- wrong/missing .wat comments
- redundant .wat calls/declarations

- change order of functions (seal_delegate_call right after seal_call)
  in decls, tests, benchmark
- fix comments, move doc comments to enum variants
- remove unnecessary empty lines

- rename runtime cost DelegateCall to DelegateCallBase
- do not set CallFlags::ALLOW_REENTRY for delegate_call

* Do not pass CallFlags::ALLOWS_REENTRY for delegate_call

* Update comment for seal_delegate_call and CallFlags

* Addressing @athei's comments (minor)

* Allow reentry for a new frame after delegate_call (revert)

* Same seal_caller and seal_value_transferred for lib contract

- test
- refactor frame args due to review
- logic for seal_caller (please review)

* Put caller on frame for delegate_call, minor fixes

* Update comment for delegate_call

* Addressing @athei's comments

* Update weights generated by benchmark

* Improve comments

* Address @HCastano's comments

* Update weights, thanks @joao-paulo-parity

* Improve InvalidCallFlags error comment

* Add Storage Info to Various Pallets (#10810)

* atomic swap

* bounties

* bounties fmt

* gilt

* indices

* nicks

* randomness-collective-flip

* recovery

* reuse maxapprovals

* Update tests.rs

* Update frame/randomness-collective-flip/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* use the correct bound

* complete recovery

* use `bounded_vec` macro

* Update tests.rs

* transaction payment

* uniques

* mmr

* example offchain worker

* beefy-mmr

* Update frame/recovery/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use BoundedVec instead of a type-parameterized BoundedString

* cargo fmt

* Update frame/atomic-swap/src/lib.rs

* use config const

* Update lib.rs

* update mel_bound

* fmt

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* fixed regression in inline docs (#10819)

* try-runtime-cli: Add to docs (#10822)

* try-runtime-cli: Add to docs of pre/post hooks

* Add note about --ws-max-out-buffer-capacity 1000

* Bump serde from 1.0.132 to 1.0.136 (#10816)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.132 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.136)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* uodate tests, ipfs tests failing

* --dev implies --tmp (#10828)

* Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)

* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments

* specify ipfs version, update tests

* Add db params for export-state CLI command (#10830)

* update tests, cleanup

* Allow `SetBalance` to handle error when trying to kill acount with reference counter. (#10826)

* bug found

* fix logic

* a little simpler

* add test

* UncheckedExtrinsic: Harden decode and clarify `EXTRINSIC_FORMAT_VERSION` (#10829)

* UncheckedExtrinsic: Harden decode and clarify `EXTRINSIC_FORMAT_VERSION`

* Apply suggestions from code review

* fix typo

* pallet-staking: Add extrinsic `force_apply_min_commission` (#10786)

* pallet-staking: Add extrinsic `force_apply_min_commission`

* Add benchmarks

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Bound iteration by  max_validator_count

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Only apply to 1 validator

* Update doc comments

* Uncomment tests

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Accept signed origins

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Remove contains_key check

* Add test for try_mutate_exists

* Impove try_mutate_exists docs

* Delete redundant try_mutate_exists tests;

* Delete residual from removed test

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Return an error when the stash does not exist

* Update try_mutate_exist doc wording

* Update frame/staking/src/pallet/mod.rs

* Apply suggestions from code review

Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Upgradable contracts using `set_code` function (#10690)

* poc logic

* set_code_hash impl, tests, benchmark

* Address @xgreenx's comments

* Move func defs closer to set_storage

* Check if code exists

- increment/decrement codehash refcount

* Document error for non-existing code hash

* Revert unrelated change

* Changes due to @athei's review

* Fix error handling

- comment errors: ReturnCodes
- update mock ext implementation
- return Error::CodeNotFound when no code for such hash

* Emit ContractCodeUpdated when setting new code_hash

* Address @athei's comments

* Move related defs to the bottom

* Minor comment update

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Improve docs

* Improve docs

* Update frame/contracts/src/wasm/runtime.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Refactor set_code_hash test

* Minor change to benchmark

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Minor change to benchmark

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Minor comment refactor

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Address @HCastano's comments

* Update seal_set_code_hash comment

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Move set_code_hash after delegate_call

* Move function to the bottom

* Moved and changed banchmark, added verify block

* Bring back previous benchmark

* Remove skip_meta for seal_set_code_hash

* Bring back skip_meta for seal_set_storage_per_new_kb

* Apply weights

Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Separate wasmi and wasmer sandbox implementations into their own modules (#10563)

* Moves wasmi specific `ImportResolver` and `MemoryTransfer` impls to submodule

* Splits context store environmental, moves impl `Externals` to wasmi backend

* Adds wasmer sandbox backend stub module

* Move sandbox impl code to backend specific modules

* Moves wasmi stuff

* Fixes value conversion

* Makes it all compile

* Remove `with_context_store`

* Moves `WasmerBackend` to the impl

* Reformat the source

* Moves wasmer MemoryWrapper

* Reformats the source

* Fixes mutability

* Moves backend impls to a submodule

* Fix visibility

* Reformat the source

* Feature gate wasmer backend module

* Moves wasmi memory allocation to backend module

* Rename WasmerBackend to Backend

* Refactor dispatch result decoding, get rid of Wasmi types in common sandbox code

* Reformat the source

* Remove redundant prefixes in backend functions

* Remove wasmer-sandbox from default features

* Post-review changes

* Add conversion soundness proof

* Remove redundant prefix

* Removes now redundant clone_inner

* Add `Error::SandboxBackend`, refactor invoke result

* Fix comments

* Rename `Error::SandboxBackend` to `Sandbox`

* Simplifies logic in `wasmer_backend::invoke`

* Fixes memory management

* Show Network ID when creating and inspecting (#10838)

Signed-off-by: Antonio Yang <yanganto@gmail.com>

* Reduce overhead of generating network event metrics (#10839)

* Fix reentrancy of FrozenBalance::died hook (#10473)

* assets: execute `died` hook outside of mutate

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* assets: extend tests for `died` hook

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* assets: update doc of FrozenBalance::died

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* assets: review fixes

- fix cases where `died` should not have been called
- use `Option<DeadConsequence>` instead of `DeadConsequence`

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* assets: update comment in mock.rs

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* assets: return `Remove` in dead_account

The return value is ignored in the only case that it is produced
by a call, but having it this way makes it more understandable.

Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo>

* allow trailing comma (#10841)

* build-script-utils: allow reading the git commit hash from env var (#10845)

* build-script-utils: allow reading the git commit hash from env var

* build-script-utils: make the env var name substrate specific

* Don't create DB leaves when syncing historic blocks (#10844)

* Don't create DB leaves when syncing historic blocks

* Changed leaves check and added test

* fmt

* cleanup, use unsigned tx when appropriate

* add weights

* fix broken link (#10846)

* subkey: Support `--version` cli command (#10853)

* subkey: Support `--version` cli command

* FMT :facepalm:

* Remove `u32_trait` (#10850)

* Remove `u32_trait`

This trait only existed because there wasn't any const generic support at time of creation. However,
we now have support for it :)

* FMT

* wasm-builder: Support latest nightly (#10837)

* wasm-builder: Support latest nightly

With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling
of optional dependencies. We currently have features that enable optional dependencies when `std` is
enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support
the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these
optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails
when someones enables these optional dependencies from the outside as well as via the `std` feature,
however we hope that no one is doing this at the moment. When namespaced features are enabled, these
dependencies needs to be enabled using `dep:dependency-name` to solve this properly.

https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features

* Remove accidentally added features

* update tests, remove unneeded params

* min authorities check before removal

* contracts: Fix `seal_call` weights (#10796)

* Fix call weights

* Fix instantiate benchmark

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Remove stale and superflous comments

* `decrement_refcount` should be infallible

* Don't hardcode increment_refcount, decrement_refcount

* Rename CopyIn/CopyOut

* Fix warning in tests

Co-authored-by: Parity Bot <admin@parity.io>

* Enable the `parking_lot` feature for `tokio` (#10855)

* [ci] fix publish-rustdocs (#10858)

* Fix beefy mock ecdsa keys (#10854)

Compressed ECDSA keys requires to have 0x02 or 0x03 as their first byte
in order to allow public key recovery.

Nevertheless the test was working because of the `unwrap_or_default()`
at the end of the conversion routine (i.e. the invalid keys were
converted to an empty vector).

* Mark sync_state_genSyncSpec JSON-RPC as safe (#10832)

* Mark sync_state_genSyncSpec JSON-RPC as safe

* Note that parameter is unused

* Ideally I'd wait for compilation to be finished before pushing, but it's really taking a long time

* Remove deny_unsafe parameter

* Remove unused dependency

* Reduce CPU overhead of gossip (#10859)

* sp-core-hashing: use the `digest::Digest` trait to handle the hashing function uniformly (#10835)

* sp-core-hashing: use sha3 instead of tiny-keccak

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* use blake2 instead of blake2-rfc

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* improve som hashing of sp-core and sp-api-proc-macro

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Some nits

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* cargo fmt

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Use fully qualified sytnax for `retain_mut` (#10865)

* add llvm (#10864)

* refactor election score (#10834)

* refactor election score

* Test for ord

* remove reference

* vec -> slice

* change iter to iter_by_significance

* improve doc

* fix typo

* add explanation about [u128; 3]

* consolidate threshold and epsilon

* random fixes

* rename

* remove Into

* make iter_by_sig private

* remove vec

* Fix tests

* Measure per byte and not kb for certain benchmarks (#10863)

* Tidy Democracy (#10867)

* add test

* Assorted refactorings

* complete test

* saturating math

* final check

* use `default`

Co-authored-by: Gav Wood <gavin@parity.io>

* cleanup storage maps, cleanup unused imports

* sc-executor: Improve logging (#10869)

Improves the logging by switching to `tracing` for a better log output. Besides that, it also adds a
trace for the function being executed.

* fix test issues

* Track allowed requests for state/warp sync (#10843)

* Track allowed requests for state/warp sync

* Added missing allowed_requests resets

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fmt

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Slots: Ensure that a race betwen finalized and best number is taken care off (#10876)

* Remove old lock file (#10879)

* Further reduce the CPU overhead of networking metrics (#10875)

* Simplify `num_connected_peers`

* Track requested peer counts

* Revert "Track requested peer counts"

This reverts commit 9f1c8704353df6afc17ed7e9f4ab8d8e29466ae4.

* Remove `substrate_sub_libp2p_peerset_num_requested` metric

* Remove two unused functions that I forgot to get rid of in previous commit

* Introduce `BoundedVec::iter_mut` (#10884)

* Introduce iters into BoundedVec

* Fix

* Remove unneeded funcs

* Update frame/support/src/storage/bounded_vec.rs

* Update frame/support/src/storage/bounded_vec.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Slots: Log total proposing duration as milliseconds (#10886)

Parachains have currently a total proposing time of 500ms, so it this
currently always prints `0`. While actually the value is not `0` ;)

* staking: Clarify reward calc docs (#10890)

* tracing: Adds `init_for_tests` (#10893)

This function is useful for tests. It will enable `TRACE` logging and also uses the libtest aware writer.

* make submissions pub (#10899)

* contracts: Allow stack height metering to be disabled (#10877)

* Allow stack height metering to be disabled

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Parity Bot <admin@parity.io>

* Rename Uniques Error::Unknown to something more sensible (#10895)

* Rename Uniques Error::Unknown to something more sensible

* Typos

* Typos

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* state-machine: Move all functionality from trie backend to the essence (#10904)

* state-machine: Move all functionality from trie backend to the essence

This is required for some future changes of me and it also makes more sense to have all the
functionality inside the essence. Besides that it changes the child root cache to directly
cache the hash.

* Update primitives/state-machine/src/trie_backend_essence.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* FMT

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* staking: Remove `SessionInterface` supertrait (#10901)

* consensus-slots: cleanup SlotDuration config (#10878)

* consensus-slots: cleanup the SlotDuration config

* fix tests

* address review comments

* add contracts pallet + rpc to runtime

* Replace libsecp256k1 with k256 in FRAME related code (#10883)

* Replace libsecp256k1 with k256 in beefy-mmr

* Port of FRAME `contracts` benchmarking from `libsecp256k1` to `k256`

* Newtype to allow `Pcg32` rng usage with `k256` in contracts benchmarks

* Use `sp-io::crypto` to generate dummy keys in `contracts` bechmarks

* More compact code

* Cargo fmt

* Build `sp-keystore` only for dev profile

* Move public key generation back to the `map`

* Clean obsolete BABE's weight data (#10748)

* Clean obsolete BABE weight data
* Take out test assertion from check closure
* Optimize metadata access using `HeaderMetadata` trait
* Apply suggestions from code review
* Introduce finalize and import pre-commit synchronous actions
* Do not hold locks between internal methods calls
* Remove unused generic bound
* Apply suggestions from code review
* Register BABE's pre-commit actions on `block_import` instead of `start_babe`
* PreCommit actions should be `Fn` instead of `FnMut`
* More robust safenet in case of malformed finality notifications

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Factor DB weights out into their own files (#10908)

* Factor DB weights out into their own files

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Review fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix CI

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Weights in own mod

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Get rid of unnecessary use of `async-std` in non-test code (#10891)

* No longer generate specs with consensus_engine field (#10346)

* No longer generate specs with consensus_engine field

* #[allow(unused)]

* Upgrading parity-scale-codec to v3 (#10825)

* Upgraded dependencies

* Adapting code to scale v3

* Empty commit to trigger CI

* Triggering CI

* Fixing UI test

* Remove superfluous dev-dep added by #9228

* Cryout for CI

* sc-cli: Fix bugs after switching to clap3 (#10920)

* sc-cli: Fix bugs after switching to clap3

Before switching to clap3 we support cli options like `--reserved-nodes A B` and after you needed to
pass `--reserved-nodes` cli option multiple times `--reserved-nodes A --reserved-nodes B`. This is
fixed by setting `multiple_occurrences(true)` option. This also done for all the other `Vec` cli
options in `sc-cli`. Besides that `--sync` wasn't supporting case insensitive parsing of the value.
This is now also supported. For both regressions a test is added. Besides that the pr removes all
the `rename_all = PascalCase` attributes, because they are not needed. All other `ArgEnum`s were
checked and all are already using `ignore_case(true)`.

* Bring back `PascalCase`, because otherwise it falls back to `kebab-case`...

* Storage benchmarking (#10897)

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP: DB benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Simplify code

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove old files

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove old files

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Minimize changes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add license

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove dependencies

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Extend template

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Linter

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Linter

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove default

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add feature

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove seed

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* CI wakeup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Review fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Adding doc

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Adding doc

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Improve template

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Do not expose columns

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix ColumnId

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Nicer template prints

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix json path

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Simplify `bench_write` logic

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Invert tx before the second commit

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add transfer assets extrinsic

* add chain ext

* Minor improvements to `bounded_vec` and `defensive`.  (#10873)

* Fix a few things in bounded_vec

* add test for try_extend

* Update frame/support/src/storage/bounded_vec.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* some review comments

* use swap

* remove clone

* use pop instead of truncate

* remove warn

* review comments

* Update frame/support/src/storage/bounded_vec.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix rustdoc

* fix links

* undo link

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Replace libsecp256k1 with secp256k1 (#10798)

* Replace libsecp256k1 with secp256k1

* Wipe ecdsa secret key from memory on drop

* Some comments for a known issue

* Safer core crypto primitives `from_slice` constructor

Previous version panics if slice lenght is not the expected one.

* Unit test fix

* Enable use of global secp256k1 context

* Better comments for ecdsa `Pair` drop

* Replace `libsecp256k1` with `seco256k1` in `beefy-mmr`

Used to convert ecdsa public key to ETH address

* Replace `libsecp256k1` with `secp256k1` in FRAME `contracts`benchmarks

* Temporary rollback of `beefy-mmr` to libsecp256k1

Check for detected build issues

* Cargo fmt

* Rollback of FRAME `contracts` benchmarks to `libsecp256k1`

* Rollback for unrelated changes

* Typo fix

* Add comments for deprecated `ecdsa_verify` and `secp256k1_ecdsa_recover`

* Integrate try-runtime into substrate node template (#10909)

* [10892-integrate-try-runtime-into-node-template] - Integrated try-runtime into node template

* [10892-integrate-try-runtime-into-node-template] Added match arms for try-runtime in command.rs

* [10892-integrate-try-runtime-into-node-template] Added match arms for try-runtime in command.rs

* Added feature flag for try-runtime in node-template/node and enabled try-runtime for node-template/runtime

* Added missing type annotations for try-runtime SubCommand in node-template

* Added missing type annotations for try-runtime SubCommand in node-template

* Implemented frame_try_runtime::TryRuntime<Block> for the node-template Runtime

* doc corrections (#10936)

* #10576: generic utility to unsubscribe from broadcast upon drop of the rx-side. (#10708)

* #10576: refactor `sc-utils::notification` and `sc-client-api::notifications`, so that they use common subscribe/unsubscribe routines

* Add some docs. Reorganise `sc-utils::notification`

* `sc-clent-api::notifications` and `sc-utils::notification` — ensure the SubscriptionGuard is dropped before the Rx-channel

* `sc-utils::pubsub::SubscriptionGuard` make it a bit more ergonomic.

Let the `Rx` to be put inside of the `SubscriptionGuard`, so that the latter shall guarantee the order:
- first unsubscribe;
- then drop the `Rx`.

* Being less zealous with splitting the modules into little pieces

* rework pubsub: the concrete usage should only define a good registry type

* sc-client-api::notifications: make it comply with the reworked pubsub

* cargo fmt

* make sc-client-api tests work

* Address the review notes

* cargo fmt

* Describe the behaviour of pubsub registry

* Doc-comments for module `sc-utils::pubsub`

* Fix: it used to send notifications regardless of the filter setup during subscription

* `sc-client-api::StorageNotifications` the API does not have to require mut-self-reference.

As a result `sc-service::Client` does not have to wrap its `storage_notifications` into a Mutex.

* cargo fmt

* Several changes addressing the notes by @bckhr.

- Remove the `impl Default for StorageNotifications<Block>`;
- no need for groupping the `remove_from` and `listen_from` into a separate `helpers` module;
- remove unnecessary import `use registry::SubscribeOp`.

* Add a doc-comment to the `sc-client::notifications::SubscribeOp`

* As per @bkchr note on the unproven assertion: behave gracefully upon receiving a duplicate subscription-ID.

* sc-utils::pubsub: log when a registry yields an ID that does point to an existing sink

* `sc-utils::notifications`: payload materialized lazily

* Update Cargo.lock (after adding `log` as a dependency to the `sc-utils`)

* `sc-client-api::notifications`: introduce a struct (instead of a type def) for the notification message

* Get rid of `sc-utils::pubsub::Channel` trait (instead just use the `sc-utils::mpsc`)

* The SubsID is no more generic: the fact it is a `Copy` is known — no need to pass it by ref

* sc-utils::pubsub internals do not have to be generic over the channel type

* Rename Hub::dispatch into Hub::send

* That method was unnecessary (`SubscriberSink::render_notification`)

* cargo fmt

* No need for a separate UnsubscribeGuard type

* Ditch the type-def of SubsID in the sc-utils::pubsub, instead — just use the crate::id_sequence::SeqID

* Return the <Registry as Dispatch>::Ret when sending an item

* Make the `Hub<M, R>::lock_registry(...)` method more ergonomic

* cargo doc links

* cargo doc links

* Use a simpler name for the type

* cargo doc links

* Derive `Default` rather than implement it

* Derive `Default` rather than implement it

* Remove an unnecessary usage of type_name

* Define a more cautious order between sinks.remove->registry.unsubscribe and registry.subscribe->sinks.insert

* Hub: lock_registry_for_tests->map_registry_for_tests — a safer choice for a public API

* Replace Mutex over the shared Registry with a ReentrableMutex+RefCell

* sc-utils::pubsub: add tests for a panicking registry

* Add the missing copyright headers

* Arc<Vec<_>> -> Arc<[_]>

* Bring back MaxNominations as a metadata constant (#10947)

* chain ext: parse params and call transfer func

* Bump futures from 0.3.16 to 0.3.19 (#10930)

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.16 to 0.3.19.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.19)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Clean up extra_constant renaming. (#10935)

Co-authored-by: Xiankun Cheng <xiankuncheng@Xiankuns-MacBook-Pro-2.local>

* Bump smallvec from 1.7.0 to 1.8.0 (#10949)

Bumps [smallvec](https://github.com/servo/rust-smallvec) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: smallvec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* new pallet: whitelist pallet (#10159)

* pallet whitelist

* refactor a bit

* fmt

* address audit

* improve tests

* return Ok + refund

* add test for dispatching failing

* add dispatch_whitelisted_call_with_preimage

* fmt

* better name

* Consume all data on decode

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add error docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove phantom data

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use rust 2021

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update crate features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_whitelist --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/whitelist/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Bump Preimage max size

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_whitelist --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/whitelist/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Parity Bot <admin@parity.io>

* Release primitive crates (#10937)

* cargo set-version --bump major -p pallet-contracts-primitives

* cargo set-version --bump major -p sp-core

* cargo set-version --bump major -p sp-runtime-interface

* cargo set-version --bump major -p sp-wasm-interface

* cargo set-version --bump major -p sp-runtime

* cargo set-version --bump major -p sp-storage

* cargo set-version --bump major -p sp-rpc

* cargo set-version --bump major -p sp-io

* cargo set-version --bump major -p sp-trie

* cargo set-version -p sp-state-machine  -- 0.12.0

* cargo set-version -p sp-externalities  -- 0.12.0

* cargo set-version -p sp-keystore -- 0.12.0

* cargo set-version --bump major -p sp-keyring

* cargo set-version --bump major -p sp-version

* cargo set-version --bump major -p sp-tracing

* cargo set-version --bump major -p sp-application-crypto

* cargo set-version --bump major -p sp-arithmetic

* cargo unleash version bump-major -p sp-runtime-interface-proc-macro

* Add codec max-encoded-len feature to sp-arithmetic

* cargo unleash version bump-major -p sp-core-hashing-proc-macro

* Update new whitelist pallet to new primitive versions (#10953)

* sp-trie: Switch to thiserror and some other small cleanups (#10954)

* sp-trie: Switch to thiserror and some other small cleanups

* Add some extra method for converting a compact proof to a memory db

* Only maintain at most 1 `UnlockChunk` per era (#10670)

* Only maintain at most 1 `UnlockChunk` per era

* Bound `unlocking`

* Run cargo +nightly-2021-10-29 fmt

* Make benchmarks stuff compile

* Update frame/staking/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Remove DerefMut; Implement neccesary methods directly

* Doc comments for new BoundedVec methods

* Fix benchmarks

* wip bonded_vec macro

* Correct rust doc

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update staking::Config impls

* Add MaxUnlockingChunks to more places

* Use defensive saturating add

* FMT

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* staking: Expose `MaxUnlockingChunks` in metadata as a const (#10958)

* Allow trailing commas for `bounded_vec!` (#10959)

* SimpleSlotWorker: Do not implement `SlotWorker` for all types implementing `SimpleSlotWorker` (#10934)

Because Rust currently doesn't support specialization, it prevents users from implementing
`SlotWorker` for their own types. This pr solves this by removing the generic implementation of
`SlotWorker` for `SimpleSlotWorker` and providing some wrapper type for that.

* Bump digest from 0.10.2 to 0.10.3 (#10960)

Bumps [digest](https://github.com/RustCrypto/traits) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/RustCrypto/traits/releases)
- [Commits](https://github.com/RustCrypto/traits/compare/digest-v0.10.2...digest-v0.10.3)

---
updated-dependencies:
- dependency-name: digest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scale-info from 2.0.0 to 2.0.1 (#10965)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/commits)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add pr-custom-review config (#10968)

* Revise how staking configurations are set (#10955)

* Revise how staking configurations are set

fixes #10938

* Fix and add additional tests

* Format

* Formatting

* Add doc

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/staking/src/tests.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Format

* Fix build

* Update weights.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>

* Bump libc from 0.2.112 to 0.2.119 (#10967)

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.112 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.112...0.2.119)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump parity-db from 0.3.5 to 0.3.8 (#10970)

Bumps [parity-db](https://github.com/paritytech/parity-db) from 0.3.5 to 0.3.8.
- [Release notes](https://github.com/paritytech/parity-db/releases)
- [Commits](https://github.com/paritytech/parity-db/commits/v0.3.8)

---
updated-dependencies:
- dependency-name: parity-db
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [EPM pallet]: remove `number of signed submissions` (#10945)

* [EPM pallet]: remove `number of signed submissions`

Closing #9229

* fix tests

* remove needless assert

* Update frame/election-provider-multi-phase/src/lib.rs

* cargo fmt

Signed-off-by: Niklas <niklasadolfsson1@gmail.com>

* fix grumbles

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* drop `num_signed_submissions` in WeightInfo too

* fix build

Co-authored-by: Parity Bot <admin@parity.io>

* Refactored block body database scheme (#10779)

* Refactored tx storage database scheme

* Bump parity-db

* fmt

* Fix handling invalid index size + test

* Removed superflous result

* Minor changes

* fmt

* Deprecate "paritydb-experimental" CLI in favour or "paritydb" (#10975)

* Deprecate paritydb-experimental

* Updated comment

* [contracts] Stabilize "seal0" `is_contract` and `caller_is_origin`  (#10971)

* stabilize `seal_is_contract`

* stabilize `seal_caller_is_origin`

* Trie version migration pallet  (#10073)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* migration pallet

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit d35f273b7d67b1b85a9e72973cab13c5c156c1d3.

* fix test

* fix child migration calls.

* useless closure

* remove remaining state_hash variables.

* Fix and add more tests

* Remove outdated comment

* useless inner hash

* fmt

* remote tests

* finally ksm works

* batches are broken

* clean the benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Update frame/state-trie-migration/src/lib.rs

* brand new version

* fix build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* new test structure

* new testing stuff from emeric

* Add commit_all, still not working

* Fix all tests

* add comment

* we have PoV tracking baby

* document stuff, but proof size is still wrong

* FUCK YEAH

* a big batch of review comments

* add more tests

* tweak test

* update config

* some remote-ext stuff

* delete some of the old stuff

* sync more files with master to minimize the diff

* Fix all tests

* make signed migration a bit more relaxed

* add witness check to signed submissions

* allow custom migration to also go above limit

* Fix these pesky tests

* ==== removal of the unsigned stuff ====

* Make all tests work again

* separate the tests from the logic so it can be reused easier

* fix overall build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Slightly better termination

* some final tweaks

* Fix tests

* Restrict access to signed migrations

* address most of the review comments

* fix defensive

* New simplified code

* Fix weights

* fmt

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make the tests correctly fail

* Fix build

* Fix build

* try and fix the benchmarks

* fix build

* Fix cargo file

* Fix runtime deposit

* make rustdoc happy

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>

* BEEFY and GRANDPA protocol names should use full genesis hash (#10974)

std::fmt::Display shows formats as reduced hash (e.g. 0xb0a8…dafe)

Use hex::encode to format full hash.

Signed-off-by: acatangiu <adrian@parity.io>

* Move weight constants to own mod (#10980)

* Move block+ext weights to own mod

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Unused import

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add block usage logs to system pallet (#10940)

* add block usage logs to system pallet

* add Debug

* use % instead of default Debug impl

* change formatting

* revert

* update runtime storage, update chain extension

* cleanup chain ext

* Fix the undeterministic storage proof recorded for the same execution (#10915)

* Add a test case for the determinism of recorded proof

* Replace HashMap with BTreeMap for the actual proof records

* cargo +nightly fmt --all

* Store the trie nodes in BTreeSet for StorageProof

* Nit

* Revert the BTreeMap changes and sort when converting to storage proof

* Remove PartialEq from StorageProof

* Remove unnecessary change

* Add `compare` method to StorageProof

* FMT

* Dummy change to trigger CI

* Use `BTreeSet` for StorageProof and keep using `Vec` for CompactProof

* Update comment on `iter_nodes`

* Revert `PartialEq` removal

* create iris ledger pallet

* ledger funcs in chain ext

* rename Iris -> IrisAssets, update readmes

* Minor Uniques pallet improvements and XCM v3 preparations (#10896)

* Introduce Helper to Uniques for benchmark stuff

* Fixes

* Formatting

* Featuregate the Helper, include ContainsPair

* Introduce & use EnsureOriginWithArg

* Benchmarking

* Docs

* More ContainsBoth helpers

* Formatting

* Formatting

* Fixes

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* election provider support: Update some test only types (#10983)

* Feedback from @XLC for Referenda Pallet (#10991)

* feedback from @xlc

* english

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make bags-list generic over node value and instantiable (#10997)

* make instantiable

* update

* cargo fmt

* Clean up

* bags-list: Make it generic over node value

* Respond to some feedback

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Add back default impl for weight update worst case

* Update to Score in more places'

* Use VoteWeight, not u64 to reduce test diff

* FMT

* FullCodec implies Codec

* formatting

* Fixup bags list remote test

Co-authored-by: doordashcon <jesse.chejieh@gmail.com>
Co-authored-by: Doordashcon <90750465+Doordashcon@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* sc-finality-grandpa: use the #[from] attriute to remove boilerplate code (#11003)

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* contracts: Add test to verify unique trie ids (#10914)

* Add test to verify unique trie ids

* Rename trie_seed to nonce

* Rename AccountCounter -> Nonce

* fmt

* sp-core: `full_crypto` doesn't imply `std` (#11006)

* sp-core: `full_crypto` doesn't imply `std`

This pr changes the feature set of `secp256k1` to not use `global-context` when only the
`full_crypto` is enabled. It will be slower when the `std` feature is not enabled as the context
always needs to be recreated, but that is fine.

* Update client/cli/src/arg_enums.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Remove unused `parent_hash` in `OverlayedChanges::into_storage_changes` (#11011)

Ref https://github.com/paritytech/substrate/pull/10922#issuecomment-1064258443

CC @cheme

* iris ledger pallet -> transfer not yet functioning

* sp-api: Don't be dirty (#11015)

Ensure that the sp api macros don't use functions without providing the full path to the function.
This hygiene ensures that we don't actually try to call a method of an imported trait for example.

* unlock and transfer works

* Update clap to the latest version (#11017)

* Update clap to the latest version

Besides that it also removes some `structopt` leftovers from some docs.

* Fix compile errors

* More fixes

* Move scripts used in CI to the new location (#11008)

Move scripts used in CI to the new location - **./scripts/ci/**

* Move github scripts

* Move more files

* Move ci scripts and fix dependencies

* Update docs/node-template-release.md

Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>

* Remove Cargo.lock

* Apply suggestions from code review

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

* Make more paths uniform

Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

* Move `sp-npos-elections-solution-type` to `frame-election-provider-support` (#11016)

* Move `sp-npos-elections-solution-type`
to `frame-election-provider-support`
First stab at it, will need to amend some more stuff

* Fixing tests

* Fixing tests

* Fixing cargo.toml for std configuration

* fmt

* Committing suggested changes
renaming, and re exporting macro.

* Removing unneeded imports

* shell.nix: Update to a newer nightly (#11028)

* update chain ext

* update naming, formatting, start tests

* Update lockfile (#11035)

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Additional `benchmark-storage` flags (#11004)

* Fix typos

* Enable overwriting handlebars template

* Optionally name json output or disable json altogether

* Don't write to json by default

* Include block id in handlebars output

* Include warmups for write benchmarks

* PR comments

* Drop unnecessary file extension

* Use more appropriate types

* Use more appropriate error message

* More use of more appropriate types

* Rework write benchmark warmups

* Run same benchmark for both read and write

* SharedData: Update locks to mention possible deadlocks (#11034)

* SharedData: Update locks to mention possible deadlocks

* Update `Cargo.lock`

* Stabilize `seal_delegate_call` (#11037)

* add tests

* re-enumerate chain ext func ids

* Bump names from 0.12.0 to 0.13.0 (#11047)

Bumps [names](https://github.com/fnichol/names) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/fnichol/names/releases)
- [Changelog](https://github.com/fnichol/names/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fnichol/names/compare/v0.12.0...v0.13.0)

---
updated-dependencies:
- dependency-name: names
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* State migration rpc (#10981)

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right …
DaviRain-Su pushed a commit to octopus-network/substrate that referenced this pull request Aug 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A2-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants