Initial Substrate Header Chain Implementation (paritytech#296)

* Add pallet template from Substrate Dev Hub

* Clean up un-needed stuff from template

* Sketch out dispatchable interface

* Introduce notion of finality chain

* Add dependencies which were removed during a rebase

* Sketch out idea for finality header-chain pallet

* Sketch out ChainVerifier trait

* Add storage parameter to verifier

* Write out some things I think I need for finality verification

* Add some pseudocode for marking finalized headers

* Remove parity_scale_codec duplicate

* Move verification logic into pallet

I've been struggling with getting the generic types between the storage and verifier
traits to play nice with each other. As a way to continue making progress I'm moving
everything to the pallet. This way I hope to make progress towards a functional
pallet.

* Start doing verification around authority set changes

* Remove commented BridgeStorage and ChainVerifier traits

* Create Substrate bridge primitives crate

* Add logic for updating scheduled authority sets

* Introduce notion of imported headers

* Implement basic header ancestry checker

* Add mock runtime for tests

* Add testing boilerplate

* Add some storage read/write sanity tests

* Add some basic header import tests

* Add tests for ancestry proofs

* Create helper for changing authority sets

* Fix authority set test

Fixes a problem with how the scheduled change was counted as well as
a SCALE encoding issue

* Correctly check for scheduled change digests

There's no guarantee that the consensus digest item will be the last
one in a header, which is how it was previously being checked.

Thanks to Andre for pointing me to the Grandpa code that does this.

* Mark imported headers as finalized when appropriate

When a header that finalizes a chain of headers is succesfully imported
we also want to mark its ancestors as finalized.

* Add helper for writing test headers

* Add test helper for scheduling authority set changes

* Bump Substrate pallet and primitives to rc6

* Remove Millau verifier implementation

* Add some doc comments

* Remove some needless returns

* Make Clippy happy

* Split block import from finalization

* Make tests compile again

* Add test for finalizing header after importing children

* Create a test stub for importing future justifications

* Start adding genesis config

* Reject justifications from future

We should only be accepting justifications for the header
which enacted the current authority set. Any ancestors of
that header which require a justification can be imported
but they must not be finalized.

* Add explanation to some `expect()` calls

* Start adding GenesisConfig

* Plug genesis config into runtime

* Remove tests module

* Check for overflow when updating authority sets

* Make verifier take ownership of headers during import

* Only store best finalized header hash

Removed the need to store the whole header, since we store
it was part of the ImportedHeaders structure anyways

* Add some helpers to ImportedHeader

* Update ancestry checker to work with ImportedHeaders

* Update ancestry tests to use ImportedHeaders

* Update import tests to use ImportedHeaders

* Clean up some of the test helpers

* Remove stray dbg!

* Add doc comments throughout

* Remove runtime related code

* Fix Clippy warnings

* Remove trait bound on ImportedHeader struct

* Simplify checks in GenesisConfig

* Rename `get_header_by_hash()`

* Alias `parity_scale_codec` to `codec`

* Reword Verifier documentation

* Missed codec rename in tests

* Split ImportError into FinalizationError

* Remove ChainVerifier trait

This trait was a remenant of the original design, and it is not required
at the moment. Something like it should be added back in the future to
ensure that other chains which conform to this interface can be used
by higher-level bridge applications.

* Fix the verifier tests so they compile

* Implement Deref for ImportedHeader

* Get rid of `new` methods for some Substrate primitives

* Ensure that a child header's number follows its parent's

* Prevent ancestry checker from aimlessly traversing to genesis

If an ancestor which was newer than the child header we were checking we
would walk all the way to genesis before realizing that we weren't related.
This commit fixes that.

* Remove redundant clones

* Ensure that old headers are not finalized

Prevents a panic where if the header being imported and `best_finalized`
were the same header the ancestry checker would return an empty list. We
had made an assumption that the list would always be populated, and if this
didn't hold we would end up panicking.

* Disallow imports at same height as `best_finalized`

* Fix Clippy warnings

* Make NextScheduledChange optional

* Rework how scheduled authority set changes are enacted

We now require a justification for headers which _enact_ changes
instead of those which _schedule_ changes. A few changes had to
be made to accomodate this, such as changing when we check for
scheduled change logs in incoming headers.

* Update documentation for Substrate Primitives

* Clarify why we skip header in requires_justification check

* Add description to assert! call

* Fix formatting within macros

* Remove unused dependencies from runtime

* Remove expect call in GenesisConfig

* Turn FinalityProof into a struct

* Add some inline TODOs for follow up PRs

* Remove test which enacted multiple changes

This should be added back at some later point in time, but right now
the code doesn't allow for this behaviour.

* Use `contains_key` when checking for header

This is better than using `get().is_some()` since we skip
decoding the storage value

* Use initial hash when updating best_finalized

* Add better checks around enacting scheduled changes

* Rename finality related functions

* Appease Clippy

bridges/modules/substrate/Cargo.toml

@@ -8,17 +8,32 @@ license = "GPL-3.0-or-later WITH Classpath-exception-2.0"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-codec = { package = "parity-scale-codec", version = "1.3.4", default-features = false }
+bp-header-chain = { path = "../../primitives/header-chain", default-features = false }
+bp-substrate = { path = "../../primitives/substrate", default-features = false }
 finality-grandpa = { version = "0.12.3", default-features = false }
 hash-db = { version = "0.15.2", default-features = false }
+serde = { version = "1.0", optional = true }
+
+[dependencies.codec]
+package = "parity-scale-codec"
+version = "1.3.1"
+default-features = false
+features = ["derive"]
 
 # Substrate Based Dependencies
+
 [dependencies.frame-support]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
 default-features = false
 git = "https://github.com/paritytech/substrate/"
 
+[dependencies.frame-system]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+default-features = false
+git = "https://github.com/paritytech/substrate/"
+
 [dependencies.sp-finality-grandpa]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
@@ -43,6 +58,12 @@ tag = 'v2.0.0-rc6'
 default-features = false
 git = "https://github.com/paritytech/substrate/"
 
+# Dev Dependencies
+[dev-dependencies.sp-io]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+git = "https://github.com/paritytech/substrate/"
+
 [dev-dependencies.sp-core]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
@@ -67,8 +88,11 @@ std = [
 	"codec/std",
 	"finality-grandpa/std",
 	"frame-support/std",
+	"frame-system/std",
+	"serde",
 	"sp-finality-grandpa/std",
 	"sp-runtime/std",
 	"sp-std/std",
 	"sp-trie/std",
+	"sp-io/std",
 ]

bridges/modules/substrate/src/justification.rs

@@ -26,6 +26,7 @@ use frame_support::RuntimeDebug;
 use sp_finality_grandpa::{AuthorityId, AuthoritySignature, SetId};
 use sp_runtime::traits::Header as HeaderT;
 use sp_std::collections::{btree_map::BTreeMap, btree_set::BTreeSet};
+use sp_std::prelude::Vec;
 
 /// Justification verification error.
 #[derive(RuntimeDebug, PartialEq)]

bridges/modules/substrate/src/lib.rs

@@ -14,5 +14,263 @@
 // You should have received a copy of the GNU General Public License
 // along with Parity Bridges Common.  If not, see <http://www.gnu.org/licenses/>.
 
+//! Substrate Bridge Pallet
+//!
+//! This pallet is an on-chain light client for chains which have a notion of finality.
+//!
+//! It has a simple interface for achieving this. First it can import headers to the runtime
+//! storage. During this it will check the validity of the headers and ensure they don't conflict
+//! with any existing headers (e.g they're on a different finalized chain). Secondly it can finalize
+//! an already imported header (and its ancestors) given a valid Grandpa justification.
+//!
+//! With these two functions the pallet is able to form a "source of truth" for what headers have
+//! been finalized on a given Substrate chain. This can be a useful source of info for other
+//! higher-level applications.
+
+#![cfg_attr(not(feature = "std"), no_std)]
+// Runtime-generated enums
+#![allow(clippy::large_enum_variant)]
+
+use bp_substrate::{AuthoritySet, ImportedHeader, ScheduledChange};
+use frame_support::{decl_error, decl_module, decl_storage, dispatch};
+use frame_system::ensure_signed;
+use sp_runtime::traits::Header as HeaderT;
+use sp_std::{marker::PhantomData, prelude::*};
+
 mod justification;
 mod storage_proof;
+mod verifier;
+
+#[cfg(test)]
+mod mock;
+
+type Hash<T> = <T as HeaderT>::Hash;
+type Number<T> = <T as HeaderT>::Number;
+
+pub trait Trait: frame_system::Trait {}
+
+decl_storage! {
+	trait Store for Module<T: Trait> as SubstrateBridge {
+		/// Hash of the best finalized header.
+		BestFinalized: T::Hash;
+		/// Headers which have been imported into the pallet.
+		ImportedHeaders: map hasher(identity) T::Hash => Option<ImportedHeader<T::Header>>;
+		/// The current Grandpa Authority set.
+		CurrentAuthoritySet: AuthoritySet;
+		/// The next scheduled authority set change.
+		///
+		// Grandpa doesn't require there to always be a pending change. In fact, most of the time
+		// there will be no pending change available.
+		NextScheduledChange: Option<ScheduledChange<Number<T::Header>>>;
+	}
+	add_extra_genesis {
+		config(initial_header): Option<T::Header>;
+		config(initial_authority_list): sp_finality_grandpa::AuthorityList;
+		config(initial_set_id): sp_finality_grandpa::SetId;
+		config(first_scheduled_change): Option<ScheduledChange<Number<T::Header>>>;
+		build(|config| {
+			assert!(
+				!config.initial_authority_list.is_empty(),
+				"An initial authority list is needed."
+			);
+
+			let initial_header = config
+				.initial_header
+				.clone()
+				.expect("An initial header is needed");
+
+			<BestFinalized<T>>::put(initial_header.hash());
+			<ImportedHeaders<T>>::insert(
+				initial_header.hash(),
+				ImportedHeader {
+					header: initial_header,
+					requires_justification: false,
+					is_finalized: true,
+				},
+			);
+
+			let authority_set =
+				AuthoritySet::new(config.initial_authority_list.clone(), config.initial_set_id);
+			CurrentAuthoritySet::put(authority_set);
+
+			if let Some(ref change) = config.first_scheduled_change {
+				<NextScheduledChange<T>>::put(change);
+			};
+		})
+	}
+}
+
+decl_error! {
+	pub enum Error for Module<T: Trait> {
+		/// This header has failed basic verification.
+		InvalidHeader,
+		/// This header has not been finalized.
+		UnfinalizedHeader,
+	}
+}
+
+decl_module! {
+	pub struct Module<T: Trait> for enum Call where origin: T::Origin {
+		type Error = Error<T>;
+
+		/// Import a signed Substrate header into the runtime.
+		///
+		/// This will perform some basic checks to make sure it is fine to
+		/// import into the runtime. However, it does not perform any checks
+		/// related to finality.
+		// TODO: Update weights [#78]
+		#[weight = 0]
+		pub fn import_signed_header(
+			origin,
+			header: T::Header,
+		) -> dispatch::DispatchResult {
+			let _ = ensure_signed(origin)?;
+			frame_support::debug::trace!(target: "sub-bridge", "Got header {:?}", header);
+
+			let mut verifier = verifier::Verifier {
+				storage: PalletStorage::<T>::new(),
+			};
+
+			let _ = verifier
+				.import_header(header)
+				.map_err(|_| <Error<T>>::InvalidHeader)?;
+
+			Ok(())
+		}
+
+		/// Import a finalty proof for a particular header.
+		///
+		/// This will take care of finalizing any already imported headers
+		/// which get finalized when importing this particular proof, as well
+		/// as updating the current and next validator sets.
+		// TODO: Update weights [#78]
+		#[weight = 0]
+		pub fn finalize_header(
+			origin,
+			hash: Hash<T::Header>,
+			finality_proof: Vec<u8>,
+		) -> dispatch::DispatchResult {
+			let _ = ensure_signed(origin)?;
+			frame_support::debug::trace!(target: "sub-bridge", "Got header hash {:?}", hash);
+
+			let mut verifier = verifier::Verifier {
+				storage: PalletStorage::<T>::new(),
+			};
+
+			let _ = verifier
+				.import_finality_proof(hash, finality_proof.into())
+				.map_err(|_| <Error<T>>::UnfinalizedHeader)?;
+
+			Ok(())
+		}
+	}
+}
+
+/// Expected interface for interacting with bridge pallet storage.
+// TODO: This should be split into its own less-Substrate-dependent crate
+pub trait BridgeStorage {
+	/// The header type being used by the pallet.
+	type Header: HeaderT;
+
+	/// Write a header to storage.
+	fn write_header(&mut self, header: &ImportedHeader<Self::Header>);
+
+	/// Get the best finalized header the pallet knows of.
+	fn best_finalized_header(&self) -> ImportedHeader<Self::Header>;
+
+	/// Update the best finalized header the pallet knows of.
+	fn update_best_finalized(&self, hash: <Self::Header as HeaderT>::Hash);
+
+	/// Check if a particular header is known to the pallet.
+	fn header_exists(&self, hash: <Self::Header as HeaderT>::Hash) -> bool;
+
+	/// Get a specific header by its hash.
+	///
+	/// Returns None if it is not known to the pallet.
+	fn header_by_hash(&self, hash: <Self::Header as HeaderT>::Hash) -> Option<ImportedHeader<Self::Header>>;
+
+	/// Get the current Grandpa authority set.
+	fn current_authority_set(&self) -> AuthoritySet;
+
+	/// Update the current Grandpa authority set.
+	///
+	/// Should only be updated when a scheduled change has been triggered.
+	fn update_current_authority_set(&self, new_set: AuthoritySet);
+
+	/// Replace the current authority set with the next scheduled set.
+	///
+	/// Returns an error if there is no scheduled authority set to enact.
+	fn enact_authority_set(&mut self) -> Result<(), ()>;
+
+	/// Get the next scheduled Grandpa authority set change.
+	fn scheduled_set_change(&self) -> Option<ScheduledChange<<Self::Header as HeaderT>::Number>>;
+
+	/// Schedule a Grandpa authority set change in the future.
+	fn schedule_next_set_change(&self, next_change: ScheduledChange<<Self::Header as HeaderT>::Number>);
+}
+
+/// Used to interact with the pallet storage in a more abstract way.
+#[derive(Default, Clone)]
+pub struct PalletStorage<T>(PhantomData<T>);
+
+impl<T> PalletStorage<T> {
+	fn new() -> Self {
+		Self(PhantomData::<T>::default())
+	}
+}
+
+impl<T: Trait> BridgeStorage for PalletStorage<T> {
+	type Header = T::Header;
+
+	fn write_header(&mut self, header: &ImportedHeader<T::Header>) {
+		let hash = header.header.hash();
+		<ImportedHeaders<T>>::insert(hash, header);
+	}
+
+	fn best_finalized_header(&self) -> ImportedHeader<T::Header> {
+		let hash = <BestFinalized<T>>::get();
+		self.header_by_hash(hash)
+			.expect("A finalized header was added at genesis, therefore this must always exist")
+	}
+
+	fn update_best_finalized(&self, hash: Hash<T::Header>) {
+		<BestFinalized<T>>::put(hash)
+	}
+
+	fn header_exists(&self, hash: Hash<T::Header>) -> bool {
+		<ImportedHeaders<T>>::contains_key(hash)
+	}
+
+	fn header_by_hash(&self, hash: Hash<T::Header>) -> Option<ImportedHeader<T::Header>> {
+		<ImportedHeaders<T>>::get(hash)
+	}
+
+	fn current_authority_set(&self) -> AuthoritySet {
+		CurrentAuthoritySet::get()
+	}
+
+	fn update_current_authority_set(&self, new_set: AuthoritySet) {
+		CurrentAuthoritySet::put(new_set)
+	}
+
+	fn enact_authority_set(&mut self) -> Result<(), ()> {
+		if <NextScheduledChange<T>>::exists() {
+			let new_set = <NextScheduledChange<T>>::take()
+				.expect("Ensured that entry existed in storage")
+				.authority_set;
+			self.update_current_authority_set(new_set);
+
+			Ok(())
+		} else {
+			Err(())
+		}
+	}
+
+	fn scheduled_set_change(&self) -> Option<ScheduledChange<Number<T::Header>>> {
+		<NextScheduledChange<T>>::get()
+	}
+
+	fn schedule_next_set_change(&self, next_change: ScheduledChange<Number<T::Header>>) {
+		<NextScheduledChange<T>>::put(next_change)
+	}
+}