BABE Epochs (#3028)

* Add `epoch` field to `SlotInfo`

* Add slot calculations

* More work on epochs in BABE

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Typo: `/` not `%` for division

* Delete useless `LastSlotInEpoch::put(false)`

* Bump `spec_version`

* Make test suite pass again

* Implement BABE epoch randomness signing

* Try to fix compilation

Currently causes a stack overflow in the compiler

* Fix rustc stack overflow

* Add missing `PartialEq` and `Eq` implementations

* Fix compile errors in test suite

* Another silly compile error

* Clone `epoch`

* Fix compile error in benchmarks

* Implement `clone` for `Epoch`

* Merge master

* AUTHORING TEST PASSES!!!

* Fix compilation

* Bump `spec_version`

* Fix compilation

* Fix compilation (again)

* Remove an outdated FIXME

* Fix run.sh and move it to scripts/

* Delete commented-out code

* Fix documentation

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Fix BABE initialization and refactor

* Respond to review

* typo

* Remove useless data in `CheckedHeader::Deferred`

* Remove `slot_number` from Epoch

It is not needed, and only served to waste space and cause confusion.

* Remove epoch from BABE digests

* Move digest.rs to primitives

* Fix incorrect warning names

* Fix compile error

* Consistent field naming for BABE digests

* More compiler error fixex

* Unbound variable

* more compile errors

* another compile error

* Fix compile errors in runtime

* another compile error

* Another compile error

* Fix wasm build

* missing import

* Fix more compile errors

* yet another compile error

* compile fix in test runtime

* Fix and simplify the BABE runtime

The BABE runtime was massively overcomplicated and also wrong.  It
assumed it needed to:

1. delay new authorities taking effect until the next epoch
2. not delay emitting `Consensus` digests to mark epoch changes

However, the first is handled by the `srml_session` crate, and the
second is flat-out incorrect: `Consensus` digests take effect
immediately.  Furthermore, `srml_babe` tried to duplicate the
functionality of `srml_session::PeriodicSession`, but did it both
clumsily and incorrectly.  Fortunately, the new code is simpler and far
more likely to be correct.

* Use `system` to get the test authorities

The genesis block used by tests defines no authorities.  Only the test
suite is affected.

* Fix test runtime impl for BabeApi::epoch() with std

* Fix compilation

* Cached authorities are in the form of an epoch

not a `Vec<AuthorityId>`.

* `slots_per_epoch` is not fixed in general

The BABE code previously assumed `slots_per_epoch` to be a constant,
but that assumption is false in general.  Furthermore, removing this
assumption also allows a lot of code to go away.

* fix compile error

* Implement epoch checker

* Fix runtime compilation

* fork-tree: add method for finding a node in the tree

* babe: register epoch transitions in fork tree and validate them

* fork-tree: add method for arbitrary pruning

* Expose the queued validator set to SRML modules

BABE needs to know not only what the current validator set is, but also
what the next validator set will be.  Expose this to clients of the
session module.

* Bump hex-literal

Hopefully this will fix the panic

* babe: prune epoch change fork tree on finality

* babe: validate epoch index on transition

* babe: persist epoch changes tree

* Fix compile error in tests

* Fix compile error in tests

* Another compile error in tests

* Fix compilation of tests

* core: move grandpa::is_descendent_of to client utils

* babe: use is_descendent_of from client utils

* babe: extract slot_number from pre_digest in import_block

* Move BABE testsuite to its own file

* Initial part of test code

* Missing `WeightMultiplierUpdate` in test-runtime

* bump `spec_version`

* Add a test that a very bogus is rejected

* Run the tests again

* Fix compiler diagnostics

* Bump `spec_version`

* Initial infrastructure for mutation testing

* Mutation testing of block import

* babe: revert epoch changes in case of block import error

* babe: fix logging target

* babe: BabeBlockImport doesn't box inner BlockImport

* babe: fix epoch check in block import

* babe: populate authorities cache on block authorship

* babe: remove unused functions

* babe: use RANDOMNESS_LENGTH const

* babe: remove unneeded config parameters

* core: revert change to hex dependency version

* cleanup gitignore

* babe: add docs to aux_schema

* babe: remove useless drops in tests

* babe: remove annoying macos smart quotes

* fork-tree: docs

* fork-tree: add tests

* babe: style

* babe: rename randomness config variable

* babe: remove randomness helper function

* babe: style fixes

* babe: add docs

* babe: fix tests

* node: bump spec_version

* babe: fix tests

Dockerfile

@@ -11,7 +11,7 @@ WORKDIR /substrate
 COPY . /substrate
 
 RUN apt-get update && \
-	apt-get upgrade -y && \
+	apt-get dist-upgrade -y && \
 	apt-get install -y cmake pkg-config libssl-dev git clang
 
 RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && \
@@ -21,7 +21,7 @@ RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && \
 	cargo install --git https://github.com/alexcrichton/wasm-gc && \
 	rustup default nightly && \
 	rustup default stable && \
-	cargo build --$PROFILE
+	cargo build "--$PROFILE"
 
 # ===== SECOND STAGE ======
 

core/client/src/client.rs

@@ -1799,6 +1799,53 @@ impl<B, E, Block, RA> backend::AuxStore for Client<B, E, Block, RA>
 		crate::backend::AuxStore::get_aux(&*self.backend, key)
 	}
 }
+
+/// Utility methods for the client.
+pub mod utils {
+	use super::*;
+	use crate::{backend::Backend, blockchain, error};
+	use primitives::H256;
+
+	/// Returns a function for checking block ancestry, the returned function will
+	/// return `true` if the given hash (second parameter) is a descendent of the
+	/// base (first parameter). If the `current` parameter is defined, it should
+	/// represent the current block `hash` and its `parent hash`, if given the
+	/// function that's returned will assume that `hash` isn't part of the local DB
+	/// yet, and all searches in the DB will instead reference the parent.
+	pub fn is_descendent_of<'a, B, E, Block: BlockT<Hash=H256>, RA>(
+		client: &'a Client<B, E, Block, RA>,
+		current: Option<(&'a H256, &'a H256)>,
+	) -> impl Fn(&H256, &H256) -> Result<bool, error::Error> + 'a
+		where B: Backend<Block, Blake2Hasher>,
+			  E: CallExecutor<Block, Blake2Hasher> + Send + Sync,
+	{
+		move |base, hash| {
+			if base == hash { return Ok(false); }
+
+			let mut hash = hash;
+			if let Some((current_hash, current_parent_hash)) = current {
+				if base == current_hash { return Ok(false); }
+				if hash == current_hash {
+					if base == current_parent_hash {
+						return Ok(true);
+					} else {
+						hash = current_parent_hash;
+					}
+				}
+			}
+
+			let tree_route = blockchain::tree_route(
+				#[allow(deprecated)]
+				client.backend().blockchain(),
+				BlockId::Hash(*hash),
+				BlockId::Hash(*base),
+			)?;
+
+			Ok(tree_route.common_block().hash == *base)
+		}
+	}
+}
+
 #[cfg(test)]
 pub(crate) mod tests {
 	use std::collections::HashMap;

core/client/src/lib.rs

@@ -60,6 +60,7 @@ pub use crate::client::{
 	BlockBody, BlockStatus, ImportNotifications, FinalityNotifications, BlockchainEvents,
 	BlockImportNotification, Client, ClientInfo, ExecutionStrategies, FinalityNotification,
 	LongestChain,
+	utils,
 };
 #[cfg(feature = "std")]
 pub use crate::notifications::{StorageEventStream, StorageChangeSet};

core/consensus/babe/Cargo.toml

@@ -19,7 +19,9 @@ client = { package = "substrate-client", path = "../../client" }
 consensus_common = { package = "substrate-consensus-common", path = "../common" }
 slots = { package = "substrate-consensus-slots", path = "../slots"  }
 runtime_primitives = { package = "sr-primitives", path = "../../sr-primitives" }
+fork-tree = { path = "../../utils/fork-tree" }
 futures = "0.1.26"
+futures03 = { package = "futures-preview", version = "0.3.0-alpha.17", features = ["compat"] }
 tokio-timer = "0.2.11"
 parking_lot = "0.8.0"
 log = "0.4.6"
@@ -28,7 +30,6 @@ rand = "0.6.5"
 merlin = "1.0.3"
 
 [dev-dependencies]
-futures03 = { package = "futures-preview", version = "0.3.0-alpha.17", features = ["compat"] }
 keyring = { package = "substrate-keyring", path = "../../keyring" }
 substrate-executor = { path = "../../executor" }
 network = { package = "substrate-network", path = "../../network", features = ["test-helpers"]}

core/consensus/babe/primitives/Cargo.toml

@@ -12,6 +12,7 @@ runtime_primitives = { package = "sr-primitives", path = "../../../sr-primitives
 substrate-primitives = { path = "../../../primitives", default-features = false }
 slots = { package = "substrate-consensus-slots", path = "../../slots", optional = true }
 parity-codec = { version = "4.1.1", default-features = false }
+schnorrkel = { version = "0.1.1", optional = true }
 
 [features]
 default = ["std"]
@@ -20,5 +21,6 @@ std = [
 	"runtime_primitives/std",
 	"substrate-client/std",
 	"parity-codec/std",
+	"schnorrkel",
 	"slots",
 ]

core/consensus/babe/src/digest.rs → core/consensus/babe/primitives/src/digest.rs

@@ -16,66 +16,85 @@
 
 //! Private implementation details of BABE digests.
 
-use primitives::sr25519::Signature;
-use babe_primitives::{self, BABE_ENGINE_ID, SlotNumber};
+#[cfg(feature = "std")]
+use substrate_primitives::sr25519::Signature;
+#[cfg(feature = "std")]
+use super::{BABE_ENGINE_ID, Epoch};
+#[cfg(not(feature = "std"))]
+use super::{VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH};
+use super::SlotNumber;
+#[cfg(feature = "std")]
 use runtime_primitives::{DigestItem, generic::OpaqueDigestItemId};
+#[cfg(feature = "std")]
 use std::fmt::Debug;
-use parity_codec::{Decode, Encode, Codec, Input};
-use schnorrkel::{vrf::{VRFProof, VRFOutput, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH}};
-
-/// A BABE pre-digest.  It includes:
-///
-/// * The public key of the author.
-/// * The VRF proof.
-/// * The VRF output.
-/// * The slot number.
-#[derive(Clone, Debug, PartialEq, Eq)]
+use parity_codec::{Decode, Encode};
+#[cfg(feature = "std")]
+use parity_codec::{Codec, Input};
+#[cfg(feature = "std")]
+use schnorrkel::vrf::{VRFProof, VRFOutput, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH};
+
+/// A BABE pre-digest
+#[cfg(feature = "std")]
+#[derive(Clone, Debug)]
 pub struct BabePreDigest {
-	pub(super) vrf_output: VRFOutput,
-	pub(super) proof: VRFProof,
-	pub(super) index: babe_primitives::AuthorityIndex,
-	pub(super) slot_num: SlotNumber,
+	/// VRF output
+	pub vrf_output: VRFOutput,
+	/// VRF proof
+	pub vrf_proof: VRFProof,
+	/// Authority index
+	pub authority_index: super::AuthorityIndex,
+	/// Slot number
+	pub slot_number: SlotNumber,
 }
 
 /// The prefix used by BABE for its VRF keys.
 pub const BABE_VRF_PREFIX: &'static [u8] = b"substrate-babe-vrf";
 
-type RawBabePreDigest = (
-	[u8; VRF_OUTPUT_LENGTH],
-	[u8; VRF_PROOF_LENGTH],
-	u64,
-	u64,
-);
+/// A raw version of `BabePreDigest`, usable on `no_std`.
+#[derive(Copy, Clone, Encode, Decode)]
+pub struct RawBabePreDigest {
+	/// Slot number
+	pub slot_number: SlotNumber,
+	/// Authority index
+	pub authority_index: super::AuthorityIndex,
+	/// VRF output
+	pub vrf_output: [u8; VRF_OUTPUT_LENGTH],
+	/// VRF proof
+	pub vrf_proof: [u8; VRF_PROOF_LENGTH],
+}
 
+#[cfg(feature = "std")]
 impl Encode for BabePreDigest {
 	fn encode(&self) -> Vec<u8> {
-		let tmp: RawBabePreDigest = (
-			*self.vrf_output.as_bytes(),
-			self.proof.to_bytes(),
-			self.index,
-			self.slot_num,
-		);
+		let tmp =  RawBabePreDigest {
+			vrf_output: *self.vrf_output.as_bytes(),
+			vrf_proof: self.vrf_proof.to_bytes(),
+			authority_index: self.authority_index,
+			slot_number: self.slot_number,
+		};
 		parity_codec::Encode::encode(&tmp)
 	}
 }
 
+#[cfg(feature = "std")]
 impl Decode for BabePreDigest {
 	fn decode<R: Input>(i: &mut R) -> Option<Self> {
-		let (output, proof, index, slot_num): RawBabePreDigest = Decode::decode(i)?;
+		let RawBabePreDigest { vrf_output, vrf_proof, authority_index, slot_number } = Decode::decode(i)?;
 
 		// Verify (at compile time) that the sizes in babe_primitives are correct
-		let _: [u8; babe_primitives::VRF_OUTPUT_LENGTH] = output;
-		let _: [u8; babe_primitives::VRF_PROOF_LENGTH] = proof;
+		let _: [u8; super::VRF_OUTPUT_LENGTH] = vrf_output;
+		let _: [u8; super::VRF_PROOF_LENGTH] = vrf_proof;
 		Some(BabePreDigest {
-			proof: VRFProof::from_bytes(&proof).ok()?,
-			vrf_output: VRFOutput::from_bytes(&output).ok()?,
-			index,
-			slot_num,
+			vrf_proof: VRFProof::from_bytes(&vrf_proof).ok()?,
+			vrf_output: VRFOutput::from_bytes(&vrf_output).ok()?,
+			authority_index,
+			slot_number,
 		})
 	}
 }
 
 /// A digest item which is usable with BABE consensus.
+#[cfg(feature = "std")]
 pub trait CompatibleDigestItem: Sized {
 	/// Construct a digest item which contains a BABE pre-digest.
 	fn babe_pre_digest(seal: BabePreDigest) -> Self;
@@ -88,8 +107,12 @@ pub trait CompatibleDigestItem: Sized {
 
 	/// If this item is a BABE signature, return the signature.
 	fn as_babe_seal(&self) -> Option<Signature>;
+
+	/// If this item is a BABE epoch, return it.
+	fn as_babe_epoch(&self) -> Option<Epoch>;
 }
 
+#[cfg(feature = "std")]
 impl<Hash> CompatibleDigestItem for DigestItem<Hash> where
 	Hash: Debug + Send + Sync + Eq + Clone + Codec + 'static
 {
@@ -108,4 +131,8 @@ impl<Hash> CompatibleDigestItem for DigestItem<Hash> where
 	fn as_babe_seal(&self) -> Option<Signature> {
 		self.try_to(OpaqueDigestItemId::Seal(&BABE_ENGINE_ID))
 	}
+
+	fn as_babe_epoch(&self) -> Option<Epoch> {
+		self.try_to(OpaqueDigestItemId::Consensus(&BABE_ENGINE_ID))
+	}
 }

core/consensus/babe/primitives/src/lib.rs

@@ -15,15 +15,22 @@
 // along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
 
 //! Primitives for BABE.
-#![deny(warnings, unsafe_code, missing_docs)]
+#![deny(warnings)]
+#![forbid(unsafe_code, missing_docs, unused_variables, unused_imports)]
 #![cfg_attr(not(feature = "std"), no_std)]
 
+mod digest;
+
 use parity_codec::{Encode, Decode};
 use rstd::vec::Vec;
 use runtime_primitives::ConsensusEngineId;
 use substrate_primitives::sr25519::Public;
 use substrate_client::decl_runtime_apis;
 
+#[cfg(feature = "std")]
+pub use digest::{BabePreDigest, CompatibleDigestItem};
+pub use digest::{BABE_VRF_PREFIX, RawBabePreDigest};
+
 /// A Babe authority identifier. Necessarily equivalent to the schnorrkel public key used in
 /// the main Babe module. If that ever changes, then this must, too.
 pub type AuthorityId = Public;
@@ -49,15 +56,29 @@ pub type SlotNumber = u64;
 /// The weight of an authority.
 pub type Weight = u64;
 
+/// BABE epoch information
+#[derive(Decode, Encode, Default, PartialEq, Eq, Clone)]
+#[cfg_attr(any(feature = "std", test), derive(Debug))]
+pub struct Epoch {
+	/// The authorities and their weights
+	pub authorities: Vec<(AuthorityId, Weight)>,
+	/// The epoch index
+	pub epoch_index: u64,
+	/// Randomness for this epoch
+	pub randomness: [u8; VRF_OUTPUT_LENGTH],
+	/// The duration of this epoch
+	pub duration: SlotNumber,
+}
+
 /// An consensus log item for BABE.
-#[derive(Decode, Encode)]
+#[derive(Decode, Encode, Clone, PartialEq, Eq)]
 pub enum ConsensusLog {
 	/// The epoch has changed. This provides information about the
 	/// epoch _after_ next: what slot number it will start at, who are the authorities (and their weights)
 	/// and the next epoch randomness. The information for the _next_ epoch should already
 	/// be available.
 	#[codec(index = "1")]
-	NextEpochData(SlotNumber, Vec<(AuthorityId, Weight)>, [u8; VRF_OUTPUT_LENGTH]),
+	NextEpochData(Epoch),
 	/// Disable the authority with given index.
 	#[codec(index = "2")]
 	OnDisabled(AuthorityIndex),
@@ -72,6 +93,12 @@ pub struct BabeConfiguration {
 	/// Dynamic slot duration may be supported in the future.
 	pub slot_duration: u64,
 
+	/// The number of slots per BABE epoch. Currently, only
+	/// the value provided by this type at genesis will be used.
+	///
+	/// Dynamic slot duration may be supported in the future.
+	pub slots_per_epoch: u64,
+
 	/// The expected block time in milliseconds for BABE. Currently,
 	/// only the value provided by this type at genesis will be used.
 	///
@@ -116,7 +143,7 @@ decl_runtime_apis! {
 		/// Dynamic configuration may be supported in the future.
 		fn startup_data() -> BabeConfiguration;
 
-		/// Get the current authorites for Babe.
-		fn authorities() -> Vec<AuthorityId>;
+		/// Get the current epoch data for Babe.
+		fn epoch() -> Epoch;
 	}
 }