Various Guide Improvements

roadmap/implementors-guide/src/SUMMARY.md

@@ -36,7 +36,14 @@
     - [Network Bridge](node/utility/network-bridge.md)
     - [Misbehavior Arbitration](node/utility/misbehavior-arbitration.md)
     - [Peer Set Manager](node/utility/peer-set-manager.md)
-- [Data Structures and Types](type-definitions.md)
+	- [Runtime API Requests](node/utility/runtime-api.md)
+- [Data Structures and Types](types/README.md)
+    - [Candidate](types/candidate.md)
+    - [Backing](types/backing.md)
+    - [Availability](types/availability.md)
+    - [Overseer and Subsystem Protocol](types/overseer-protocol.md)
+	- [Runtime](types/runtime.md)
+	- [Chain](types/chain.md)
 
 [Glossary](glossary.md)
 [Further Reading](further-reading.md)

roadmap/implementors-guide/src/node/availability/bitfield-distribution.md

@@ -6,11 +6,7 @@ Validators vote on the availability of a backed candidate by issuing signed bitf
 
 `ProtocolId`: `b"bitd"`
 
-Input:
-
-- `DistributeBitfield(relay_parent, SignedAvailabilityBitfield)`: distribute a bitfield via gossip to other validators.
-- `NetworkBridgeUpdate(NetworkBridgeUpdate)`
-
+Input: [`BitfieldDistributionMessage`](../../overseer-protocol.md#bitfield-distribution-message)
 Output:
 
 - `NetworkBridge::RegisterEventProducer(ProtocolId)`

roadmap/implementors-guide/src/node/backing/candidate-backing.md

@@ -2,26 +2,26 @@
 
 The Candidate Backing subsystem ensures every parablock considered for relay block inclusion has been seconded by at least one validator, and approved by a quorum. Parablocks for which no validator will assert correctness are discarded. If the block later proves invalid, the initial backers are slashable; this gives polkadot a rational threat model during subsequent stages.
 
-Its role is to produce backable candidates for inclusion in new relay-chain blocks. It does so by issuing signed [`Statement`s](../../type-definitions.html#statement-type) and tracking received statements signed by other validators. Once enough statements are received, they can be combined into backing for specific candidates.
+Its role is to produce backable candidates for inclusion in new relay-chain blocks. It does so by issuing signed [`Statement`s](../../types/backing.html#statement-type) and tracking received statements signed by other validators. Once enough statements are received, they can be combined into backing for specific candidates.
 
 Note that though the candidate backing subsystem attempts to produce as many backable candidates as possible, it does _not_ attempt to choose a single authoritative one. The choice of which actually gets included is ultimately up to the block author, by whatever metrics it may use; those are opaque to this subsystem.
 
 Once a sufficient quorum has agreed that a candidate is valid, this subsystem notifies the [Provisioner](../utility/provisioner.html), which in turn engages block production mechanisms to include the parablock.
 
 ## Protocol
 
-The [Candidate Selection subsystem](candidate-selection.html) is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingMessage`s](../../type-definitions.html#candidate-backing-message), and passes them to this subsystem.
+The [Candidate Selection subsystem](candidate-selection.html) is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingMessage`s](../../types/overseer-protocol.html#candidate-backing-message), and passes them to this subsystem.
 
-This subsystem validates the candidates and generates an appropriate [`Statement`](../../type-definitions.html#statement-type). All `Statement`s are then passed on to the [Statement Distribution subsystem](statement-distribution.html) to be gossiped to peers. When this subsystem decides that a candidate is invalid, and it was recommended to us to second by our own Candidate Selection subsystem, a message is sent to the Candidate Selection subsystem with the candidate's hash so that the collator which recommended it can be penalized.
+This subsystem validates the candidates and generates an appropriate [`Statement`](../../types/backing.html#statement-type). All `Statement`s are then passed on to the [Statement Distribution subsystem](statement-distribution.html) to be gossiped to peers. When this subsystem decides that a candidate is invalid, and it was recommended to us to second by our own Candidate Selection subsystem, a message is sent to the Candidate Selection subsystem with the candidate's hash so that the collator which recommended it can be penalized.
 
 ## Functionality
 
 The subsystem should maintain a set of handles to Candidate Backing Jobs that are currently live, as well as the relay-parent to which they correspond.
 
 ### On Overseer Signal
 
-* If the signal is an [`OverseerSignal`](../../type-definitions.html#overseer-signal)`::StartWork(relay_parent)`, spawn a Candidate Backing Job with the given relay parent, storing a bidirectional channel with the Candidate Backing Job in the set of handles.
-* If the signal is an [`OverseerSignal`](../../type-definitions.html#overseer-signal)`::StopWork(relay_parent)`, cease the Candidate Backing Job under that relay parent, if any.
+* If the signal is an [`OverseerSignal`](../../types/overseer-protocol.html#overseer-signal)`::StartWork(relay_parent)`, spawn a Candidate Backing Job with the given relay parent, storing a bidirectional channel with the Candidate Backing Job in the set of handles.
+* If the signal is an [`OverseerSignal`](../../types/overseer-protocol.html#overseer-signal)`::StopWork(relay_parent)`, cease the Candidate Backing Job under that relay parent, if any.
 
 ### On `CandidateBackingMessage`
 
@@ -39,7 +39,7 @@ The subsystem should maintain a set of handles to Candidate Backing Jobs that ar
 
 The Candidate Backing Job represents the work a node does for backing candidates with respect to a particular relay-parent.
 
-The goal of a Candidate Backing Job is to produce as many backable candidates as possible. This is done via signed [`Statement`s](../../type-definitions.html#statement-type) by validators. If a candidate receives a majority of supporting Statements from the Parachain Validators currently assigned, then that candidate is considered backable.
+The goal of a Candidate Backing Job is to produce as many backable candidates as possible. This is done via signed [`Statement`s](../../types/backing.html#statement-type) by validators. If a candidate receives a majority of supporting Statements from the Parachain Validators currently assigned, then that candidate is considered backable.
 
 ### On Startup
 

roadmap/implementors-guide/src/node/backing/candidate-selection.md

@@ -6,18 +6,18 @@ This subsystem includes networking code for communicating with collators, and tr
 
 This subsystem is only ever interested in parablocks assigned to the particular parachain which this validator is currently handling.
 
-New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an appropriate message to the [Candidate Backing subsystem](candidate-backing.html) to generate an appropriate [`Statement`](../../type-definitions.html#statement-type).
+New parablock candidates may arrive from a potentially unbounded set of collators. This subsystem chooses either 0 or 1 of them per relay parent to second. If it chooses to second a candidate, it sends an appropriate message to the [Candidate Backing subsystem](candidate-backing.html) to generate an appropriate [`Statement`](../../types/backing.html#statement-type).
 
 In the event that a parablock candidate proves invalid, this subsystem will receive a message back from the Candidate Backing subsystem indicating so. If that parablock candidate originated from a collator, this subsystem will blacklist that collator. If that parablock candidate originated from a peer, this subsystem generates a report for the [Misbehavior Arbitration subsystem](../utility/misbehavior-arbitration.html).
 
 ## Protocol
 
-Input: None
+Input: [`CandidateSelectionMessage`](../../types/overseer-protocol#candidate-selection-message)
 
 Output:
 
 - Validation requests to Validation subsystem
-- [`CandidateBackingMessage`](../../type-definitions.html#candidate-backing-message)`::Second`
+- [`CandidateBackingMessage`](../../types/overseer-protocol.html#candidate-backing-message)`::Second`
 - Peer set manager: report peers (collators who have misbehaved)
 
 ## Functionality

roadmap/implementors-guide/src/node/backing/statement-distribution.md

@@ -35,7 +35,7 @@ The Statement Distribution subsystem sends statements to peer nodes and detects
 
 ## Peer Receipt State Machine
 
-There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any [`SignedStatement`](../../type-definitions.html#signed-statement-type), validate that the provided signature does in fact sign the included data. Note that each individual parablock candidate gets its own instance of this state machine; it is perfectly legal to receive a `Valid(X)` before a `Seconded(Y)`, as long as a `Seconded(X)` has been received.
+There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any [`SignedStatement`](../../types/backing.html#signed-statement-type), validate that the provided signature does in fact sign the included data. Note that each individual parablock candidate gets its own instance of this state machine; it is perfectly legal to receive a `Valid(X)` before a `Seconded(Y)`, as long as a `Seconded(X)` has been received.
 
 A: Initial State. Receive `SignedStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedStatement` variant: drop it.
 B: Receive any `SignedStatement`: extract `Statement`, forward to Candidate Backing. Receive `OverseerMessage::StopWork`: proceed to C.

roadmap/implementors-guide/src/node/overseer.md

@@ -24,7 +24,7 @@ The hierarchy of subsystems:
 
 ```
 
-The overseer determines work to do based on block import events and block finalization events. It does this by keeping track of the set of relay-parents for which work is currently being done. This is known as the "active leaves" set. It determines an initial set of active leaves on startup based on the data on-disk, and uses events about blockchain import to update the active leaves. Updates lead to [`OverseerSignal`](../type-definitions.html#overseer-signal)`::StartWork` and [`OverseerSignal`](../type-definitions.html#overseer-signal)`::StopWork` being sent according to new relay-parents, as well as relay-parents to stop considering. Block import events inform the overseer of leaves that no longer need to be built on, now that they have children, and inform us to begin building on those children. Block finalization events inform us when we can stop focusing on blocks that appear to have been orphaned.
+The overseer determines work to do based on block import events and block finalization events. It does this by keeping track of the set of relay-parents for which work is currently being done. This is known as the "active leaves" set. It determines an initial set of active leaves on startup based on the data on-disk, and uses events about blockchain import to update the active leaves. Updates lead to [`OverseerSignal`](../types.overseer-protocol.html#overseer-signal)`::StartWork` and [`OverseerSignal`](../types/overseer-protocol.html#overseer-signal)`::StopWork` being sent according to new relay-parents, as well as relay-parents to stop considering. Block import events inform the overseer of leaves that no longer need to be built on, now that they have children, and inform us to begin building on those children. Block finalization events inform us when we can stop focusing on blocks that appear to have been orphaned.
 
 The overseer's logic can be described with these functions:
 

roadmap/implementors-guide/src/node/utility/availability-store.md

@@ -25,11 +25,7 @@ There may be multiple competing blocks all ending the availability phase for a p
 
 ## Protocol
 
-Input:
-
-- QueryPoV(candidate_hash, response_channel)
-- QueryChunk(candidate_hash, validator_index, response_channel)
-- StoreChunk(candidate_hash, validator_index, inclusion_proof, chunk_data)
+Input: [`AvailabilityStoreMessage`](../../types/overseer-protocol.html#availability-store-message)
 
 ## Functionality
 
@@ -49,7 +45,7 @@ On `QueryPoV` message:
 
 On `QueryChunk` message:
 
-- Determine if we have the chunk indicated by the parameters and return it via the response channel if so.
+- Determine if we have the chunk indicated by the parameters and return it and its inclusion proof via the response channel if so.
 
 On `StoreChunk` message:
 

roadmap/implementors-guide/src/node/utility/candidate-validation.md

@@ -6,7 +6,7 @@ This subsystem is responsible for handling candidate validation requests. It is
 
 Input:
 
-- [`CandidateValidationMessage`](../../type-definitions.html#validation-request-type)
+- [`CandidateValidationMessage`](../../types/overseer-protocol.html#validation-request-type)
 
 ## Functionality
 

roadmap/implementors-guide/src/node/utility/network-bridge.md

@@ -10,27 +10,8 @@ So in short, this Subsystem acts as a bridge between an actual network component
 
 ## Protocol
 
-> REVIEW: I am designing this using dynamic dispatch based on a ProtocolId discriminant rather than doing static dispatch to specific subsystems based on a concrete network message type. The reason for this is that doing static dispatch might break the property that Subsystem implementations can be swapped out for others. So this is actually implementing a subprotocol multiplexer. Pierre tells me this is OK for our use-case ;). One caveat is that now all network traffic will also flow through the overseer, but this overhead is probably OK.
-
-```rust
-use sc-network::ObservedRole;
-
-struct View(Vec<Hash>); // Up to `N` (5?) chain heads.
-
-enum NetworkBridgeEvent {
- PeerConnected(PeerId, ObservedRole), // role is one of Full, Light, OurGuardedAuthority, OurSentry
- PeerDisconnected(PeerId),
- PeerMessage(PeerId, Bytes),
- PeerViewChange(PeerId, View), // guaranteed to come after peer connected event.
- OurViewChange(View),
-}
-```
-
-Input:
-
-- RegisterEventProducer(`ProtocolId`, `Fn(NetworkBridgeEvent) -> AllMessages`): call on startup.
-- ReportPeer(PeerId, cost_or_benefit)
-- SendMessage(`[PeerId]`, `ProtocolId`, Bytes): send a message to multiple peers.
+Input: [`NetworkBridgeMessage`](../../types/overseer-protocol.html#network-bridge-message)
+Output: Varying, based on registered event producers.
 
 ## Functionality
 

roadmap/implementors-guide/src/node/utility/provisioner.md

@@ -10,11 +10,11 @@ There are several distinct types of provisionable data, but they share this prop
 
 ### Backed Candidates
 
-The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backed candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's; the provisioner must ensure that block authors are aware of all available [`BackedCandidate`s](../../type-definitions.html#backed-candidate).
+The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backed candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's; the provisioner must ensure that block authors are aware of all available [`BackedCandidate`s](../../types/backing.html#backed-candidate).
 
 ### Signed Bitfields
 
-[Signed bitfields](../../type-definitions.html#signed-availability-bitfield) are attestations from a particular validator about which candidates it believes are available.
+[Signed bitfields](../../types/availability.html#signed-availability-bitfield) are attestations from a particular validator about which candidates it believes are available.
 
 ### Misbehavior Reports
 
@@ -32,7 +32,7 @@ Dispute resolution is complex and is explained in substantially more detail [her
 
 ## Protocol
 
-Input: [`ProvisionerMessage`](../../type-definitions.html#provisioner-message). Backed candidates come from the [Candidate Backing subsystem](../backing/candidate-backing.html), signed bitfields come from the [Bitfield Distribution subsystem](../availability/bitfield-distribution.html), and misbehavior reports and disputes come from the [Misbehavior Arbitration subsystem](misbehavior-arbitration.html).
+Input: [`ProvisionerMessage`](../../types/overseer-protocol.html#provisioner-message). Backed candidates come from the [Candidate Backing subsystem](../backing/candidate-backing.html), signed bitfields come from the [Bitfield Distribution subsystem](../availability/bitfield-distribution.html), and misbehavior reports and disputes come from the [Misbehavior Arbitration subsystem](misbehavior-arbitration.html).
 
 At initialization, this subsystem has no outputs. Block authors can send a `ProvisionerMessage::RequestBlockAuthorshipData`, which includes a channel over which provisionable data can be sent. All appropriate provisionable data will then be sent over this channel, as it is received.
 

roadmap/implementors-guide/src/node/utility/runtime-api.md

@@ -0,0 +1,19 @@
+# Runtime API
+
+The Runtime API subsystem is responsible for providing a single point of access to runtime state data via a set of pre-determined queries. This prevents shared ownership of a blockchain client resource by providing
+
+## Protocol
+
+Input: [`RuntimeApiMessage`](../../types/overseer-protocol.html#runtime-api-message)
+
+Output: None
+
+## Functionality
+
+On receipt of `RuntimeApiMessage::Request(relay_parent, request)`, answer the request using the post-state of the relay_parent provided and provide the response to the side-channel embedded within the request.
+
+> TODO Do some caching. The underlying rocksdb already has a cache of trie nodes so duplicate requests are unlikely to hit disk. Not required for functionality.
+
+## Jobs
+
+> TODO Don't limit requests based on parent hash, but limit caching. No caching should be done for any requests on relay_parents that are not live based on `StartWork` or `StopWork` messages. Maybe with some leeway for things that have just been stopped.

roadmap/implementors-guide/src/runtime/configuration.md

@@ -2,7 +2,7 @@
 
 This module is responsible for managing all configuration of the parachain host in-flight. It provides a central point for configuration updates to prevent races between configuration changes and parachain-processing logic. Configuration can only change during the session change routine, and as this module handles the session change notification first it provides an invariant that the configuration does not change throughout the entire session. Both the [scheduler](scheduler.html) and [inclusion](inclusion.html) modules rely on this invariant to ensure proper behavior of the scheduler.
 
-The configuration that we will be tracking is the [`HostConfiguration`](../type-definitions.html#host-configuration) struct.
+The configuration that we will be tracking is the [`HostConfiguration`](../types/runtime.html#host-configuration) struct.
 
 ## Storage
 

roadmap/implementors-guide/src/runtime/inclusioninherent.md

@@ -16,7 +16,7 @@ Included: Option<()>,
 
 ## Entry Points
 
-* `inclusion`: This entry-point accepts two parameters: [`Bitfields`](../type-definitions.html#signed-availability-bitfield) and [`BackedCandidates`](../type-definitions.html#backed-candidate).
+* `inclusion`: This entry-point accepts two parameters: [`Bitfields`](../types/availability.html#signed-availability-bitfield) and [`BackedCandidates`](../types/backing.html#backed-candidate).
     1. The `Bitfields` are first forwarded to the `process_bitfields` routine, returning a set of freed cores. Provide a `Scheduler::core_para` as a core-lookup to the `process_bitfields` routine. Annotate each of these freed cores with `FreedReason::Concluded`.
     1. If `Scheduler::availability_timeout_predicate` is `Some`, invoke `Inclusion::collect_pending` using it, and add timed-out cores to the free cores, annotated with `FreedReason::TimedOut`.
     1. Invoke `Scheduler::schedule(freed)`