WIP: Consensus tech report

[skip ci]

ouroboros-consensus/docs/report/.gitignore

@@ -0,0 +1,7 @@
+*.aux
+*.log
+*.out
+*.pdf
+*.toc
+*.bbl
+*.blg

ouroboros-consensus/docs/report/chapters/byron.tex

@@ -0,0 +1,6 @@
+\chapter{Byron}
+
+Some details specific to the Byron ledger.
+EBBs already discussed at length in \cref{ebbs}.
+
+\section{Update proposals}

ouroboros-consensus/docs/report/chapters/conclusions.tex

@@ -0,0 +1 @@
+\chapter{Conclusions}

ouroboros-consensus/docs/report/chapters/ebbs.tex

@@ -0,0 +1,21 @@
+\chapter{Epoch Boundary Blocks}
+\label{ebbs}
+
+Discuss that although EBBs are a Byron concern, their presence has far reaching
+consequences on the consensus later. In hindsight, we should have tried harder
+to not deal with them at all from the beginning; we did not anticipate quite how
+bad the situation would be. We now have a plan for getting rid of them
+(\cref{decontamination-plan}) but it will be a fairly long term thing and it
+might not happen at all, depending on quite how much time is available for
+removing tech debt.
+
+
+\section{Introduction}
+
+\section{Consequences}
+
+\subsection{Chain selection}
+\label{ebb-chain-selection}
+
+\section{Elimination}
+\label{decontamination-plan}

ouroboros-consensus/docs/report/chapters/future.tex

@@ -0,0 +1,20 @@
+\chapter{Future work}
+
+\section{On abstraction}
+
+ledger integration: as things were changing a lot, it made sense for consensus to define the ledger API internally and have the integration be done consensus side. but as things are stabilizing, it might make more sense for that abstraction to live externally, so that you can literally plug in shelley into consensus and we don't have to do anything
+
+\section{Genesis}
+\label{future:genesis}
+
+\section{On-disk ledger state}
+
+\duncan
+
+Sketch out what we think it could look like
+Consequences for the design
+
+\section{Transaction TTL}
+\label{future:ttl}
+
+Describe that the mempool could have explicit support for TTL, but that right now we don't (and why this is ok: the ledger anyway checks tx TTL). We should discuss why this is not an attack vector (transactions will either be included in the blockchain or else will be chucked out because some of their inputs will have been used).

ouroboros-consensus/docs/report/chapters/hfc.tex

@@ -0,0 +1,28 @@
+\chapter{The Hard Fork Combinator}
+\label{hfc}
+
+\section{Adjustments}
+
+In this section we discuss some adjustments we had to make to the existing
+design of the consensus layer to make the HFC possible
+
+\subsection{Simplifying chain selection}
+\label{simplifying-chain-selection}
+
+Chain selection used to be given the full fragment; now it only gets the tip.
+
+This is non-trivial; we have a long comment explaining this for
+\lstinline!preferAnchoredCandidate!; we should move (copy?) that discussion
+here, and also discuss this comment from \lstinline!preferCandidate!:
+
+\begin{lstlisting}
+  -- NOTE: An assumption that is quite deeply ingrained in the design of the
+  -- consensus layer is that if a chain can be extended, it always should (e.g.,
+  -- see the chain database spec in @ChainDB.md@). This means that any chain
+  -- is always preferred over the empty chain, and 'preferCandidate' does not
+  -- need (indeed, cannot) be called if our current chain is empty.
+\end{lstlisting}
+
+
+\subsection{Removing the assumption that slot/time conversion is always possible}
+\label{removing-known-slot-assumption}

ouroboros-consensus/docs/report/chapters/intro.tex

@@ -0,0 +1,57 @@
+\chapter{Introduction}
+
+The IOHK Consensus and Storage layer, or \emph{the consensus layer} for short,
+is a critical piece of infrastructure in the Cardano Node. It orchestrates
+between the \emph{network layer} below it and the \emph{ledger layer} above it.
+
+The network layer is a highly concurrent piece of software that deals with
+low-level concerns; its main responsiblity is to transmit data efficiently
+across the network. Although it primarily transmits blocks and block headers, it
+does not interpret them and does not need to know much about them. In the few
+cases where it \emph{does} need to make some block-specific decisions, it
+calls back into the consensus layer to do so.
+
+The ledger layer by contrast exclusively deals with high-level concerns. It is
+entirely stateless: its main responsibility is to define a single pure
+function describing how the ledger state is transformed by blocks (verifying
+that blocks are valid in the process). It is only concerned with linear history;
+it is not aware of the presence of multiple competing chains or the roll backs
+required when switching from one chain to another.
+
+The consensus layer meanwhile mediates between these two layers. It includes a
+bespoke storage layer that provides efficient access to the current ledger state
+as well as both (recent) \emph{past} ledger states (required in order to be able
+to validate and switch to competing chains) as well as (views on near)
+\emph{future} ledger states (required to be able to validate block headers
+without access to the corresponding block bodies). The storage layer also
+provides direct access to the blocks on the blockchain itself,  so that they can
+be efficiently streamed to clients (via the network layer). When there are
+competing chains, the consensus layer decides which chain is preferable and
+should be adopted, and it decides when to \emph{contribute} to the chain
+(produce new blocks). All ``executive decisions'' about the chain are made in
+and by the consensus layer.
+
+Lastly, as well we see, the consensus layer is highly abstract and places a
+strong emphasis on compositionality, making it useable with many different
+consensus algorithms and ledgers. Importantly, compositionality enables the
+\emph{hard fork combinator} to combine multiple ledgers and regard them as a
+single block chain.
+
+The goal of this document is to outline the design goals for the consensus
+layer, how we achieved them, and where there is still scope for improvement. We
+will both describe \emph{what} the consensus layer is, and \emph{why} it is the
+way it is. Throughout will also discuss what \emph{didn't} work, approaches we
+considered but rejected, or indeed adopted but later abandoned; discussing these
+dead ends is sometimes at least as informative as discussing the solution that
+did work.
+
+We will consider some of the trade-offs we have had to make, how they
+affected the development, and discuss which of these trade-offs should perhaps
+be reconsidered. We will also take a look at how the design can scale to
+facilitate future requirements, and which requirements will be more problematic
+and require more large-scale refactoring.
+
+The target audience for this document is primarily developers working on the
+consensus layer. It may also be of more broader interest to people generally
+interested in the Cardano block chain, although we will assume that the
+reader has a technical background.