Merge #3408 #3414

3408: P2P review of merged PRs r=coot a=coot

* #3369
* #3370

- snocket: move bang patterns to the top of the recursive call
- snocket: fixed some typos
- snocket: remote path for local sockets
- handshake: renamed MsgProposeVersions' as MsgReplyVersions
- handshake: test that acceptOrRefuse is symmetric
- handshake: fixed some typos
- handshake: implement more properties
- handshake: updated documentation


3414: [CAD-3383] Show progress of --store-ledger every 1000 blocks r=EncodePanda a=EncodePanda



Co-authored-by: Marcin Szamotulski <profunctor@pm.me>
Co-authored-by: EncodePanda <paul.szulc@gmail.com>

docs/network-spec/miniprotocols.tex

@@ -233,7 +233,7 @@ \section{Overview of all implemented Mini Protocols}
     {https://input-output-hk.github.io/ouroboros-network/ouroboros-network/Ouroboros-Network-Protocol-LocalStateQuery-Type.html\#t:LocalStateQuery}
 
 \section{CBOR and CDDL}
-All mini-protocols are encoded using coincise binary object representation
+All mini-protocols are encoded using concise binary object representation
 (CBOR), see~\url{https://cbor.io}.  Each codec comes along with a specification
 written in CDDL,
 see~\url{https://cbor-wg.github.io/cddl/draft-ietf-cbor-cddl.html}.
@@ -359,7 +359,7 @@ \section{Handshake mini-protocol}
 \newcommand{\StPropose}{\state{StPropose}}
 \newcommand{\StConfirm}{\state{StConfirm}}
 \newcommand{\MsgProposeVersions}{\msg{MsgProposeVersions}}
-\newcommand{\MsgProposeVersionsX}{\msg{MsgProposeVersions'}}
+\newcommand{\MsgReplyVersions}{\msg{MsgReplyVersion}}
 \newcommand{\MsgAcceptVersion}{\msg{MsgAcceptVersion}}
 \newcommand{\MsgRefuse}{\msg{MsgRefuse}}
 
@@ -381,7 +381,7 @@ \subsection{Description}
 supported protocol versions and callback functions for handling the protocol parameters.
 These callback functions are specific for the supported protocol versions.
 
-The handshake mini protocol is designed to handle simultantous TCP open.
+The handshake mini protocol is designed to handle simultaneous TCP open.
 
 \subsection{State machine}
 
@@ -402,21 +402,21 @@ \subsection{State machine}
     \node[state, right of=Confirm] (StDone)    at (7,  0) {\StDone};
 
     \draw (StPropose) --node[above=1em]{\MsgProposeVersions} (StConfirm);
-    \draw[->] (StConfirm.10)  to [out=15,  in=150] node[above]{\MsgAcceptVersion}     (StDone.170);
-    \draw[->] (StConfirm.350) to [out=-15, in=210] node[below]{\MsgRefuse}            (StDone.190);
-    \draw[->] (StConfirm)     -- node[fill=white,above=-0.8em] {\MsgProposeVersionsX} (StDone.west);
+    \draw[->] (StConfirm.10)  to [out=15,  in=150] node[above]{\MsgAcceptVersion}   (StDone.170);
+    \draw[->] (StConfirm.350) to [out=-15, in=210] node[below]{\MsgRefuse}          (StDone.190);
+    \draw[->] (StConfirm)     -- node[fill=white,above=-0.8em] {\MsgReplayVersions} (StDone.west);
   \end{tikzpicture}
 \end{figure}
 
 Messages of the protocol:
 \begin{description}
 \item [\MsgProposeVersions{} {\boldmath $(versionTable)$}]
       The client proposes a number of possible versions and protocol parameters.
-\item [\MsgProposeVersionsX{} {\boldmath $(versionTable)$}]
-      In TCP simultanous open the client will receive \MsgProposeVersionsX{}
+\item [\MsgReplayVersions{} {\boldmath $(versionTable)$}]
+      In TCP simultaneous open the client will receive \MsgReplayVersions{}
       (which was sent as \MsgProposeVersions{}) as a reply to its own
       \MsgProposeVersions{}; thus both \MsgProposeVersions{} and
-      \MsgProposeVersionsX{} have to have the same CBOR encoding.
+      \MsgReplayVersions{} have to have the same CBOR encoding.
 \item [\MsgAcceptVersion{} {\boldmath $(versionNumber,extraParameters)$}]
       The server accepts $versionNumber$ and returns possible extra protocol parameters.
 \item [\MsgRefuse{} {\boldmath $(reason)$}]
@@ -429,7 +429,7 @@ \subsection{State machine}
   \multicolumn{4}{|c|}{Transition table} \\ \hline
     from       & message/event         & parameters                        & to \\ \hline\hline
     \StPropose & \MsgProposeVersions   & $versionTable$                    & \StConfirm \\ \hline
-    \StConfirm & \MsgProposeVersionsX  & $versionTable$                    & \StDone \\ \hline
+    \StConfirm & \MsgReplayVersions    & $versionTable$                    & \StDone \\ \hline
     \StConfirm & \MsgAcceptVersion     & $(versionNumber,extraParameters)$ & \StDone \\ \hline
     \StConfirm & \MsgRefuse            & $reason$                          & \StDone \\ \hline
   \end{tabular}
@@ -474,7 +474,7 @@ \subsection{Client and Server Implementation}
   Otherwise: Test the proposed protocol parameters of the selected protocol version
 \item
   If the test refuses the parameters:
-  Reply with \MsgRefuse(\Refused), the selected version number and an error message.
+    Reply with \MsgRefuse(\Refused), the selected version number and an error message.
 \item
   Otherwise:
   Encode the extra parameters and
@@ -487,12 +487,63 @@ \subsection{Client and Server Implementation}
 The server simply ignores the CBOR terms that represent the protocol parameters of unsupported
 version.
 
-In case of simultaneous open TCP connection, both handshake clients will send
-their \MsgProposeVersions{}, both will interpet the incoming message as
-\MsgProposeVersionsX{} (thus both must have the same encoding, the
-implementation can disntiguish them by the protocol state).  Both clients
-should choose the highest version of the protocol available.  If any side does
-not accept any version (or its parameters) it can reset the connection.
+In case of simultaneous open of a TCP connection, both handshake clients will
+send their \MsgProposeVersions{}, both will interpret the incoming message as
+\MsgReplayVersions{} (thus both must have the same encoding, the implementation
+can distinguish them by the protocol state).  Both clients should choose the
+highest version of the protocol available.  If any side does not accept any
+version (or its parameters) it can reset the connection.
+
+The protocol does not forbid, nor could it detect a usage of
+\MsgReplyVersions{} outside of TCP simultaneous open.  The process of
+choosing between proposed and received version must be symmetric, in the
+following sense.
+
+\begin{description}
+    We use \texttt{acceptable :: vData -> vData -> Accept vData}
+    function to compute accepted version data from proposed and received data,
+    where
+\begin{code}
+\begin{verbatim}
+  data Accept vData = Accept vData
+                    | Refuse Text
+                    deriving Eq
+\end{verbatim}
+\end{code}
+    See
+    \href{https://input-output-hk.github.io/ouroboros-network/ouroboros-network-framework/Ouroboros-Network-Protocol-Handshake-Version.html#t:Acceptable}{ref}.
+    Both \texttt{acceptable proposed received} and \texttt{acceptable received
+    proposed} must satisfy the following condition:
+    \begin{itemize}
+      \item if either of them accepts a version by returning \texttt{Accept},
+        the other one must accept the same value, i.e. in this case
+        \texttt{acceptable proposed received == acceptable received proposed}
+      \item if either of them refuses to accept (returns \textt{Refuse reason})
+        the other one shall return \texttt{Refuse} as well.
+    \end{itemize}
+\end{description}
+Note that the above condition guarantees that if either side returns
+\texttt{Accept} then the connection will not be closed by the remote end.
+A weaker condition, in which the returns values are equal if they both return
+\texttt{Accept}, does not guarantee this property.  We also verify that the
+whole Handshake protocol, not just the \texttt{acceptable} satisfies the above
+property, see
+\href{https://github.com/input-output-hk/ouroboros-network/blob/master/ouroboros-network/protocol-tests/Ouroboros/Network/Protocol/Handshake/Test.hs}{Ouroboros-Network
+test suite}.
+
+The fact that we are using non-injective encoding in the handshake protocol
+side steps typed-protocols strong typed-checked properties.  For injective
+codecs (i.e. codecs for which each message has a distinguished encoding), both
+sides of typed-protocols are always at the same state (once all in-flight
+message arrived).  This is no longer true in general, however this is still
+true for the handshake protocol.  Event though the opening message
+\texttt{MsgProposeVersions} of a simultaneous open, will materialise on the
+other side as termination message \texttt{MsgReplyVersions}, the same will
+happen to the \texttt{MsgProposeVersion} transmitted in the other direction.
+We include a special test case
+(\href{https://github.com/input-output-hk/ouroboros-network/blob/master/ouroboros-network/protocol-tests/Ouroboros/Network/Protocol/Handshake/Test.hs\#L551}{\texttt{prop\_channel\_simultaneous\_open}})
+to verify that simultaneous open well behaves and does not lead to protocol
+errors.
 
 The handshake mini protocol runs before the MUX/DEMUX itself is initialised.
 Each message is transmitted within a single MUX segment, i.e. with a proper
@@ -502,7 +553,7 @@ \subsection{Client and Server Implementation}
 
 \subsection{CDDL encoding specification}\label{handshake-cddl}
 There are two flavours of the mini-protocol which only differ with type
-instatiations, e.g. different protocol versions and version data carried in
+instantiations, e.g. different protocol versions and version data carried in
 messages.  First one is used by the node to node protocol the other by node to
 client protocol.
 

ouroboros-consensus-cardano/tools/db-analyser/Analysis.hs

@@ -215,17 +215,20 @@ storeLedgerStateAt slotNo (AnalysisEnv { db, registry, initLedger, cfg, limit, l
           newLedger     = either (error . show) lrResult $ runExcept $ appliedResult
       when (blockSlot blk >= slotNo) $ storeLedgerState blk newLedger
       when (blockSlot blk > slotNo) $ issueWarning blk
+      when ((unBlockNo $ blockNo blk) `mod` 1000 == 0) $ reportProgress blk
       return (continue blk, newLedger)
 
     continue :: blk -> NextStep
     continue blk
       | blockSlot blk >= slotNo = Stop
       | otherwise               = Continue
 
-    issueWarning blk = putStrLn $ "Snapshot was created at " <>
-                         show (blockSlot blk) <> " " <>
-                         "because there was no block forged at requested " <>
-                         show slotNo <> ". "
+    issueWarning blk   = putStrLn $ "Snapshot was created at " <>
+                           show (blockSlot blk) <> " " <>
+                           "because there was no block forged at requested " <>
+                           show slotNo <> ". "
+    reportProgress blk = putStrLn $ "... reached slot " <>
+                           show (blockSlot blk)
 
     storeLedgerState ::
          blk

ouroboros-network-framework/src/Ouroboros/Network/Protocol/Handshake/Client.hs

@@ -44,8 +44,8 @@ handshakeClientPeer codec@VersionDataCodec {encodeData, decodeData}
   Yield (ClientAgency TokPropose) (MsgProposeVersions $ encodeVersions encodeData versions) $
 
     Await (ServerAgency TokConfirm) $ \msg -> case msg of
-      MsgProposeVersions' vMap -> 
-        -- simultanous open; 'accept' will choose version (the greatest common
+      MsgReplyVersions vMap -> 
+        -- simultaneous open; 'accept' will choose version (the greatest common
         -- version), and check if we can accept received version data.
         Done TokDone $ case acceptOrRefuse codec acceptVersion versions vMap of
           Right r -> Right r
@@ -96,7 +96,7 @@ acceptOrRefuse
   -> Versions vNumber vData r
   -> Map vNumber vParams
   -- ^ proposed versions received either with `MsgProposeVersions` or
-  -- `MsgProposeVersions'`
+  -- `MsgReplyVersions`
   -> Either (RefuseReason vNumber) (r, vNumber, vData)
 acceptOrRefuse VersionDataCodec {decodeData}
                acceptVersion versions versionMap =

ouroboros-network-framework/src/Ouroboros/Network/Protocol/Handshake/Codec.hs

@@ -50,7 +50,7 @@ import           Ouroboros.Network.Protocol.Handshake.Type
 -- protocol.
 --
 -- Note: 'extra' type param is instantiated to 'DictVersion'; 'agreedOptions'
--- is instatiated to 'NodeToNodeVersionData' in "Ouroboros.Network.NodeToNode"
+-- is instantiated to 'NodeToNodeVersionData' in "Ouroboros.Network.NodeToNode"
 -- or to '()' in "Ouroboros.Network.NodeToClient".
 --
 data VersionDataCodec bytes vNumber vData = VersionDataCodec {
@@ -135,10 +135,10 @@ codecHandshake versionNumberCodec = mkCodecCborLazyBS encodeMsg decodeMsg
         <> CBOR.encodeWord 0
         <> encodeVersions versionNumberCodec vs
 
-      -- Although `MsgProposeVersions'` shall not be sent, for testing purposes
-      -- it is useful to have an encoder for it.
-      encodeMsg (ServerAgency TokConfirm) (MsgProposeVersions' vs) =
-           CBOR.encodeListLen 2
+      -- Although `MsgReplyVersions` shall not be sent, for testing purposes it
+      -- is useful to have an encoder for it.
+      encodeMsg (ServerAgency TokConfirm) (MsgReplyVersions vs)
+        = CBOR.encodeListLen 2
         <> CBOR.encodeWord 0
         <> encodeVersions versionNumberCodec vs
 
@@ -167,7 +167,7 @@ codecHandshake versionNumberCodec = mkCodecCborLazyBS encodeMsg decodeMsg
           (ServerAgency TokConfirm, 0, 2) -> do
             l  <- CBOR.decodeMapLen
             vMap <- decodeVersions versionNumberCodec l
-            pure $ SomeMessage $ MsgProposeVersions' vMap
+            pure $ SomeMessage $ MsgReplyVersions vMap
           (ServerAgency TokConfirm, 1, 3) -> do
             v <- decodeTerm versionNumberCodec <$> CBOR.decodeTerm
             case v of

ouroboros-network-framework/src/Ouroboros/Network/Protocol/Handshake/Type.hs

@@ -83,12 +83,12 @@ instance Protocol (Handshake vNumber vParams) where
         -> Message (Handshake vNumber vParams) StPropose StConfirm
 
       -- |
-      -- `MsgProposeVersions'` received as a response to 'MsgProposeVersions'.
-      -- It is not supported to explicitly send this message. It can only be
-      -- received as a copy of 'MsgProposeVersions' in a simultanous open
+      -- `MsgReplyVersions` received as a response to 'MsgProposeVersions'.  It
+      -- is not supported to explicitly send this message. It can only be
+      -- received as a copy of 'MsgProposeVersions' in a simultaneous open
       -- scenario.
       --
-      MsgProposeVersions'
+      MsgReplyVersions
         :: Map vNumber vParams
         -> Message (Handshake vNumber vParams) StConfirm StDone
 
@@ -123,13 +123,13 @@ instance Protocol (Handshake vNumber vParams) where
 
 -- $simultanous-open
 --
--- On simultanous open both sides will send `MsgProposeVersions`, which will be
--- decoded as `MsgProposeVersions'` which is a terminal message of the
--- protocol.  It is important to stress that in this case both sides will make
--- the choice which version and parameters to pick.  Our algorithm for picking
--- version is symmetric, which ensures that both sides will endup with the same
--- choice.  If one side decides to refuse the version it will close the
--- connection, without sending the reason to the other side.
+-- On simultaneous open both sides will send `MsgProposeVersions`, which will be
+-- decoded as `MsgReplyVersions`.  It is a terminal message of the protocol.  It
+-- is important to stress that in this case both sides will make the choice
+-- which version and parameters to pick.  Our algorithm for picking version is
+-- symmetric, which ensures that both sides will end up with the same choice.
+-- If one side decides to refuse the version it will close the connection,
+-- without sending the reason to the other side.
 
 deriving instance (Show vNumber, Show vParams)
     => Show (Message (Handshake vNumber vParams) from to)

ouroboros-network-framework/src/Ouroboros/Network/Protocol/Handshake/Version.hs

@@ -97,7 +97,7 @@ data VersionMismatch vNum where
   InconsistentVersion :: vNum -> VersionMismatch vNum
 
 --
--- Simple version negotation
+-- Simple version negotiation
 --
 
 -- | Singleton smart constructor for 'Versions'.