Add documentation on ChainDB registry.

ouroboros-consensus/docs/report/chapters/storage/chaindb.tex

@@ -330,3 +330,56 @@ \subsection{GC delay}
 loss, and then restart with a smaller chain}, and clock changes as well, as
 iterators asking for blocks that now live on distant chains, are not important
 use cases. We could therefore decide to remove it altogether.
+
+\subsection{Resources}
+\label{chaindb:resources}
+
+In the current implementation, the consensus layer runs with a dedicated
+\lstinline!ResourceRegistry! which will live during the whole execution of the
+consensus layer. We will refer to this registry as the \emph{consensus
+  registry}. Such registry will have a reference (or at least a transitive
+reference through some data allocated in the registry or even a child registry)
+to every resource that has a lifetime equal to the one of the consensus layer
+itself.
+
+In the case of the Chain DB, before its initialization, a new child registry
+will be allocated in the consensus registry. We will refer to this child
+registry as the \emph{ChainDB registry}. This registry will be provided to the
+initialization of the immutable database (see
+\ref{immutable:implementation:resources}), the initialization of the volatile
+database (see \ref{volatile:immplementation:resources}) and to spawn background
+threads managed by the ChainDB itself.
+
+After the Immutable and Volatile DBs are initialized, the ChainDB will spawn
+some threads that make use of resources in the ChainDB itself so they have to be
+deallocated in the right order (see \ref{storage:resourceregistry}). To ensure
+this, the ChainDB itself, once initialized, will be registered in the ChainDB
+registry as the youngest resource and therefore will be the first deallocated
+resource. The closing function will make use of the registry to release and
+deallocate resources and eventually close the registry itself which should
+consist mostly on just releasing the registry as it should be empty at that
+point. As a safeguard, if some resources are still open, closing the registry
+will release them too.
+
+As the ChainDB registry is a child registry of the consensus registry, whenever
+the consensus layer is shutting down, the ChainDB will be properly deallocated
+together with all the open resources in it.
+
+\paragraph{Followers and iterators}
+
+Followers are created outside of the scope of the ChainDB and although they make
+use of the shared ChainDB, they are managed in the relevant thread that is
+consulting the ChainDB. If an exception is raised on that thread (e.g. a
+connection with a peer is consulting the database and the peer resets the
+connection), we only want the followers associated with that connection to be
+deallocated and not the whole ChainDB. For this reason, each consuming thread will
+have its own registry which will be closed in presence of an exception in that
+thread, and this is why these resources are not allocated in the ChainDB
+registry.
+
+However, when closing the ChainDB there is a need for the Followers and
+Iterators to close, for every running consulting thread, before the database is
+closed or we will face further errors as the resources being used would then be
+closed. To ensure deallocation of the Followers and Iterators when the ChainDB
+is being shut down, the closing function of the ChainDB will first release every
+Follower and Iterator and then proceed with closing the actual database.

ouroboros-consensus/docs/report/chapters/storage/immutabledb.tex

@@ -922,3 +922,28 @@ \subsection{Recovery}
 operations. For example, while appending a block, we let the second disk write
 fail. This is another way of testing whether we can correctly recover from a
 write that was aborted half-way through.
+
+\subsection{Resources}
+\label{immutable:implementation:resources}
+
+Using the \lstinline!ResourceRegistry! introduced in
+\ref{storage:resourceregistry}, the Immutable database has to keep track of file
+handles for the chunk, primary and secondary indices (see
+\ref{immutable:implementation:chunk-layout} and
+\ref{immutable:implementation:indices}). Such handles will be allocated in the
+resource registry provided and will be closed when closing the Immutable
+database. Similarly, the current implementation spawns a child thread which is
+responsible to expire chunks that have been unused for some time. Such thread
+will also live in the resource registry provided.
+
+If a thread that uses the chunk, primary, and secondary indices would
+re-allocate them (by releasing them as resources and then allocate new
+resources), then these would be younger than things that were allocated in the
+registry after the original handles were allocated and which could potentially
+hold references to those original versions. If no care is put here, the new
+handles would be deallocated before those other entities that use them and there
+could happen that such entities attempt to access file handles which are now
+closed. This is specifically the case for threads that are spawned by the chain
+database itself as described in \ref{chaindb:resources}. To circumvent this, the
+file handles will instead be replaced with new ones retaining the same
+\emph{age} than their previously allocated counterparts.

ouroboros-consensus/docs/report/chapters/storage/overview.tex

@@ -18,14 +18,83 @@ \section{Components}
 
 Discuss the immutable/volatile split (we reference this section for that).
 
+
 \section{In memory}
 \label{storage:inmemory}
 
 TODO: After we discussed the overview, we should give an overview of everything
 we store in memory in any component, so that we have a better understanding of
 memory usage of the chain DB as a whole.
 
+\subsection{Resource registry}
+\label{storage:resourceregistry}
+
+In order to deal with resource allocation and deallocation, we use the
+abstraction of the \lstinline!ResourceRegistry!. The resource registry is a
+generalization of the \lstinline!bracket! pattern. Using a bracket imposes strict
+rules on the scope of the allocated variables, as once the body of the
+\lstinline!bracket! call finishes, the resources will be deallocated.
+
+On some situations this is too constraining and resources, despite the need to
+ensure they are deallocated, need to live for longer periods or be shared by
+multiple consumers. The registry itself lives in a \lstinline!bracket!-like
+environment, but the resources allocated in it can be opened and closed at any
+point in the lifetime of the registry and will ultimately be closed when the
+registry is being closed if they are still open at that time.
+
+The allocation function for a given resource will run with exceptions
+unconditionally masked, therefore on an uninterruptible fashion. Because of
+this, it is important that such functions are fast. Note that once the
+allocation call finishes, the resource is now managed by the registry and will
+be deallocated in case an exception is thrown.
+
+A resource that is allocated will be returned coupled with its
+\lstinline!ResourceKey! that can be used to release the resource as it holds a
+reference to the registry in which it is allocated.
+
+Special care has to be put when resources are dependent one to another. For
+example, a thread allocated in the registry might hold some mutable reference to
+a file handle that is replaced at certain points during the execution. In such
+cases, the sequence of deallocation must take this into account and deallocate
+the resources in reverse order of dependency. For this purpose, the resource
+registry offers an API to replace a specific resource without altering its
+position in the deallocation sequence.
+
+Also when deallocating the resources, we must ensure that the right order of
+deallocation is preserved. Right order here means that as resources that were
+allocated later than others could potentially use the latter, the later ones
+should probably be deallocated before the earlier ones (unless otherwise taken
+care of). Resources in the registry are indexed by their \emph{age} which is a
+meaningless backwards counter. A resource is considered older than another if
+its age is greater than the one of the other resource. Conversely, a resource is
+considered younger if the opposite holds.
+
+\subsubsection{Temporary registries}
+\label{storage:temporaryregs}
+
+When some resources are not meant to be allocated in a registry, one can take
+advantage of resource registries as a temporary container for those resources.
+For this purpose, the \lstinline!WithTempRegistry! is made available. It is
+basically a \lstinline!StateT! computation which will check that the resource
+was indeed transferred properly to the returned value and then the registry will
+vanish.
+
+Using this construction will ensure that if an exception is thrown while the
+resources are being allocated but before they are kept track by the resulting
+state, they will be released as the registry itself will close in presence of an
+exception.
+
+However, if these allocations are done in the scope of an ``upper'' registry, in
+which the resource is meant to be tracked, they will indeed be tracked by the
+upper registry which will deallocate them in presence of an exception, so using
+a temporary registry is not required.
+
+Note that once the resource was transferred to the final state, no more tracking
+is performed and the resource could be leaked. It is then responsibility of the
+resulting state to eventually deallocate the resource.
+
 \subsection{Chain fragments}
+
 \label{storage:fragments}
 
 \subsection{Extended ledger state}

ouroboros-consensus/docs/report/chapters/storage/volatiledb.tex

@@ -398,3 +398,20 @@ \subsection{Recovery}
 before it. As mentioned at the start of this chapter, it is not crucial to
 recover every single block. Therefore, we do not try to deserialise the blocks
 after a corrupt one.
+
+\subsection{Resources}
+\label{volatile:implementation:resources}
+
+Using the \lstinline!ResourceRegistry! introduced in
+\ref{storage:resourceregistry}, the Volatile DB has to keep track of a file
+handle for the current open file. Such a handle will be allocated in the
+resource registry provided and will be closed when closing the Immutable
+database.
+
+Note that to preserve the order in the sequence of deallocations, the file
+handle cannot be deallocated and re-allocated on demand as this will make it be
+deallocated earlier than resources that hold a reference to it. Specifically
+threads that are spawned by the chain database itself as described in
+\ref{chaindb:resources}. To circumvent this, the file handle will instead be
+replaced with a new one retaining the same \emph{age} than its previously
+allocated counterpart.