UTxO-HD release

[jasagredo]

This issue is an aggregate of dangling issues here and there about UTxO-HD, to better reflect the current status of the feature, with updated information.

Current status

The UTxO-HD feature is based currently on the 9.1 cardano-node release.

Repository	Branch
ouroboros-consensus	link
cardano-api	link
cardano-cli	link
cardano-node	link

• Development work is done. Feature is complete.
• Testing passed on top of the 8.9 node release.
• Benchmarking is taking place on the benchmarking cluster. Should there be regressions, new issues will be opened.
  • Reported CPU > 85% slots is drastically increased. Seems like this is an artifact of the analysis. The performance team will look into this.
  • UTxO-HD heap increase in latest benchmarks ouroboros-consensus#1192
  • Nodes stuck in forging loop ouroboros-consensus#1193
• Testing will be conducted on this version. (link)
• Documentation is lacking Document feature UTxO-HD v0.1 ouroboros-consensus#126
• Git history is messy. The journey has been long with numerous rebases with hundreds of conflicts each.

High level description of UTxO-HD and features

UTxO-HD reworks the consensus codebase to decouple the LedgerState from the UTxO set, such that the latter can be stored on the disk therefore lowering the memory consumption of the node. It has two modes of operation: in-memory and on-disk, which use the same API so should be indistinguishable from the node's point of view.

The in-memory mode should work in a very similar way to the current node, with similar performance metrics. The on-disk mode will considerably lower the memory usage of the node at the cost of some performance. This is a tradeoff users will have to consider.

Using the on-disk mode, even a node running on a Raspberry Pi 5 (8GB of RAM) was able to follow the chain.

Currently on-disk uses LMDB underneath. In the long run, the LSM-tree library is being developed to provide a more performant on-disk backend.

In terms of functionality, a UTxO-HD node does not have more functionalities than a normal node.

You can try the feature already by building cardano-node from the utxo-hd-9.1 branch. It should work both with cabal and with nix.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[syndeton]

Should we be able to build utxo-hd-9.1.1? Or should we hold off for now?
I'm getting the following error when building .... Thanks

.fatal: Could not parse object 'c09534a2cbaeacf1d49a4a49478ccc61cb49799d'.
HEAD is now at 1751eea4e UTXO-HD 9.1
HEAD is now at bcf0e26e1 UTXO-HD 9.1
fatal: Could not parse object 'c09534a2cbaeacf1d49a4a49478ccc61cb49799d'.

Note: I'm able to build utxo-hd-9.1

[jasagredo]

I'm sorry, I moved the tips of the branches a bit and maybe I moved the tip of the ouroboros-consensus' utxo-hd-9.0 branch (although I though I had only added new commits on top). The source-repository-package stanzas in cardano-node@utxo-hd-9.1 branch should point to the tip of the utxo-hd-9.1 branches in cardano-cli and cardano-api and to the utxo-hd-9.0 branch in ouroboros-consensus.

[syndeton]

Just wanted to say great job with utxo-hd. It's looking like my server costs could be cut by half using it. A big help for those smaller struggling pools. You guys are heroes.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[jasagredo]

UTxO-HD branch on ouroboros-consensus (IntersectMBO/ouroboros-consensus#1267 + IntersectMBO/ouroboros-consensus#1304) is ready to be merged into main.

Before doing so, we will conduct a validation benchmarking run. We need to rebase the UTxO-HD branch on top of the latest node release (10.2.0).

Node 10.2 (#6050) currently fails to build so we will have to wait for that PR to build.

The process will be:

1. Keep the utxo-hd-main branch on top of main, for eventual merge.
2. Wait for cardano-node 10.2 release.
3. Backport utxo-hd to the version of consensus used in 10.2.
4. Use an s-r-p in cardano-api and port the utxo-hd branch on top of that one.
5. Use an s-r-p in cardano-cli and port the utxo-hd branch on top of that one.
6. Use an s-r-p in cardano-node and port the utxo-hd branch on top of that one.
7. Run the benchmarks and compare with 10.2.
8. If successful, merge the consensus branch into main.
9. Prepare a branch in cardano-api, cardano-cli and cardano-node main branches that integrate the changes for that consensus branch, for the next release engineer to use when integrating the next node version.