Updating cardano-ledger

[JaredCorduan]

I have updated the cardano ledger package to the current master, and ouroboros-network to my last commit in IntersectMBO/ouroboros-network#2971.

• The ledger state now stores an PulsingRewUpdate instead of a
  RewardUpdate. The PulsingRewUpdate allows us to compute the staking
  rewards incrementally instead of all at once. This involves a new
  serialization format for the reward updates.
• Annotated decoders are now used for PParams and PParamsDelta (the
  data structures now store their own serialization). Additionally, the
  LedgerState also now uses an annotated decoder. The serialization in
  consensus has been changed to use the cbor-in-cbor encoding for these
  types (including in the queries).
• A bug was fixed in the reward calculation which was introduced in
  commit 31083fc5cecbe17f3628393072094b130058edd9.
• The ledger repo now uses ghc-8.10.4 and cabal-3.4.0.0.

Question is it okay that I have removed the FromCBOR instances of the protocol parameter updates and the ledger state? I did add the necessary functions to serialize them.

[JaredCorduan]

is this alright?

[Jimbo4350]

I think if we're not interested in the intermediate calculation Aeson.Null would probably be more appropriate.

[DavidEichmann]

There is only a ToJSON instance and no FromJSON instance. Is this because the output is only intended as debugging output for consumption by 3rd party tools? Or something along those lines?

[JaredCorduan]

that's my understanding @DavidEichmann , but @Jimbo4350 would know better.

[gitmachtl]

There is only a ToJSON instance and no FromJSON instance. Is this because the output is only intended as debugging output for consumption by 3rd party tools? Or something along those lines?

yes. almost every SPO is using the JSON output for daily operation, calculations, leaderlogs, etc... or are we talking about another ToJSON here?

[JaredCorduan]

I am now considering if it would be better to do the cbor-in-cbor encoding in the ledger package itself, so that there could be a FromCBOR instance of LedgerState and thus break less downstream. I'll maybe tackle that on Monday if the others agree.

[JaredCorduan]

I've talked with @nc6 about this, and we decided that not having these odd cbor-in-cbor FromCBOR instances is probably best. So now I just need to make sure that node is adapted to handle the Annotator (LedgerState era).

[JaredCorduan]

I converted this back to draft, since the cbor-in-cbor encoding in the consensus query needs to be reverted.

[gufmar]

As someone who is not directly involved, I would like to add a thought:
The reward calculation takes place with a safety margin of 48 hours after the last epoch change. the payment takes place days later.
So there should be no scientific or technical reason why this calculation is carried out by all nodes at exactly the same time. Theoretically, it should be sufficient to define that the calculation must be completed within a time window of e.g. 4 hours from the 48h point. With the current or even lower saturation point, a large pool currently generates about 55 blocks per epoch, i.e. one every 2 hours on average. With the current fixed and uncoordinated 48h starting point, however, it is still certain to hit about 20-40 block generators.
What is the objection to the fact that for this time frame of the rewards calculation, an advance determination of the allocated slots is made internally in the node, e.g. for all slots of the next 20 minutes. The reward calculation only takes place if it becomes apparent that no slot is pending anyway. Otherwise, a new check is carried out after this block, up to a maximum of e.g. +4 hours. Then the reward calculation is carried out even if a block is pending, which will be a practically very rare case.

[JaredCorduan]

That is correct @gufmar , the nodes can do the calculation at any point in the epoch after the stability window. We have a fix in the ledger code that essentially just computes the rewards for a a small number of pools (probably one given the current number of registered stake pools) at a time. see IntersectMBO/cardano-ledger#2142

We are just trying to handle all the dependencies now.

[JaredCorduan]

Replaced by #2474

jc/update-ledger-to-ba74779256 was an overly-ambitious (and now incorrect) branch name 😄