Light NetworkLayer: lightSync #3175

HeinrichApfelmus · 2022-03-16T17:32:55Z

Issue number

Overview

Light-mode (Epic ADP-1422) aims to make synchronisation to the blockchain faster by trusting an off-chain source of aggregated blockchain data.

In this pull request, we implement lightSync, the "crown jewel" 👑 of light-mode:

lightSync drives a ChainFollower with LightBlocks by polling an external blockchain data source (represented by LightSyncSource) as opposed to being pushed by a cardano-node.
prop_followLightSync tests this rigorously by simulating a randomly evolving blockchain with rollbacks.

Details

lightSync is implemented and tested, but propagating it to NetworkLayer is left for a future pull request.
secondsPerSlot is currently hard-coded to 2 seconds; also left for the future when the shape of LightLayer becomes more clear.

Comments

I have opted to not use quickcheck-state-machine, as I felt that the trouble of setting up the Reference type machinery did not seem worth the benefit.
- However, I have kept the principle of first generating a history (genChainHistory) and then driving a (monadic) state machine with it (evalMockMonad) — it was just easier to do it "by hand".
- That said, labelling histories to check that interesting histories have been generated would be a good idea — but I'm sufficiently confident in the present implementation that I would like to postpone this for a future time, also because I envision some synergy between randomized history generation and Delta encodings.
- By the same token, I have forgone shrinking — as long as they are no bugs, we don't need to shrink any. :)
genChainHistory generates random histories of BlockHeader with correct parentHeaderHash. Extending this to valid histories of Block in the future would allows us to test UTxO and address discovery state as well.

HeinrichApfelmus · 2022-03-17T11:34:09Z

bors try

iohk-bors · 2022-03-17T12:16:32Z

try

Build failed:

buildkite/cardano-wallet

paweljakubas · 2022-03-23T19:23:12Z

lib/core/src/Cardano/Wallet/Network/Light.hs

+        -- or whether the chain has rolled back already.
+    , getBlockHeaderAtHeight :: BlockHeight -> m (Maybe BlockHeader)
+        -- ^ Get the 'BlockHeader' at a given block height.
+        -- Returns 'Nothing' if there is no block at this height (anymore).


not better to return whatever is present up to block height?

Also thinking about source abstraction. Maybe it would be worth to sketch the abtraction that would handle this light wallet source and node source? I think we can do it later, now we can demonstrate it just works. But maybe we can do something now. What do you think? Too soon? Or you see already silluette of this abstraction?

Also thinking about source abstraction.

I've had some discussion with Yura about this, and the conclusion was "too soon". I was thinking of some sort of general LightLayer abstraction indeed, but the details seemed unclear. We decided to postpone a decision there, and I chose to introduce the smallest type that can be used to implement lightSync instead.

not better to return whatever is present up to block height?

In the lightSync procedure, getBlockHeaderAtHeight is only used to move a pointer around, here to a block slightly older than the tip of the chain (by subtracting stabilityWindow from the height).

The actual data for blocks is returned by getNextBlocks and getAddressTxs.

lib/core/src/Cardano/Wallet/Network/Light.hs

paweljakubas

The logic of light syncing looks reasonable. Nice mocking tests (eval fullFollower === eval lightSyncFollower) . I think it begs to use state machine testing as we have well defined states and known transitions - could be done in next PRs if you finally are persuaded (I read your comment why you resign from using this approach). Maybe we could catch something. It would be good to introduce rollbacks distributions (probably scaled Poisson with lambda=1 with cutoff) to mimic what we have, so many 2-3 depth rollbacks, rare higher depth rollbacks, cut with security parameter. So we could defined chain evolution with this rollback influencer, apply very realistic blocks...and verify consensus. just food for thought

lib/core/src/Cardano/Wallet/Network/Light.hs

lib/core/test/unit/Cardano/Wallet/Network/LightSpec.hs

Unisay · 2022-03-28T14:17:10Z

lib/core/test/unit/Cardano/Wallet/Network/LightSpec.hs

+data Free f a
+    = Free (f (Free f a))
+    | Pure a


Why invent free monad?

https://hackage.haskell.org/package/free-5.1.7/docs/Control-Monad-Free.html#t:Free

Many reasons, actually. 😅

Main reason: The wallet does not yet import use a free package or one of its many variants. I didn't want to introduce a package-level idiom without team discussion, but I felt comfortable introduce a local idiom. (My preference there would actually be my own operational. 😅 )

We already have another hand-baked monad, the LSQ monad.

The definition of Free is ~20, fairly simple lines. This is hard to beat.

HeinrichApfelmus · 2022-03-28T18:12:29Z

Thanks for taking a look, Pawel, Yura. 😊

rollbacks distributions (probably scaled Poisson with lambda=1 with cutoff)

Using a Poission distribution to mimic the real chain is an interesting idea. 🤔 However, I figured that for the purpose of stress-testing the code, using a simple uniform distribution (choose) would be best:

    backward = do
        m <- choose (0, min (NE.length chain1 - 1) mockStabilityWindow)
        let chain2 = NE.fromList (NE.drop m chain1)

For testing, we actually want the rollbacks to be worse than in the real world chain. 😂

Includes testing on a randomly evolving `MockChain` history with rollbacks.

Co-authored-by: paweljakubas <jakubas.pawel@gmail.com> Co-authored-by: Yura Lazarev <Unisay@users.noreply.github.com>

HeinrichApfelmus · 2022-03-28T18:14:03Z

bors merge

3175: Light NetworkLayer: lightSync r=HeinrichApfelmus a=HeinrichApfelmus ### Issue number ADP-1422, ADP-1427 ### Overview [Light-mode][] (Epic ADP-1422) aims to make synchronisation to the blockchain faster by trusting an off-chain source of aggregated blockchain data. [light-mode]: https://input-output-hk.github.io/cardano-wallet/design/specs/light-mode In this pull request, we implement `lightSync`, the "crown jewel" 👑 of light-mode: * `lightSync` drives a `ChainFollower` with `LightBlocks` by polling an external blockchain data source (represented by `LightSyncSource`) as opposed to being pushed by a cardano-node. * `prop_followLightSync` tests this rigorously by simulating a randomly evolving blockchain with rollbacks. ### Details * `lightSync` is implemented and tested, but propagating it to `NetworkLayer` is left for a future pull request. * `secondsPerSlot` is currently hard-coded to 2 seconds; also left for the future when the shape of `LightLayer` becomes more clear. ### Comments * I have opted to *not* use [quickcheck-state-machine][qsm], as I felt that the trouble of setting up the `Reference` type machinery did not seem worth the benefit. * However, I have kept the principle of first generating a history (`genChainHistory`) and then driving a (monadic) state machine with it (`evalMockMonad`) — it was just easier to do it "by hand". * That said, *labelling* histories to check that interesting histories have been generated would be a good idea — but I'm sufficiently confident in the present implementation that I would like to postpone this for a future time, also because I envision some synergy between randomized history generation and `Delta` encodings. * By the same token, I have forgone shrinking — as long as they are no bugs, we don't need to shrink any. :) * `genChainHistory` generates random histories of `BlockHeader` with correct `parentHeaderHash`. Extending this to valid histories of `Block` in the future would allows us to test `UTxO` and address discovery state as well. [qsm]: https://well-typed.com/blog/2019/01/qsm-in-depth/ Co-authored-by: Heinrich Apfelmus <heinrich.apfelmus@iohk.io>

iohk-bors · 2022-03-28T18:15:04Z

Canceled.

HeinrichApfelmus · 2022-03-28T20:42:53Z

bors merge

iohk-bors · 2022-03-28T21:23:41Z

Build succeeded:

HeinrichApfelmus changed the base branch from master to HeinrichApfelmus/ADP-1422/applyBlockEvents March 16, 2022 23:08

HeinrichApfelmus force-pushed the HeinrichApfelmus/ADP-1422/lightSync branch 4 times, most recently from 49aedec to 2a2030b Compare March 17, 2022 11:04

iohk-bors bot added a commit that referenced this pull request Mar 17, 2022

Try #3175:

a240210

HeinrichApfelmus requested review from paweljakubas and Unisay March 17, 2022 12:18

Base automatically changed from HeinrichApfelmus/ADP-1422/applyBlockEvents to master March 21, 2022 14:04

paweljakubas reviewed Mar 23, 2022

View reviewed changes

lib/core/src/Cardano/Wallet/Network/Light.hs Outdated Show resolved Hide resolved

paweljakubas reviewed Mar 23, 2022

View reviewed changes

lib/core/src/Cardano/Wallet/Network/Light.hs Show resolved Hide resolved

paweljakubas reviewed Mar 23, 2022

View reviewed changes

lib/core/src/Cardano/Wallet/Network/Light.hs Show resolved Hide resolved

paweljakubas approved these changes Mar 23, 2022

View reviewed changes

paweljakubas reviewed Mar 23, 2022

View reviewed changes

lib/core/src/Cardano/Wallet/Network/Light.hs Outdated Show resolved Hide resolved

Unisay reviewed Mar 28, 2022

View reviewed changes

lib/core/test/unit/Cardano/Wallet/Network/LightSpec.hs Outdated Show resolved Hide resolved

Unisay reviewed Mar 28, 2022

View reviewed changes

HeinrichApfelmus and others added 2 commits March 28, 2022 20:12

Implement lightSync

3372c06

Includes testing on a randomly evolving `MockChain` history with rollbacks.

Apply suggestions from code review

c95f444

Co-authored-by: paweljakubas <jakubas.pawel@gmail.com> Co-authored-by: Yura Lazarev <Unisay@users.noreply.github.com>

HeinrichApfelmus force-pushed the HeinrichApfelmus/ADP-1422/lightSync branch from 3ad6102 to c95f444 Compare March 28, 2022 18:13

Regenerate nix

53aedc8

iohk-bors bot merged commit d602ee8 into master Mar 28, 2022

iohk-bors bot deleted the HeinrichApfelmus/ADP-1422/lightSync branch March 28, 2022 21:23

WilliamKingNoel-Bot pushed a commit that referenced this pull request Mar 28, 2022

docs: Merge #3175

7252c98

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Light NetworkLayer: lightSync #3175

Light NetworkLayer: lightSync #3175

HeinrichApfelmus commented Mar 16, 2022 •

edited

Loading

HeinrichApfelmus commented Mar 17, 2022

iohk-bors bot commented Mar 17, 2022

paweljakubas Mar 23, 2022

paweljakubas Mar 23, 2022

HeinrichApfelmus Mar 28, 2022 •

edited

Loading

HeinrichApfelmus Mar 28, 2022

paweljakubas left a comment •

edited

Loading

Unisay Mar 28, 2022

HeinrichApfelmus Mar 28, 2022

HeinrichApfelmus commented Mar 28, 2022

HeinrichApfelmus commented Mar 28, 2022

iohk-bors bot commented Mar 28, 2022

HeinrichApfelmus commented Mar 28, 2022

iohk-bors bot commented Mar 28, 2022

Light NetworkLayer: lightSync #3175

Light NetworkLayer: lightSync #3175

Conversation

HeinrichApfelmus commented Mar 16, 2022 • edited Loading

Issue number

Overview

Details

Comments

HeinrichApfelmus commented Mar 17, 2022

iohk-bors bot commented Mar 17, 2022

try

paweljakubas Mar 23, 2022

Choose a reason for hiding this comment

paweljakubas Mar 23, 2022

Choose a reason for hiding this comment

HeinrichApfelmus Mar 28, 2022 • edited Loading

Choose a reason for hiding this comment

HeinrichApfelmus Mar 28, 2022

Choose a reason for hiding this comment

paweljakubas left a comment • edited Loading

Choose a reason for hiding this comment

Unisay Mar 28, 2022

Choose a reason for hiding this comment

HeinrichApfelmus Mar 28, 2022

Choose a reason for hiding this comment

HeinrichApfelmus commented Mar 28, 2022

HeinrichApfelmus commented Mar 28, 2022

iohk-bors bot commented Mar 28, 2022

HeinrichApfelmus commented Mar 28, 2022

iohk-bors bot commented Mar 28, 2022

HeinrichApfelmus commented Mar 16, 2022 •

edited

Loading

HeinrichApfelmus Mar 28, 2022 •

edited

Loading

paweljakubas left a comment •

edited

Loading