Add epoch stakepool size indexer #850

eyeinsky · 2022-12-05T16:51:54Z

PLT-171: marconi indexer: stake pool delegation of a particular epoch

Diff is easiest to review by looking at the separate commits as they do isolated things.

Also added a longer commit messages for the foldBlocks adaption commit 440ff75.

Test with: cabal run marconi-test -- -p '/prop_epoch_stakepool_size/'

Run on testnet or mainnet:

CONFIG="..." # node config is required for folding over ledger states
SOCKET="..."
MAGIC='--mainnet' # or --testne-magic 2 # 2 for preprod
DB_PATH="..."
# rm -f "$DB_PATH/*" # might need to remove previous content on second run

cabal run marconi:exe:marconi -- \
      --disable-utxo --disable-datum --disable-script-tx \
      $MAGIC \
      --node-config-path "$CONFIG" \
      --socket-path "$SOCKET" \
      -d "$DB_PATH"

Depends on these PRs on cardano-node repo (same content, one is just for the 1.35 release branch):

Pre-submit checklist:

Branch
- Tests are provided (if possible)
- Commit sequence broadly makes sense
- Key commits have useful messages
- Formatting, PNG optimization, etc. are updated
- Important changes are reflected in changelog.d of the affected packages
PR
- Self-reviewed the diff
- Useful pull request description
- Reference the ADR in the PR and reference the PR in the ADR (if revelant)
- Reviewer requested

koslambrou

My main issue with foldBlocks is that you can't resume. You always need to start from genesis whenever you stop and restart the indexer. I guess we want to work on this later?

Also, thanks for the tests :) Learned more about stake pool registration

cardano-streaming/src/Cardano/Streaming.hs

marconi/test/Helpers.hs

cardano-streaming/src/Cardano/Streaming/Helpers.hs

changelog.d/20221205_182223_markus.lall_plt_171.rst

eyeinsky · 2022-12-06T14:14:08Z

Ran it on mainnet, preliminary info here https://input-output.atlassian.net/browse/PLT-171?focusedCommentId=136344

marconi/test/Helpers.hs

cardano-streaming/src/Cardano/Streaming.hs

cardano-streaming/src/Cardano/Streaming/Helpers.hs

eyeinsky · 2022-12-07T11:11:26Z

@raduom The comment on rollbackRingBuffer went outdated as it was removed, but the answer to why it was here in the first place is here #850 (comment)

eyeinsky · 2022-12-07T13:40:05Z

My main issue with foldBlocks is that you can't resume. You always need to start from genesis whenever you stop and restart the indexer. I guess we want to work on this later?

Here are some more elaborate ideas that could help with it:

library-ize cardano-node: make it a library which could be used to build an indexer. It would have an API that would stream blocks and ledger states within the same process (= no double RAM use). I.e, cardano-node would be compiled into the indexer, and had an appropriate API.

The node's library part would take care of networking, discovering and talking to other nodes, getting up to sync (essentially doing what currently the node running side-by-side with an indexer is doing). And addressing the security aspect mentioned in the other feature request ("The node is part of the trusted base of the system"), it shouldn't participate actively in the network, it will just be a passive listener. I think it would still need to have ledger state, as that is used to validate incoming blocks (transactions). If validation is not required then perhaps ledger state could be done away with entirely.
Since cardano-node is already able to serialize ledger state somehow (because it can be restarted) then using cardano-node as a library would solve the restart issue.

Aside from issue linked above, there has been a proposal by @erikd to pass pieces of ledger state over the chainsync protocol. But I think having a protocol to pass info over a socket in a situation where both a node and dbsync are typically running on the same machine, is just overhead -- just compile the node into the indexer, and the problems of double memory use, serializing and deserializing ledger state, coming up with a protocol on how to query for pieces of ledger state, etc. would all go away.

marconi/test/Integration.hs

koslambrou · 2022-12-13T15:48:32Z

You still have some conflicts

eyeinsky · 2022-12-14T13:53:10Z

Rebased on to lates master and pushed.

These PRs in cardano-node would still be needed to be merged:

The CI seems to fail for unrelated reasons there (as far as I can tell).

kayvank · 2022-12-19T19:10:16Z

cardano-streaming/cardano-streaming.cabal

+    , base                          >=4.9 && <5
+    , base16-bytestring
+    , bytestring
+    , cardano-api


Aren't these IOG dependencies?

kayvank · 2022-12-19T20:12:23Z

marconi/src/Marconi/Index/EpochStakepoolSize.hs

+sqlite :: SQL.Connection -> S.Stream (S.Of Event) IO r -> S.Stream (S.Of Event) IO r
+sqlite c source = do
+  lift $ SQL.execute_ c
+      "CREATE TABLE IF NOT EXISTS stakepool_delegation (poolId BLOB NOT NULL, lovelace INT NOT NULL, epochNo INT NOT NULL)"


I think we may need a primary key on this table, or some indicator that make a ROW unique, besides the implicit SQLite Row-Id.
We don't drop the table, we recreate if it doesn't exist, and start inserting it to it, so it is possible to insert the same events multiple times which I don't think is correct

…no-api - Cardano.Streaming.Callbacks: Add primitive callback based versions of local chainsync to Cardano.Streaming. `blocksCallback` and `blocksCallbackPipelined` both take a callback when iterating over the received blocks from local chainsync connection - Cardano.Streaming: streaming package's versions to stream blocks or ledger states. `foldLedgerState` does essentially what `foldBlocks` does in cardano-api:Cardano.Api.LedgerState, with the difference that it doesn't drop connection when it's up to date with regarding to the node, but keeps listening.

The `toEvents` and `getStakeMap` make up the core of the implementation in Marconi.Index.EpochStakepoolSize.

* Add suffixes to sqlite databases * Add ledger state streaming, equivalent to foldBlocks defined in cardano-api - Cardano.Streaming.Callbacks: Add primitive callback based versions of local chainsync to Cardano.Streaming. `blocksCallback` and `blocksCallbackPipelined` both take a callback when iterating over the received blocks from local chainsync connection - Cardano.Streaming: streaming package's versions to stream blocks or ledger states. `foldLedgerState` does essentially what `foldBlocks` does in cardano-api:Cardano.Api.LedgerState, with the difference that it doesn't drop connection when it's up to date with regarding to the node, but keeps listening. * Add epoch stakepool size indexer The `toEvents` and `getStakeMap` make up the core of the implementation in Marconi.Index.EpochStakepoolSize. * Refactor marconi test for reuse in epoch stakepool size indexer * Add test for epoch stakepool size indexer * Add changelog entry * Disable test and add TODO * Fix warnings * Fix marconi.cabal

eyeinsky requested review from kayvank, koslambrou and raduom December 5, 2022 16:51

koslambrou approved these changes Dec 5, 2022

View reviewed changes

catch-21 reviewed Dec 6, 2022

View reviewed changes

marconi/test/Helpers.hs Outdated Show resolved Hide resolved

catch-21 reviewed Dec 6, 2022

View reviewed changes

marconi/test/Helpers.hs Show resolved Hide resolved

raduom reviewed Dec 7, 2022

View reviewed changes

eyeinsky mentioned this pull request Dec 7, 2022

CIP-0078? | Extended Local Chain Sync Protocol cardano-foundation/CIPs#375

Closed

catch-21 reviewed Dec 13, 2022

View reviewed changes

marconi/test/Integration.hs Show resolved Hide resolved

eyeinsky force-pushed the ml/plt-171 branch from 3a70015 to a2e2434 Compare December 13, 2022 15:01

eyeinsky force-pushed the ml/plt-171 branch from 175956c to d8dd1cd Compare December 14, 2022 11:43

kayvank reviewed Dec 19, 2022

View reviewed changes

eyeinsky force-pushed the ml/plt-171 branch from d8dd1cd to 6ef420e Compare January 16, 2023 17:44

eyeinsky added 6 commits January 17, 2023 13:27

Add suffixes to sqlite databases

9778bf6

Add epoch stakepool size indexer

2478ff7

The `toEvents` and `getStakeMap` make up the core of the implementation in Marconi.Index.EpochStakepoolSize.

Refactor marconi test for reuse in epoch stakepool size indexer

e4dabcd

Add test for epoch stakepool size indexer

c205dde

Add changelog entry

4cb3dee

eyeinsky force-pushed the ml/plt-171 branch from 6ef420e to 4cb3dee Compare January 17, 2023 12:45

eyeinsky and others added 3 commits January 17, 2023 14:50

Disable test and add TODO

e208d32

Fix warnings

a8f3793

Fix marconi.cabal

4f01f90

eyeinsky merged commit 6990714 into main Jan 17, 2023

eyeinsky deleted the ml/plt-171 branch January 17, 2023 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add epoch stakepool size indexer #850

Add epoch stakepool size indexer #850

eyeinsky commented Dec 5, 2022 •

edited

Loading

koslambrou left a comment

eyeinsky commented Dec 6, 2022

eyeinsky commented Dec 7, 2022

eyeinsky commented Dec 7, 2022

koslambrou commented Dec 13, 2022

eyeinsky commented Dec 14, 2022

kayvank Dec 19, 2022

kayvank Dec 19, 2022

Add epoch stakepool size indexer #850

Add epoch stakepool size indexer #850

Conversation

eyeinsky commented Dec 5, 2022 • edited Loading

koslambrou left a comment

Choose a reason for hiding this comment

eyeinsky commented Dec 6, 2022

eyeinsky commented Dec 7, 2022

eyeinsky commented Dec 7, 2022

koslambrou commented Dec 13, 2022

eyeinsky commented Dec 14, 2022

kayvank Dec 19, 2022

Choose a reason for hiding this comment

kayvank Dec 19, 2022

Choose a reason for hiding this comment

eyeinsky commented Dec 5, 2022 •

edited

Loading