store: change to save DA and headers #218

tac0turtle · 2021-03-16T14:25:41Z

Description

Implement saving of header and DA header

Closes: #182

lgtm-com · 2021-03-20T01:14:08Z

This pull request introduces 2 alerts when merging 493c7e8 into 74c5b7d - view on LGTM.com

new alerts:

2 for Useless assignment to local variable

liamsi · 2021-03-20T07:41:08Z

rpc/core/blocks.go

+// todo: use IPLD plugin interface
+// func Block(ctx *rpctypes.Context, heightPtr *int64) (*ctypes.ResultBlock, error) {
+// 	height, err := getHeight(env.BlockStore.Height(), heightPtr)
+// 	if err != nil {
+// 		return nil, err
+// 	}
+
+// 	block := env.BlockStore.LoadBlock(height)
+// 	blockMeta := env.BlockStore.LoadBlockMeta(height)
+// 	if blockMeta == nil {
+// 		return &ctypes.ResultBlock{BlockID: types.BlockID{}, Block: block}, nil
+// 	}
+// 	return &ctypes.ResultBlock{BlockID: blockMeta.BlockID, Block: block}, nil
+// }


This would be a very simple function, that a) gets the block hash for the requested height, b) gets the DA header for that hash, c) uses the DA header to actually get the block data (e.g. via RetrieveBlockData of adr002 / the PRs that @evan-forbes is working on). Not sure if a) + b) could be one step instead.

tac0turtle · 2021-03-20T15:38:11Z

Note to self: Blockchain reactor and possibly consensus need to fetch the block in order to conduct replay against the app.

liamsi · 2021-03-24T08:23:27Z

This is blocked on the reading portion of ADR 002, right? #194

tac0turtle · 2021-04-04T14:00:30Z

This is blocked on the reading portion of ADR 002, right? #194

yes, otherwise there is nothing to verify for fast sync and consensus .

liamsi · 2021-04-15T10:08:14Z

Removed the stale label. We want to attack in the next few weeks now that the read portion is also there.
cc @evan-forbes @Wondertan @marbar3778

liamsi · 2021-04-16T14:26:32Z

Quoting @marbar3778:

Note to self: Blockchain reactor and possibly consensus need to fetch the block in order to conduct replay against the app.

So one way to unblock this PR, or rather chunk up the missing pieces into smaller would be to disable fast-sync first and use RetrieveBlockData during consensus. We can later figure out how to do syncing (basically fetching the DA header and calling RetrieveBlockData too).
Is this a correct summary of our brief discussion, Marko?

Just for context @Wondertan and @evan-forbes started working or will start working shortly on the using-RetrieveBlockData during consensus part. Does it make sense to hold off merging this until that part is already done?

Wondertan · 2021-04-16T15:03:39Z

basically fetching the DA header and calling RetrieveBlockData too

@liamsi, that's for full node, but in case I am a light node, how does chain syncing works for me? Is this simply fetching headers from last to first and sampling related blocks in the same order?

liamsi · 2021-04-16T15:30:22Z

Essentially, yes. There are basically two kinds of light nodes:

Nodes that only rely on DA during consensus: they receive the DA header with a proposal, and yes, they sample as you said.
Actual light-clients can either:
- rely on an honest majority assumption; these only follow headers and verify signatures - basically how the light-client works in tendermint right now
- do DA sampling, so they first have to get the DA header too and sample as you mentioned above too

Nodes that want to actually sync data and replay state still need to be defined more concretely. But they could download the data in parallel (doesn't have to be in order).

Wondertan · 2021-04-16T15:47:11Z

@liamsi, I might have missed mentioning this in the question, but I meant a case where a new node joins an existing network. Imagine the network is more than 10000 epochs large and the new node that does not want to rely on honest majority needs to sync the chain.

Btw, is the case above called a replay in tm terms? Or replay is only related to state, while chain sync is specific for chain validation? AFAIK, most blockchains combine those two terms, but in this case, state validity is not a concern for Celestial consensus, thus they are separated.

adlerjohn · 2021-04-16T16:05:41Z

The particular deployed node can define how it's parametrized. Light nodes work as @Wondertan described (download all block headers, do DAS for each block), while superlight nodes rely on an honest majority assumption. See: https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/node_types.md#node-type-definitions

Which ones is used depends on what the node runner wants.

liamsi

@marbar3778 we decided to store the DAHeader inside of BlockMeta for now instead. Any chance you could revive this PR including this change (basically merging in master and some minor changes to the API) soon?
Related:

tac0turtle · 2021-06-03T22:03:55Z

@marbar3778 we decided to store the DAHeader inside of BlockMeta for now instead. Any chance you could revive this PR including this change (basically merging in master and some minor changes to the API) soon?
Related:
* #374

* specifically: https://github.com/lazyledger/lazyledger-core/pull/374/files#r645077665

yea will resurrect it tomorrow and over the weekend. Thanks for the ping

liamsi · 2021-08-16T09:19:03Z

@marbar3778 we decided that for our devnet (see #491) we will try to keep the changes to tendermint as minimal as possible. Hence, we will just keep the vanilla tendermint store and use a dedicated "Celestia node" that connects to a tendermint node e.g. via RPC to retrieve the block, erasure code it (again) and store and serve it to DAS nodes.

I'm going to close this issue for now but please do not delete the branch. We might come back to this at a later point in time to only store the block data once.

tac0turtle added 4 commits March 9, 2021 14:45

begin store interface migration

5388c95

regenerate blockchain messages

9f701f6

blockchain messages

7b5db66

Merge branch 'master' into marko/store

77d4e81

tac0turtle mentioned this pull request Mar 16, 2021

store: change to save DA and headers #190

Closed

celestiaorg deleted a comment from lgtm-com bot Mar 16, 2021

store tests

493c7e8

Wondertan mentioned this pull request Mar 19, 2021

DA ready consensus #205

Closed

liamsi reviewed Mar 20, 2021

View reviewed changes

work on blokchain

e432bf8

This comment has been minimized.

Sign in to view

github-actions bot added the Stale label Apr 4, 2021

tac0turtle added S:blocked Status Blocked and removed Stale labels Apr 4, 2021

This comment has been minimized.

Sign in to view

github-actions bot added the Stale label Apr 15, 2021

liamsi removed the Stale label Apr 15, 2021

liamsi mentioned this pull request Apr 20, 2021

Replace BlockID with the HeaderHash #287

Closed

liamsi removed the S:blocked Status Blocked label Apr 29, 2021

liamsi mentioned this pull request Jun 3, 2021

Save and load block data using IPFS #374

Merged

liamsi reviewed Jun 3, 2021

View reviewed changes

tac0turtle added 3 commits June 14, 2021 17:14

Merge branch 'master' into marko/store

cac7844

some comment outs and amend block store interface

ce52e44

rpc fix, blockstore interface amendment

8ae87cc

evan-forbes mentioned this pull request Jun 16, 2021

Remove the PartSetHeader where possible #418

Closed

3 tasks

This was referenced Jun 25, 2021

Revert writing to local dag during proposal #436

Closed

Return a proof that a transaction or message has been included in a block #416

Closed

liamsi closed this Aug 16, 2021

tac0turtle deleted the marko/store branch September 2, 2021 07:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

store: change to save DA and headers #218

store: change to save DA and headers #218

tac0turtle commented Mar 16, 2021 •

edited by liamsi

Loading

lgtm-com bot commented Mar 20, 2021

liamsi Mar 20, 2021

tac0turtle commented Mar 20, 2021

This comment has been minimized.

liamsi commented Mar 24, 2021

This comment has been minimized.

tac0turtle commented Apr 4, 2021

This comment has been minimized.

liamsi commented Apr 15, 2021

liamsi commented Apr 16, 2021

Wondertan commented Apr 16, 2021

liamsi commented Apr 16, 2021

Wondertan commented Apr 16, 2021 •

edited

Loading

adlerjohn commented Apr 16, 2021 •

edited

Loading

liamsi left a comment

tac0turtle commented Jun 3, 2021

liamsi commented Aug 16, 2021

store: change to save DA and headers #218

store: change to save DA and headers #218

Conversation

tac0turtle commented Mar 16, 2021 • edited by liamsi Loading

Description

lgtm-com bot commented Mar 20, 2021

liamsi Mar 20, 2021

Choose a reason for hiding this comment

tac0turtle commented Mar 20, 2021

This comment has been minimized.

liamsi commented Mar 24, 2021

This comment has been minimized.

tac0turtle commented Apr 4, 2021

This comment has been minimized.

liamsi commented Apr 15, 2021

liamsi commented Apr 16, 2021

Wondertan commented Apr 16, 2021

liamsi commented Apr 16, 2021

Wondertan commented Apr 16, 2021 • edited Loading

adlerjohn commented Apr 16, 2021 • edited Loading

liamsi left a comment

Choose a reason for hiding this comment

tac0turtle commented Jun 3, 2021

liamsi commented Aug 16, 2021

tac0turtle commented Mar 16, 2021 •

edited by liamsi

Loading

Wondertan commented Apr 16, 2021 •

edited

Loading

adlerjohn commented Apr 16, 2021 •

edited

Loading