Feature: digest tracker #1370

xDimon · 2022-10-18T22:28:37Z

Referenced issues

Resolves #1363

Description of the Change

Observe digest and call special observer for each digest type
Move epoch descriptor mechanism to babe config repository
Move babe util implementation to babe config repository
Mode dispatch of finalisation event from block tree to babe

Benefits

Handling next epoch data
Conditional (by babe state) using runtime

codecov · 2022-10-19T13:42:01Z

Codecov Report

Merging #1370 (0f59fd6) into master (ed531b0) will decrease coverage by 0.41%.
The diff coverage is 14.40%.

❗ Current head 0f59fd6 differs from pull request most recent head dac2c10. Consider uploading reports for the commit dac2c10 to get more accurate results

@@            Coverage Diff             @@
##           master    #1370      +/-   ##
==========================================
- Coverage   24.85%   24.43%   -0.42%     
==========================================
  Files         629      633       +4     
  Lines       23794    24035     +241     
  Branches    12314    12470     +156     
==========================================
- Hits         5915     5874      -41     
- Misses      12603    12906     +303     
+ Partials     5276     5255      -21

Impacted Files	Coverage Δ
core/blockchain/block_tree.hpp	`100.00% <ø> (ø)`
core/blockchain/impl/block_tree_impl.cpp	`19.48% <0.00%> (-1.72%)`	⬇️
core/blockchain/impl/block_tree_impl.hpp	`33.33% <ø> (ø)`
core/blockchain/impl/cached_tree.hpp	`75.00% <ø> (ø)`
core/consensus/authority/impl/schedule_node.hpp	`37.50% <0.00%> (+4.16%)`	⬆️
core/consensus/babe/babe_config_repository.hpp	`100.00% <ø> (ø)`
core/consensus/babe/babe_error.cpp	`0.00% <0.00%> (ø)`
core/consensus/babe/babe_util.hpp	`100.00% <ø> (ø)`
core/consensus/babe/impl/babe_config_node.hpp	`0.00% <0.00%> (ø)`
core/consensus/babe/impl/babe_impl.hpp	`100.00% <ø> (ø)`
... and 34 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

test/core/host_api/offchain_extension_test.cpp

core/host_api/impl/offchain_extension.cpp

core/runtime/common/runtime_instances_pool.hpp

core/blockchain/impl/digest_tracker_impl.cpp

core/consensus/babe/impl/babe_config_node.cpp

core/consensus/babe/impl/babe_config_repository_impl.cpp

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

refactor: epoch descriptor storage moved to babe config repository refactor: babe util implementation moved to babe config repository Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

…error Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

…not in fast-sync) Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

Signed-off-by: safinsaf <safinsaft@gmail.com>

iceseer · 2022-11-10T07:49:02Z

core/consensus/babe/impl/babe_config_repository_impl.cpp

+
+      prune(block_info);
+
+      if (block_info.number % (kSavepointEachSuchBlock / 10) == 0) {


А почему в 10 раз чаще?
и тогда нужно поставить static_assert(kSavepointEachSuchBlock >= 10).

ходим с периодом kSavepointEachSuchBlock в цикле выше

iceseer · 2022-11-10T07:56:52Z

core/consensus/babe/impl/babe_config_repository_impl.cpp

+    // Save state on finalized part of blockchain
+    if (need_to_save) {
+      if (auto save_res = save(); save_res.has_error()) {
+        SL_WARN(logger_, "Can't re-save state: {}", save_res.error());


Мы должны продолжить работу если стейт не сохранился?

Насколько понимаю, мы будем продолжать пытаться сохранить стейт в этот чекпоинт пока находимся внутри этого периода, но как только перешагнем через границу kSavepointEachSuchBlock, будем сохранять уже следующий...насколько критичен пропуск если не удалось сделать сейвпоинт в пределах одного периода?

Yep, it not problem. It can be succeed next time. Anyway at the next start it can be recovered from last savepoint

iceseer · 2022-11-10T08:05:40Z

core/consensus/babe/impl/babe_config_repository_impl.cpp

+    BOOST_ASSERT(last_saved_state_block_ <= finalized_block.number);
+
+    auto saving_state_node = getNode(finalized_block);
+    BOOST_ASSERT_MSG(saving_state_node != nullptr,
+                     "Finalized block must have associated node");
+    const auto saving_state_block = saving_state_node->block;
+
+    // Does not need to save
+    if (last_saved_state_block_ >= saving_state_block.number) {
+      return outcome::success();
+    }


Мне кажется Ассерт и выражение под if противоречат друг другу. Если ситуация (last_saved_state_block_ > finalized_block.number) является нормальной и обрабатывается, то возможно ассерт лишний.

iceseer · 2022-11-10T08:07:21Z

core/consensus/babe/impl/babe_config_repository_impl.cpp

+    const auto last_savepoint =
+        (last_saved_state_block_ / kSavepointEachSuchBlock)
+        * kSavepointEachSuchBlock;


округление до кратного периоду встречается несколько раз, возможно имеет смысл вынести в отдельный метод.

xDimon force-pushed the feature/digest_tracker branch from fd7d08f to fde8929 Compare October 19, 2022 03:43

xDimon force-pushed the feature/digest_tracker branch from 903134e to bec874a Compare October 20, 2022 10:44

turuslan requested changes Oct 27, 2022

View reviewed changes

test/core/host_api/offchain_extension_test.cpp Outdated Show resolved Hide resolved

core/host_api/impl/offchain_extension.cpp Outdated Show resolved Hide resolved

core/runtime/common/runtime_instances_pool.hpp Outdated Show resolved Hide resolved

xDimon force-pushed the feature/digest_tracker branch 5 times, most recently from a4e1a7b to 8a7be9f Compare November 3, 2022 09:39

xDimon requested review from iceseer and kamilsa November 3, 2022 09:40

xDimon marked this pull request as ready for review November 3, 2022 09:45

xDimon force-pushed the feature/digest_tracker branch 15 times, most recently from 2ed3ee0 to 2e5a0f0 Compare November 8, 2022 09:10

xDimon requested a review from turuslan November 8, 2022 11:21

turuslan approved these changes Nov 8, 2022

View reviewed changes

xDimon force-pushed the feature/digest_tracker branch from 2e5a0f0 to a5cd511 Compare November 8, 2022 12:51

xDimon added 18 commits November 9, 2022 04:34

update: new hunter config (libp2p updated)

8cb43d7

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

feature: digest tracker

ae6ad24

refactor: epoch descriptor storage moved to babe config repository refactor: babe util implementation moved to babe config repository Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: on-disabled babe digest handling

3e7ddff

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: skip handling of wrong using on-disabled digest

c11b1ab

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: skip handling of wrong using on-disabled digest

bdf4fcb

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

refactor: remove handling of OnDisable babe-digest

4c3ad1a

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

refactor: rename to make short

8a05e3d

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: encoding NextConfigData

b1cee21

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: remove useless code

bb1a341

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: tests

1de52de

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: build on macos

9ea53cc

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: test

3c66735

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

feature: validation of block with disallowed slot assignment returns …

5bf1b6a

…error Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

refactor: move dispatching runtime version to babe (to do it if babe …

6e47847

…not in fast-sync) Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: test of listeners at disabled backward

06e5efe

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: tests

44f955f

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: block removal while recovery in progress

edec1d9

Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>

fix: review issues

a641045

Signed-off-by: safinsaf <safinsaft@gmail.com>

xDimon force-pushed the feature/digest_tracker branch from c40ced6 to a641045 Compare November 8, 2022 20:34

git: Merge branch 'master' into feature/digest_tracker

dac2c10

xDimon enabled auto-merge (squash) November 9, 2022 10:10

iceseer approved these changes Nov 10, 2022

View reviewed changes

xDimon merged commit 79ae107 into master Nov 10, 2022

xDimon deleted the feature/digest_tracker branch November 10, 2022 08:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: digest tracker #1370

Feature: digest tracker #1370

xDimon commented Oct 18, 2022 •

edited

Loading

codecov bot commented Oct 19, 2022 •

edited

Loading

iceseer Nov 10, 2022

iceseer Nov 10, 2022

iceseer Nov 10, 2022

xDimon Nov 10, 2022

iceseer Nov 10, 2022

iceseer Nov 10, 2022


		prune(block_info);

		if (block_info.number % (kSavepointEachSuchBlock / 10) == 0) {

Feature: digest tracker #1370

Feature: digest tracker #1370

Conversation

xDimon commented Oct 18, 2022 • edited Loading

Referenced issues

Description of the Change

Benefits

codecov bot commented Oct 19, 2022 • edited Loading

Codecov Report

iceseer Nov 10, 2022

Choose a reason for hiding this comment

iceseer Nov 10, 2022

Choose a reason for hiding this comment

iceseer Nov 10, 2022

Choose a reason for hiding this comment

xDimon Nov 10, 2022

Choose a reason for hiding this comment

iceseer Nov 10, 2022

Choose a reason for hiding this comment

iceseer Nov 10, 2022

Choose a reason for hiding this comment

xDimon commented Oct 18, 2022 •

edited

Loading

codecov bot commented Oct 19, 2022 •

edited

Loading