blockstore: send duplicate proofs for chained merkle root conflicts #835

AshwinSekar · 2024-04-16T05:05:26Z

alternative implementation of #102

AshwinSekar · 2024-04-16T05:07:14Z

ledger/src/blockstore.rs

@@ -542,12 +542,6 @@ impl Blockstore {
        let candidate_erasure_set = ErasureSetId::new(slot, candidate_fec_set_index);
        let candidate_erasure_meta: ErasureMeta = deserialize(candidate_erasure_meta.as_ref())?;

-        // Add this candidate to erasure metas to avoid blockstore lookup in future


no reason to do this as it is only called at the end of do_insert_shreds which does not need to perform any lookups after.

ledger/src/blockstore.rs

codecov-commenter · 2024-04-16T16:46:01Z

Codecov Report

Attention: Patch coverage is 96.20690% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 81.9%. Comparing base (f133697) to head (48958a3).

Additional details and impacted files

@@           Coverage Diff            @@
##           master     #835    +/-   ##
========================================
  Coverage    81.9%    81.9%            
========================================
  Files         855      855            
  Lines      232134   232385   +251     
========================================
+ Hits       190134   190544   +410     
+ Misses      42000    41841   -159

behzadnouri

overall this looks less convoluted that the previous code.

behzadnouri · 2024-04-18T15:43:53Z

ledger/src/blockstore_meta.rs

+    /// Offset to the index of the first received coding shred
+    first_received_coding_shred_offset: usize,


Why do we need an "offset" here instead of just first_received_coding_shred_index?

I used an offset to avoid having to convert the index from u32 to usize, but I think it should be fine to just store the index directly.

behzadnouri · 2024-04-18T15:49:54Z

ledger/src/blockstore_meta.rs

For easier review, can you please move all ErasureMeta changes to a separate PR?

behzadnouri · 2024-04-18T15:57:34Z

ledger/src/blockstore.rs

+                    ShredType::Code,
+                );
+                let shred = Shred::new_from_serialized_shred(
+                    self.get_shred_from_just_inserted_or_db(&just_inserted_shreds, shred_id)


just_inserted_shreds should always include this shred because this is a new ErasureMeta, no?

behzadnouri · 2024-04-18T16:03:02Z

ledger/src/blockstore.rs

+                    merkle_root_meta.first_received_shred_type(),
+                );
+                let shred = Shred::new_from_serialized_shred(
+                    self.get_shred_from_just_inserted_or_db(&just_inserted_shreds, shred_id)


similarly here, shouldn't we expect shred to be already in just_inserted_shreds?

behzadnouri · 2024-04-18T16:06:42Z

ledger/src/blockstore.rs

+                next fec set shred {next_erasure_set:?} type {:?} chains to merkle root {chained_merkle_root:?}.
+                Reporting as duplicate",
+                shred.shred_type(),
+                next_shred_type,


https://rust-lang.github.io/rust-clippy/master/index.html#/uninlined_format_args

mergify · 2024-04-23T17:37:45Z

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

behzadnouri · 2024-04-23T18:12:27Z

ledger/src/blockstore.rs

+        let (next_shred_index, next_shred_type) = if let Some(merkle_root_meta_entry) =
+            merkle_root_metas.get(&next_erasure_set)
+        {
+            (
+                merkle_root_meta_entry.as_ref().first_received_shred_index(),
+                merkle_root_meta_entry.as_ref().first_received_shred_type(),
+            )
+        } else if let Some(merkle_root_meta) = self.merkle_root_meta(next_erasure_set).unwrap() {
+            (
+                merkle_root_meta.first_received_shred_index(),
+                merkle_root_meta.first_received_shred_type(),
+            )
+        } else {
+            // No shred from the next fec set has been received
+            return true;


Might be more readable to define next_merkle_root_meta as below and work with that.

let Some(next_merkle_root_meta) = merkle_root_metas .get(&next_erasure_set) .map(WorkingEntry::as_ref) .map(Cow::Borrowed) .or_else(|| { self.merkle_root_meta(next_erasure_set) .unwrap() .map(Cow::Owned) }) else { return true; // No shred from the next fec set has been received }; let next_shred_id = ShredId::new( slot, next_merkle_root_meta.first_received_shred_index(), next_merkle_root_meta.first_received_shred_type(), );

behzadnouri · 2024-04-23T18:20:08Z

ledger/src/blockstore.rs

+        let (prev_shred_index, prev_shred_type) =
+            if let Some(prev_merkle_root_meta_entry) = merkle_root_metas.get(&prev_erasure_set) {
+                (
+                    prev_merkle_root_meta_entry
+                        .as_ref()
+                        .first_received_shred_index(),
+                    prev_merkle_root_meta_entry
+                        .as_ref()
+                        .first_received_shred_type(),
+                )
+            } else {
+                let merkle_root_meta = self
+                    .merkle_root_meta(prev_erasure_set)
+                    .unwrap()
+                    .expect("merkle root meta must exist for erasure meta");
+                (
+                    merkle_root_meta.first_received_shred_index(),
+                    merkle_root_meta.first_received_shred_type(),
+                )
+            };
+        let prev_shred_id = ShredId::new(slot, prev_shred_index, prev_shred_type);


Similarly here:

let prev_merkle_root_meta = merkle_root_metas .get(&prev_erasure_set) .map(WorkingEntry::as_ref) .map(Cow::Borrowed) .or_else(|| { self.merkle_root_meta(prev_erasure_set) .unwrap() .map(Cow::Owned) }) .expect("merkle root meta must exist for erasure meta"); let prev_shred_id = ShredId::new( slot, prev_merkle_root_meta.first_received_shred_index(), prev_merkle_root_meta.first_received_shred_type(), );

behzadnouri · 2024-04-23T18:27:19Z

ledger/src/blockstore.rs

+            }
+            let (slot, _) = erasure_set.store_key();
+            // First coding shred from this erasure batch, check the forward merkle root chaining
+            if !self.has_duplicate_shreds_in_slot(slot) {


Personally rather avoid excessively indented code. This could be:

if self.has_duplicate_shreds_in_slot(slot) { continue; }

to avoid extra indentation.

behzadnouri · 2024-04-23T18:27:53Z

ledger/src/blockstore.rs

+            }
+            let (slot, _) = erasure_set.store_key();
+            // First shred from this erasure batch, check the backwards merkle root chaining
+            if !self.has_duplicate_shreds_in_slot(slot) {


similarly here:

if self.has_duplicate_shreds_in_slot(slot) { continue; }

…835) * blockstore: send duplicate proofs for chained merkle root conflicts * pr feedback: avoid deserialization, inline warn logs * remove allow[dead_code] after rebase * pr feedback: cow to simplify, avoid extra indentation (cherry picked from commit 175e152)

…licts (backport of #835) (#1009) blockstore: send duplicate proofs for chained merkle root conflicts (#835) * blockstore: send duplicate proofs for chained merkle root conflicts * pr feedback: avoid deserialization, inline warn logs * remove allow[dead_code] after rebase * pr feedback: cow to simplify, avoid extra indentation (cherry picked from commit 175e152) Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>

…licts (backport of anza-xyz#835) (anza-xyz#1009) blockstore: send duplicate proofs for chained merkle root conflicts (anza-xyz#835) * blockstore: send duplicate proofs for chained merkle root conflicts * pr feedback: avoid deserialization, inline warn logs * remove allow[dead_code] after rebase * pr feedback: cow to simplify, avoid extra indentation (cherry picked from commit 175e152) Co-authored-by: Ashwin Sekar <ashwin@anza.xyz>

AshwinSekar commented Apr 16, 2024

View reviewed changes

ledger/src/blockstore.rs Outdated Show resolved Hide resolved

AshwinSekar force-pushed the chain-dup-proof-altern branch from dff346d to 19a153d Compare April 16, 2024 05:35

AshwinSekar marked this pull request as ready for review April 16, 2024 05:36

AshwinSekar requested a review from behzadnouri April 16, 2024 05:36

AshwinSekar mentioned this pull request Apr 16, 2024

blockstore: send duplicate proofs for chained merkle root conflicts #102

Closed

behzadnouri reviewed Apr 18, 2024

View reviewed changes

AshwinSekar mentioned this pull request Apr 22, 2024

blockstore: store first received coding shred index in ErasureMeta #961

Merged

AshwinSekar force-pushed the chain-dup-proof-altern branch 2 times, most recently from 9364871 to 271cca6 Compare April 22, 2024 15:16

AshwinSekar added the feature-gate Pull Request adds or modifies a runtime feature gate label Apr 22, 2024

mergify bot mentioned this pull request Apr 22, 2024

v1.18: blockstore: store first received coding shred index in ErasureMeta (backport of #961) #981

Merged

AshwinSekar added 2 commits April 22, 2024 23:22

blockstore: send duplicate proofs for chained merkle root conflicts

ff444e1

pr feedback: avoid deserialization, inline warn logs

e17bd8b

AshwinSekar force-pushed the chain-dup-proof-altern branch from 271cca6 to e17bd8b Compare April 22, 2024 23:24

remove allow[dead_code] after rebase

48958a3

AshwinSekar added the v1.18 label Apr 23, 2024

behzadnouri previously approved these changes Apr 23, 2024

View reviewed changes

AshwinSekar dismissed behzadnouri’s stale review via 02ac1d1 April 23, 2024 20:12

pr feedback: cow to simplify, avoid extra indentation

4c79fca

AshwinSekar force-pushed the chain-dup-proof-altern branch from 02ac1d1 to 4c79fca Compare April 23, 2024 20:15

behzadnouri approved these changes Apr 23, 2024

View reviewed changes

AshwinSekar added the automerge automerge Merge this Pull Request automatically once CI passes label Apr 23, 2024

mergify bot merged commit 175e152 into anza-xyz:master Apr 23, 2024
49 checks passed

mergify bot mentioned this pull request Apr 23, 2024

v1.18: blockstore: send duplicate proofs for chained merkle root conflicts (backport of #835) #1009

Merged

AshwinSekar mentioned this pull request Apr 23, 2024

Feature Gate: Generate duplicate proofs for chained merkle root conflicts #1010

Open

AshwinSekar mentioned this pull request May 3, 2024

v1.18: Remove the chained merkle root check #1161

Closed

AshwinSekar mentioned this pull request May 14, 2024

blockstore: always send a coding shred for chained merkle root conflicts #1353

Merged

mergify bot mentioned this pull request Jun 4, 2024

v1.18: blockstore: always send a coding shred for chained merkle root conflicts (backport of #1353) #1594

Closed

behzadnouri mentioned this pull request Aug 2, 2024

rolls out chained Merkle shreds to ~5% of testnet #2389

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blockstore: send duplicate proofs for chained merkle root conflicts #835

blockstore: send duplicate proofs for chained merkle root conflicts #835

AshwinSekar commented Apr 16, 2024 •

edited

Loading

AshwinSekar Apr 16, 2024

codecov-commenter commented Apr 16, 2024 •

edited

Loading

behzadnouri left a comment

behzadnouri Apr 18, 2024

AshwinSekar Apr 22, 2024

behzadnouri Apr 18, 2024

behzadnouri Apr 18, 2024

behzadnouri Apr 18, 2024

behzadnouri Apr 18, 2024

mergify bot commented Apr 23, 2024

behzadnouri Apr 23, 2024

behzadnouri Apr 23, 2024

behzadnouri Apr 23, 2024

behzadnouri Apr 23, 2024

		/// Offset to the index of the first received coding shred
		first_received_coding_shred_offset: usize,

blockstore: send duplicate proofs for chained merkle root conflicts #835

blockstore: send duplicate proofs for chained merkle root conflicts #835

Conversation

AshwinSekar commented Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

codecov-commenter commented Apr 16, 2024 • edited Loading

Codecov Report

behzadnouri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Apr 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AshwinSekar commented Apr 16, 2024 •

edited

Loading

codecov-commenter commented Apr 16, 2024 •

edited

Loading