Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental download code #1627

Open
wants to merge 8 commits into
base: download-experiment
Choose a base branch
from

Conversation

ogenev
Copy link
Member

@ogenev ogenev commented Jan 9, 2025

Merging my experimental download code to a new branch.

The code loads the CSV file with block hashes from block 14,000,000 till The Merge (15,537,393) and starts to request from the network block bodies and receipts in a configurable number of batches. When all data from the current batch is downloaded. it proceeds to the next batch of blocks.

TO switch between find content with census and peer scoring and recursive find content queries without census, set the CENSUS const (true for census, false - no census).
A hacky OfferReport is used for peer scoring as a shortcut but can also represent a FIndContent failure.).

The CSV data set can be downloaded from here and needs to be placed in the trin-history/src folder.

To run the downloader, just start Trin with the history network enabled.

Notes:

  • running the script without a full view of the network and peer scoring is faster. (This may be due to a difference in the recursive find content and native find content implementations in Trin, or due to some inefficiencies in the downloader code. Needs more investigation.)
  • a reasonable BATCH_SIZE is 20-30 blocks per batch (40-60 content types), bumping this number higher than 40-50 blocks will cause Trin's uTP to stale.

@ogenev ogenev changed the title Download experimental code Experimental download code Jan 9, 2025
@ogenev ogenev force-pushed the findcontent-experiment branch from fd8def6 to 194e26c Compare January 9, 2025 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant