Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mitigate archival node data size growth #6119

Closed
bowenwang1996 opened this issue Jan 19, 2022 · 2 comments
Closed

Mitigate archival node data size growth #6119

bowenwang1996 opened this issue Jan 19, 2022 · 2 comments
Assignees
@posvyatokum
Labels
A-storage Area: storage and databases C-enhancement Category: An issue proposing an enhancement or a PR with one. Node Node team P-high Priority: High T-node Team: issues relevant to the node experience team

Comments

@bowenwang1996
Copy link
Collaborator

bowenwang1996 commented Jan 19, 2022 •
edited by exalate-issue-sync bot
Loading

Context can be found [here](https://near.zulipchat.com/#narrow/stream/308695-nearinc.2Fprivate/topic/archival.20node.20data.20size). TLDR: archival node data usage grows at a very fast pace and since we require node operators to use SSDs to operate an archival node, it creates a considerable amount of financial burden for them, which is highly undesirable and unsustainable. Some ideas to mitigate the growth include the following:

  • Move most of the archival node data usage to HDD and only use SSD for the head of the chain so that an archival node can keep up with the network. This could be implemented through garbage collection where instead of removing data, we write archival nodes' data to a separate storage.
  • Do not store data that can be reconstructed for archival nodes. For example, partial encoded chunks can be reconstructed from full chunks and we do not need to store them for archival nodes (behind what is needed for block processing). This alone can potentially save us 1/3 of the storage needed. When a request comes in for old partial encoded chunks, we can compute them on the fly.
  • (Longer-term) Create data snapshots periodically and do not store full history on archival nodes. The snapshot can be stored in a decentralized storage and its metadata can be posted on-chain. We can develop a separate service that serves historical data by routing its query to the relevant snapshot (effective shard historical data by epoch)
@bowenwang1996 bowenwang1996 added C-enhancement Category: An issue proposing an enhancement or a PR with one. A-storage Area: storage and databases T-node Team: issues relevant to the node experience team labels Jan 19, 2022
@bowenwang1996
Copy link
Collaborator Author

cc @mina86 @posvyatokum

mina86 added a commit to mina86/rust-rocksdb that referenced this issue Jan 21, 2022
@mina86
Support configuring bottom-most compression level
b26c986 
Weirdly, RocksDB C interface does not have a way to just set the
`enabled` flag of the bottommost compression options so I’ve decided
to replicate that interface in Rust as well.  As far as I understand,
this unfortunately means that to set bottommost compression one needs
to set the type and then also use one of the other methods for
configuring the compression.

Issue: rust-rocksdb#260
Issue: near/nearcore#6119
@mina86 mina86 mentioned this issue Jan 21, 2022
Merged
@bowenwang1996 bowenwang1996 added the P-high Priority: High label Jan 24, 2022
@mina86 mina86 mentioned this issue Mar 17, 2022
Merged
near-bulldozer bot pushed a commit that referenced this issue Mar 21, 2022
@mina86
neard: add recompress-storage command (#6447)
2030c75 
Add recompress-storage command which reads all the data from storage
and saves it in a new location effectively recompressing all the SST
files.  This is meant as a temporary tool to speed up migration to
zstd compression.  Without it, especially on archival nodes,
recompressing data may take a long time as most SST files can stay
untouched for epochs.

Issue: #6119
mina86 added a commit to mina86/nearcore that referenced this issue Mar 21, 2022
@mina86
neard: document ‘recompress-storage’ command more
6460603 
Issue: near#6119
@mina86 mina86 mentioned this issue Mar 21, 2022
Merged
mina86 added a commit to mina86/nearcore that referenced this issue Mar 22, 2022
@mina86
tools: add recompress.sh tool for gcing and recompressing database
b7700e0 
Issue: near#6119
@mina86 mina86 mentioned this issue Mar 22, 2022
Closed
near-bulldozer bot pushed a commit that referenced this issue Mar 22, 2022
@mina86
neard: document ‘recompress-storage’ command more (#6469)
a83db11 
Issue: #6119
mina86 added a commit to mina86/nearcore that referenced this issue Mar 22, 2022
@mina86
neard: recompress-storage: add option to clear out columns
913cd51 
Add --clean-partial-chunks and --clear-trie-changes options to clear out
the two respective columns.  Data in ColPartialChunks can be recomputed
and data in ColTrieChanges is only used by non-archival nodes and can be
deleted when running archival node.

Issue: near#6119
mina86 added a commit to mina86/nearcore that referenced this issue Mar 22, 2022
@mina86
neard: recompress-storage: add option to clear out columns
76dc225 
Add --clean-partial-chunks and --clear-trie-changes options to clear out
the two respective columns.  Data in ColPartialChunks can be recomputed
and data in ColTrieChanges is only used by non-archival nodes and can be
deleted when running archival node.

Issue: near#6119
Issue: near#6242
Issue: near#6250
@mina86 mina86 mentioned this issue Mar 23, 2022
Merged
mina86 added a commit to mina86/nearcore that referenced this issue Mar 23, 2022
@mina86
neard: use u64 counters in recompress-storage to avoid overflows
0a9223c 
Issue: near#6119
@mina86 mina86 mentioned this issue Mar 23, 2022
Merged
near-bulldozer bot pushed a commit that referenced this issue Mar 23, 2022
@mina86
neard: use u64 counters in recompress-storage to avoid overflows (#6478)
04f580a 
Issue: #6119
near-bulldozer bot pushed a commit that referenced this issue Apr 7, 2022
@mina86
neard: recompress_storage: clean out unnecessary columns (#6477)
da7a465 
When recompressing database of an archival node, skip
ColPartialChunks, ColInvalidChunks and ColTrieChanges columns which
can be safely deleted.  Data in the first one can be reconstructed
from ColChunks, ColInvalidChunks is only needed at head and the last
is never read by archival nodes.

Mostly for testing, if someone wants to keep those columns,
offer --keep-partial-chunks, --keep-invalid-chunks
and --keep-trie-changes switches.  They are always on when dealing
with non-archival node.

Issue: #6119
Issue: #6242
Issue: #6250
mina86 added a commit to mina86/nearcore that referenced this issue Apr 7, 2022
@mina86
neard: add recompress-storage command (near#6447)
e7c4ca0 
This is commit 2030c75 upstream.

Add recompress-storage command which reads all the data from storage
and saves it in a new location effectively recompressing all the SST
files.  This is meant as a temporary tool to speed up migration to
zstd compression.  Without it, especially on archival nodes,
recompressing data may take a long time as most SST files can stay
untouched for epochs.

Issue: near#6119
mina86 added a commit to mina86/nearcore that referenced this issue Apr 7, 2022
@mina86
neard: document ‘recompress-storage’ command more (near#6469)
d4a87e2 
This is commit a83db11 upstream.

Issue: near#6119
mina86 added a commit to mina86/nearcore that referenced this issue Apr 7, 2022
@mina86
neard: use u64 counters in recompress-storage to avoid overflows (nea…
cb611db 
…r#6478)

This is commit 04f580a upstream.

Issue: near#6119
mina86 added a commit to mina86/nearcore that referenced this issue Apr 7, 2022
@mina86
neard: recompress_storage: clean out unnecessary columns (near#6477)
5646135 
This is commit da7a465 upstream.

When recompressing database of an archival node, skip
ColPartialChunks, ColInvalidChunks and ColTrieChanges columns which
can be safely deleted.  Data in the first one can be reconstructed
from ColChunks, ColInvalidChunks is only needed at head and the last
is never read by archival nodes.

Mostly for testing, if someone wants to keep those columns,
offer --keep-partial-chunks, --keep-invalid-chunks
and --keep-trie-changes switches.  They are always on when dealing
with non-archival node.

Issue: near#6119
Issue: near#6242
Issue: near#6250
mina86 added a commit to mina86/nearcore that referenced this issue Apr 8, 2022
@mina86
neard: enable target:"recompress" info logs by default
08b9686 
The target is used by recompress_storage.  Initially the command used
no target but then target was added which caused the log lines to be
ignored by default.

Issue: near#6119
@mina86 mina86 mentioned this issue Apr 8, 2022
Merged
mina86 added a commit to mina86/nearcore that referenced this issue Apr 8, 2022
@mina86
recompress_storage: fix handling of rc columns
43bf732 
Store::iter strips reference count from reference counted columns and
writing returned value to a new database leads to corrupted data since
the count is no longer present.  The correct way to deal with raw
values saved in the database is to use Store::iter_without_rc_logic.
Switch recompress_storage to do that.

Issue: near#6119
@mina86 mina86 mentioned this issue Apr 8, 2022
Merged
near-bulldozer bot pushed a commit that referenced this issue Apr 8, 2022
@mina86
recompress_storage: fix handling of rc columns (#6565)
640c7e0 
Store::iter strips reference count from reference counted columns and
writing returned value to a new database leads to corrupted data since
the count is no longer present.  The correct way to deal with raw
values saved in the database is to use Store::iter_without_rc_logic.
Switch recompress_storage to do that.

Issue: #6119
near-bulldozer bot pushed a commit that referenced this issue Apr 11, 2022
@mina86
neard: enable target:"recompress" info logs by default (#6557)
48a1eb4 
The target is used by recompress_storage.  Initially the command used
no target but then target was added which caused the log lines to be
ignored by default.

Issue: #6119
mina86 added a commit that referenced this issue Apr 13, 2022
@mina86
pytest: add end-to-end recompress_storage sanity test
f128fad 
Introduce a test which verifies that recompress_storage command does
not corrupt the database and that nodes can continue working after
it’s used.

Issue: #6119
mina86 added a commit that referenced this issue Apr 13, 2022
@mina86
pytest: add end-to-end recompress_storage sanity test
1f9fc93 
Introduce a test which verifies that recompress_storage command does
not corrupt the database and that nodes can continue working after
it’s used.

Issue: #6119
@mina86 mina86 mentioned this issue Apr 13, 2022
Merged
near-bulldozer bot pushed a commit that referenced this issue Apr 16, 2022
@mina86
pytest: add end-to-end recompress-storage sanity test (#6601)
ad413b8 
Introduce a test which verifies that recompress-storage command
doesn’t corrupt the database and that nodes can continue working
after it’s used.

Issue: #6119
@mina86 mina86 mentioned this issue Apr 20, 2022
Merged
mina86 added a commit to near/node-docs that referenced this issue Apr 20, 2022
@mina86
Add ‘Recompressing archival node storage’ page (#21)
d80d192 
Issue: near/nearcore#6119
@exalate-issue-sync exalate-issue-sync bot added T-nodeX and removed T-node Team: issues relevant to the node experience team labels Jun 28, 2022
@matklad matklad added T-node Team: issues relevant to the node experience team and removed T-nodeX labels Aug 4, 2022
@mina86 mina86 assigned posvyatokum and unassigned mina86 Oct 27, 2022
@exalate-issue-sync
Copy link

Michał Nazarewicz commented:

Last point is now tracked in https://pagodaplatform.atlassian.net/browse/ND-162

@gmilescu gmilescu added the Node Node team label Oct 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@posvyatokum posvyatokum

Labels
A-storage Area: storage and databases C-enhancement Category: An issue proposing an enhancement or a PR with one. Node Node team P-high Priority: High T-node Team: issues relevant to the node experience team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mina86 @gmilescu @matklad @posvyatokum @bowenwang1996