core/rawdb, cmd, ethdb, eth: implement freezer tail deletion #23954
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rjl493456442
requested review from
holiman,
karalabe and
zsfelfoldi
as code owners
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
e313ed9
to
9b08ce3
Compare
holiman
reviewed
Nov 24, 2021
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
3 times, most recently
from
91e6c0e
to
8619875
Compare
Closed
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
8619875
to
09afc60
Compare
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
2 times, most recently
from
2c29e6f
to
63ca13f
Compare
holiman
reviewed
Jan 10, 2022
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
faed9c6
to
8c7bb69
Compare
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
8c7bb69
to
2a1d49f
Compare
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
2a1d49f
to
ea04113
Compare
rjl493456442
force-pushed
the
freezer-tail-deletion
branch
from
ea04113
to
964ae61
Compare
|
We need to bump the database verison. |
rjl493456442
commented
Feb 16, 2022
|
Now the freezer is fully compatible with the old version Geth. It's unnecessary to bump the database version. |
sidhujag
pushed a commit
to syscoin/go-ethereum
that referenced
this pull request
Mar 11, 2022
…m#23954) * core/rawdb, cmd, ethdb, eth: implement freezer tail deletion * core/rawdb: address comments from martin and sina * core/rawdb: fixes cornercase in tail deletion * core/rawdb: separate metadata into a standalone file * core/rawdb: remove unused code * core/rawdb: add random test * core/rawdb: polish code * core/rawdb: fsync meta file before manipulating the index * core/rawdb: fix typo * core/rawdb: address comments
JacekGlen
pushed a commit
to JacekGlen/go-ethereum
that referenced
this pull request
May 26, 2022
…m#23954) * core/rawdb, cmd, ethdb, eth: implement freezer tail deletion * core/rawdb: address comments from martin and sina * core/rawdb: fixes cornercase in tail deletion * core/rawdb: separate metadata into a standalone file * core/rawdb: remove unused code * core/rawdb: add random test * core/rawdb: polish code * core/rawdb: fsync meta file before manipulating the index * core/rawdb: fix typo * core/rawdb: address comments
cp-wjhan
pushed a commit
to cp-yoonjin/go-wemix
that referenced
this pull request
Nov 16, 2022
…m#23954) * core/rawdb, cmd, ethdb, eth: implement freezer tail deletion * core/rawdb: address comments from martin and sina * core/rawdb: fixes cornercase in tail deletion * core/rawdb: separate metadata into a standalone file * core/rawdb: remove unused code * core/rawdb: add random test * core/rawdb: polish code * core/rawdb: fsync meta file before manipulating the index * core/rawdb: fix typo * core/rawdb: address comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the tail deletion of freezer.
File level deletion
The tail deletion is at file level by design. It means whenever data items at tail are deleted, they will not be removed from the file immediately. Instead they will be hidden until items in the tail data file are all deleted then the entire data file will be dropped.
Reasons
Performance
The reason for designing this lazy deletion mechanism is mainly for saving cost of updating index file. The lookup in freezer is using item id to calculate the relative position in index file, read the data position info from the index file and finally read the data from data file.
However in order to delete the tail items from the data file, all the remaining items in this data file must be shifted forward and all of them need to be re-indexed by modifying the corresponding position information in the index file. This action is super expensive. So deletion by file is a good choice in terms of avoiding re-index.
Atomicity
Another big reason for lazy deletion is it's easier to gain atomicity guarantee. The deletion procedure contains the mutation to index file and data file. These two mutations should be applied at the same time, or apply none of them.
In this design we only need to update the index file atomically. The de-referenced data file can be dropped any time, even after the crash. For sake of simplification, it's a big win.