Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database metadata refactor #6555

Merged
merged 25 commits into from
Feb 26, 2024
Merged

Conversation

fab-10
Copy link
Contributor

@fab-10 fab-10 commented Feb 9, 2024

PR description

Besu storage could have different formats, currently Forest or Bonsai, and each format can have different versions. We used to have only one identifier for these two properties, but this is not optimal, for clarity and does not play well with the fact that the storage is implemented via plugins.

Besu should only know about the format, and tell the plugin which is the required format, then the version of the format is something internal to the plugin itself.

So I moved the DataStorageFormat to the plugin API project, and this is the reason for the many file changed, most of them are just changes to reflect this move.
Then inside the RocksDB plugin I introduced the new class VersionedStorageFormat to managed the versioning of RocksDB databases.

The database metadata file, reflects this change, having an entry for the format and one for the version, for example, the default Bonsai storage will have this metadata (note the versioning of the metadata itself v2 object, so we can introduce a v3 if needed in future):

{
  "v2": {
    "format": "BONSAI",
    "version": 2
  }
}

To correct the missed introduction of the new version, when the variables storage format was introduced, now for both Forest and Bonsai, a new version has been introduced, the version 2 for each format, being 1 the version pre variables storage.

At startup the existing metadata, that we call v1 format, is automatically translated to the new format.

Check have been added to detect not managed (up|down)grades, so we can inform the user of the right steps, if the process in not automated.

It is possible to downgrade, running the subcommand storage revert-metadata v2-to-v1 before installing the previous binaries.

Test to perform:

  • Vanilla new installation, verify that the default format and version is used and written in DATABASE_METADATA.json
  • New installation with custom format selection, verify that the selected format and version is used and written in DATABASE_METADATA.json
  • Upgrade existing installation >=23.4.4, verify that DATABASE_METADATA.json has been translated to the new format
  • Upgrade existing installation >=23.4.4 then downgrade, verify that after the downgrade Besu refuses to start since the metadata is not recognized
  • Upgrade existing installation >=23.4.4 then revert metadata, then downgrade, verify that after the downgrade Besu starts correctly
  • Upgrade existing installation <23.4.4, verify that variables storage migration is performed and DATABASE_METADATA.json has the new format
  • Upgrade existing installation <23.4.4 then downgrade, verify that after the downgrade Besu refuses to start since the metadata is not recognized
  • Upgrade existing installation <23.4.4 first revert variables-storage, then revert metadata, then downgrade, verify that after the downgrade Besu starts correctly

Fixed Issue(s)

fixes #6408

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
#	besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java
#	besu/src/test/java/org/hyperledger/besu/controller/MergeBesuControllerBuilderTest.java
#	besu/src/test/java/org/hyperledger/besu/controller/QbftBesuControllerBuilderTest.java
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/storage/StorageProvider.java
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/KeyValueSegmentIdentifier.java
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/KeyValueStorageProvider.java
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/storage/BonsaiWorldStateKeyValueStorage.java
#	ethereum/core/src/test-support/java/org/hyperledger/besu/ethereum/core/InMemoryKeyValueStorageProvider.java
#	ethereum/core/src/test/java/org/hyperledger/besu/ethereum/trie/bonsai/AbstractIsolationTests.java
#	ethereum/eth/src/jmh/java/org/hyperledger/besu/ethereum/eth/sync/worldstate/WorldStateDownloaderBenchmark.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/fastsync/worldstate/FastWorldDownloadStateTest.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/fastsync/worldstate/PersistDataStepTest.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/snapsync/PersistDataStepTest.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/snapsync/SnapWorldDownloadStateTest.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/snapsync/TaskGenerator.java
#	ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/snapsync/request/heal/StorageTrieNodeHealingRequestTest.java
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/KeyValueSegmentIdentifier.java
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
#	ethereum/core/src/main/java/org/hyperledger/besu/ethereum/chain/GenesisState.java
#	ethereum/core/src/test-support/java/org/hyperledger/besu/ethereum/core/InMemoryKeyValueStorageProvider.java
#	ethereum/core/src/test/java/org/hyperledger/besu/ethereum/chain/GenesisStateTest.java
#	ethereum/core/src/test/java/org/hyperledger/besu/ethereum/trie/bonsai/AbstractIsolationTests.java
#	plugin-api/src/main/java/org/hyperledger/besu/plugin/services/storage/DataStorageFormat.java
#	plugins/rocksdb/src/main/java/org/hyperledger/besu/plugin/services/storage/rocksdb/RocksDBKeyValueStorageFactory.java
#	plugins/rocksdb/src/test/java/org/hyperledger/besu/plugin/services/storage/rocksdb/RocksDBKeyValueStorageFactoryTest.java
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Copy link

github-actions bot commented Feb 9, 2024

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.
  • I thought about the changelog and included a changelog update if required.
  • If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests
  • I thought about running CI.
  • If I did not run CI, I ran as much locally as possible before pushing.

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
@fab-10 fab-10 marked this pull request as ready for review February 13, 2024 19:08
@fab-10 fab-10 added the doc-change-required Indicates an issue or PR that requires doc to be updated label Feb 13, 2024
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
CHANGELOG.md Outdated Show resolved Hide resolved
fab-10 and others added 2 commits February 15, 2024 11:06
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
# Conflicts:
#	besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java
#	plugin-api/build.gradle
#	plugin-api/src/main/java/org/hyperledger/besu/plugin/services/BesuConfiguration.java
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Copy link
Contributor

@gfukushima gfukushima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

@jframe jframe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# Conflicts:
#	CHANGELOG.md
#	plugin-api/build.gradle
@fab-10 fab-10 merged commit e0bedff into hyperledger:main Feb 26, 2024
55 checks passed
@fab-10 fab-10 deleted the database-metadata-refactor branch February 26, 2024 12:58
@alexandratran alexandratran removed the doc-change-required Indicates an issue or PR that requires doc to be updated label Feb 27, 2024
suraneti pushed a commit to suraneti/besu that referenced this pull request Feb 28, 2024
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
amsmota pushed a commit to Citi/besu that referenced this pull request Apr 16, 2024
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: amsmota <antonio.mota@citi.com>
amsmota pushed a commit to Citi/besu that referenced this pull request Apr 16, 2024
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: amsmota <antonio.mota@citi.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DATABASE_METADATA.json enhancement
4 participants