Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test][Misc] Fix libsecp256k1 init for unit test setup + Allow to specify blocks storage directory #2326

Merged

Conversation

furszy
Copy link

@furszy furszy commented Apr 20, 2021

Grouped two different, zero risk, modifications here to not have to create two/three small and straightforward PRs:

1) Unit Tests:

  • Fixing basic unit test setup issue, was not initializing the libsecp256k1 context. Thus why we have been forced to use the extended unit testing setup for almost every test case (which includes CConnman, scheduler, validation interface, etc.. that are not needed so often).
  • Refactor: BasicTestingSetup receiving the network in the constructor.

2) Back ported the ability to specify a custom directory for the blocks storage.

Since the actual block files taking up more and more space, it may be desirable to have them stored in a different location then the data directory (use case: SSD for chainstate, etc., HD for blocks).
This PR adds a -blocksdir option that allows one to keep the blockfiles external from the data directory (instead of creating symlinks).

furszy and others added 4 commits April 20, 2021 10:45
As the libsecp256k1 context was only initialized in the TestingSetup, we have been forced to use it when sometimes it is not necessary (TestingSetup is a complete setup that includes CConnman, scheduler, validation interface, etc).
@furszy furszy self-assigned this Apr 20, 2021
@furszy furszy changed the title [Test][Misc] Fix libsecp256k1 init for unit test setup + Allow to specify blocks storage directory [WIP][Test][Misc] Fix libsecp256k1 init for unit test setup + Allow to specify blocks storage directory Apr 21, 2021
@furszy furszy changed the title [WIP][Test][Misc] Fix libsecp256k1 init for unit test setup + Allow to specify blocks storage directory [Test][Misc] Fix libsecp256k1 init for unit test setup + Allow to specify blocks storage directory Apr 24, 2021
@furszy furszy added this to the 6.0.0 milestone Apr 25, 2021
Copy link

@random-zebra random-zebra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Couple minor nits/typos in the comments. Otherwise ACK.

test/functional/feature_blocksdir.py Outdated Show resolved Hide resolved
test/functional/feature_blocksdir.py Outdated Show resolved Hide resolved
jonasschnelli and others added 8 commits April 27, 2021 22:11
This commit attempts to clarify and correct the `-blocksdir` argument
description and default value. `-blocksdir` does not refer to the full
path to the actual `blocks` directory, but rather the root/parent
directory which contains the `blocks` directory. Accordingly, the
default value is `<datadir>` and not `<datadir>/blocks`. It also
attempts to clarify that only the `.dat` files containing block data are
impacted by `-blocksdir`, not the index files.
The blocks directory is net specific by definition.

Also this prevents the side effect of calling GetBlocksDir(false) in the
non-mainnet environment.
A new node should not create an unused `blocks` directory in the root of
the data directory when `-testnet` or `-regtest` is specified.
Rather than both.

Introduced in 386a6b6
This commit does not change behavior as GetBlocksDir() with unset
"-blocksdir" returns the same path.
Use case: TryCreateDirectory(GetDataDir() / "blocks" / "index") would
fail if the blocks directory was not explicitly created before.

The line that did so was in a weird location and could be removed as a
result.
@furszy furszy force-pushed the 2021_misc_unitTest_plus_blocksdir branch from 8b954c6 to 50e66c2 Compare April 28, 2021 01:12
@furszy
Copy link
Author

furszy commented Apr 28, 2021

updated, typos corrected.

@Fuzzbawls
Copy link
Collaborator

functionally good. might as well document the new option in the release notes template here so it doesn't need to be done later

@furszy
Copy link
Author

furszy commented Apr 29, 2021

Updated per feedback, added -blocksdir init option to the release-notes.

Fuzzbawls
Fuzzbawls previously approved these changes May 1, 2021
Copy link
Collaborator

@Fuzzbawls Fuzzbawls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like something went awry when adding to the release notes. minor issue though and can simply be cleaned up later.

otherwise, ACK ff767e1

doc/release-notes.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@Fuzzbawls Fuzzbawls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK b0e0d55

Copy link

@random-zebra random-zebra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK b0e0d55 and merging...

@random-zebra random-zebra merged commit edfaec6 into PIVX-Project:master May 1, 2021
random-zebra added a commit to random-zebra/PIVX that referenced this pull request May 5, 2021
furszy added a commit that referenced this pull request May 15, 2021
…tem.h

3e48fc1 Renamed and moved util.h/cpp files to util/system.h & util/system.cpp (furszy)
7954cef Style cleanup. (Jim Posen)
48801a9 Use common SetDataDir method to create temp directory in tests. (winder)
06464d3 flatfile: Unit tests for FlatFileSeq methods. (Jim Posen)
bbb9a90 scripted-diff: Rename CBlockDiskPos to FlatFilePos. (Jim Posen)
472857a Move CDiskBlockPos from chain to flatfile. (Jim Posen)
7fa47bb validation: Refactor file flush logic into FlatFileSeq. (Jim Posen)
bf09076 validation: Refactor block file pre-allocation into FlatFileSeq. (Jim Posen)
c813080 validation: Refactor OpenDiskFile into method on FlatFileSeq. (Jim Posen)
2619fe8 validation: Extract basic block file logic into FlatFileSeq class. (Jim Posen)
e65acf0 util: Move CheckDiskSpace to util. (Jim Posen)

Pull request description:

  This is part of the block logic refactoring and encapsulation work coming from upstream (work started in #2326 and #2328 due the possible blocks db corruption issue).
  Needed for two future works: (1) faster block validation using a new cache structure and (2) the future possible custom BIP157 (new sync protocol that supersedes BIP37 for light clients -- pending research needed after finishing v6.0 priorities --)

  Essentially, it's encapsulating the sequenced files open/read/write functionalities inside a new `flatfile` module,  and extending the unit test coverage to validate its correct behavior.

  Adapted the following PRs:

  * bitcoin#15118.
  * bitcoin#13145.
  * Plus added `util.h/cpp` files mv to `util/system.h` & `util/system.cpp`.

ACKs for top commit:
  random-zebra:
    utACK 3e48fc1

Tree-SHA512: c21ffb6ede054a279f9792c1cbe645b948078bd6bc607d32438f0601fd4df3650c0a34b349e46b4ea69b48f9e6c7bb18d258e139c6e1a47452ac9ea4f3bbee25
@furszy furszy deleted the 2021_misc_unitTest_plus_blocksdir branch November 29, 2022 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants