-
Notifications
You must be signed in to change notification settings - Fork 650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel preprocessing of blocks + transactions #1360
Merged
Merged
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
8999eac
Removed unused _unlinked_index
pmconrad 6849984
Fixed fork_db handling in pop_block
pmconrad 73d8b08
Added --revalidate-blockchain
pmconrad 34baaed
Use externally provided skip flags for replay
pmconrad bd8ed04
Removed skip_authority flag because its used interchangably with skip…
pmconrad 56dfa9c
Removed unused skip_validate flag introduced in #dca5c95 2016-01-04
pmconrad 1a78e1b
Add warning when revalidating with checkpoints
pmconrad cf2d6e7
Make distinction between skip_flag cases clearer
pmconrad d644f20
Prove that irrelevant signature detection depends on sorting order
pmconrad 0c4c133
Fixed typo
pmconrad e5ba271
Parallelize loading/saving object_database
pmconrad 033ddea
Introduce precomputable_transaction, and clearable_block in tests
pmconrad d2d8b29
Preprocess blocks + transactions in parallel, working version
pmconrad 9946bde
Get rid of possibly uninitialized local variable
pmconrad cb2244f
Changed push_transaction to accept precomputable_transaction instead …
pmconrad 883ec6a
Avoid one level of indirection on precompute block
pmconrad d032890
Make pubkey comparator generally available
pmconrad 2c01109
Address some review comments
pmconrad File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Parallelize loading/saving object_database
- Loading branch information
commit e5ba271c264c2c4173e12f37db9d0def1e068ab8
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule fc
updated
24 files
+3 −1 | CMakeLists.txt | |
+4 −4 | include/fc/asio.hpp | |
+0 −12 | include/fc/container/deque.hpp | |
+5 −0 | include/fc/crypto/sha1.hpp | |
+84 −51 | include/fc/static_variant.hpp | |
+3 −2 | include/fc/thread/future.hpp | |
+106 −0 | include/fc/thread/parallel.hpp | |
+2 −0 | include/fc/thread/task.hpp | |
+24 −4 | include/fc/thread/thread.hpp | |
+10 −11 | src/asio.cpp | |
+7 −1 | src/io/fstream.cpp | |
+2 −2 | src/stacktrace.cpp | |
+31 −0 | src/static_variant.cpp | |
+6 −1 | src/thread/context.hpp | |
+202 −0 | src/thread/parallel.cpp | |
+52 −51 | src/thread/thread.cpp | |
+22 −3 | src/thread/thread_d.hpp | |
+1 −0 | tests/CMakeLists.txt | |
+6 −6 | tests/io/stream_tests.cpp | |
+0 −4 | tests/io/tcp_test.cpp | |
+19 −7 | tests/run-parallel-tests.sh | |
+3 −3 | tests/stacktrace_test.cpp | |
+342 −0 | tests/thread/parallel_tests.cpp | |
+49 −0 | tests/variant_test.cpp |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IO isn't necessarily the bottleneck, e. g. when restarting a node the previously saved data might still be in system buffers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is
save
aka writing but not reading. When we're writing, usually the data has changed, even if it didn't change much, since we're writing to temporary (aka different) files, so the buffer doesn't help.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course disk buffer helps when writing.
On a system with a slow disk and the bare minimum of RAM you're right, parallelizing won't help - but it won't hurt either.
With sufficient buffer memory, or on a RAM-disk, parallel saving should be faster.