This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Block log util#6884 #6913
Merged
Block log util#6884 #6913
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spoonincode
suggested changes
Mar 11, 2019
programs/eosio-blocklog/main.cpp
Outdated
| uint64_t lastIndBufLen= indFileLen & (fposListLen-1); //fposListLen is a power of 2 so -1 creates low bits all 1 | ||
| if (!lastIndBufLen) //will write integral number of bufLen and lastIndBufLen one time to index file | ||
| lastIndBufLen= bufLen; | ||
| uint64_t indPos= lseek(fout,indFileLen-lastIndBufLen,ios::beg); |
There was a problem hiding this comment.
Looks like this should be SEEK_SET not ios::beg
programs/eosio-blocklog/main.cpp
Outdated
| @@ -145,9 +149,12 @@ void blocklog::set_program_options(options_description& cli) | |||
| "Do not pretty print the output. Useful if piping to jq to improve performance.") | |||
| ("as-json-array", bpo::bool_switch(&as_json_array)->default_value(false), | |||
| "Print out json blocks wrapped in json array (otherwise the output is free-standing json objects).") | |||
| ("make-index", bpo::bool_switch(&make_index)->default_value(false), | |||
| "Create blocks.index from blocks.log. Must give 'blocks-dir'. Give 'output-file' relative to blocks-dir (default is blocks.index).") | |||
| ("trim-block-log", bpo::bool_switch(&make_index)->default_value(false), | |||
heifner
suggested changes
Mar 12, 2019
programs/eosio-blocklog/main.cpp
Outdated
| //struct used by truncBlockLog() and makeIndex() to read first 18 bytes of a block from blocks.log | ||
| struct __attribute__((packed)) BlockStart { //first 18 bytes of each block | ||
| block_timestamp_type timestamp; | ||
| account_name prodname; |
There was a problem hiding this comment.
Should be producer to match block_header.
Add static_assert( sizeof(block_header) == whatever_it_currently_is, "Update when block_header updated");
There was a problem hiding this comment.
Can we really not just use eosio::chain::block_header here? Does the deserialization markedly slow things down or something?
|
The struct eosio::chain::block_header is not packed so it does not match
the layout on disk which is packed. However I had to get rid of my packed
struct for another reason. I could not find any way to indicate [[packed]]
that was recognized by all compilers. Now I simply have this code:
constexpr int blknum_offset{14}; *//offset from start of block to 4 byte
block number *//this offset is valid for version 1 and 2 block logs,
version is checked before using blknum_offset
I seek to block_start + blknum_offset before reading the block number. When
I am done with the code for chopping off the start of blocks.log (in
addition to chopping off the tail) I will push all the changes.
…On Wed, Mar 13, 2019 at 12:40 PM Matt Witherspoon ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In programs/eosio-blocklog/main.cpp
<#6913 (comment)>:
> @@ -169,6 +176,176 @@ void blocklog::initialize(const variables_map& options) {
}
+//struct used by truncBlockLog() and makeIndex() to read first 18 bytes of a block from blocks.log
+struct __attribute__((packed)) BlockStart { //first 18 bytes of each block
+ block_timestamp_type timestamp;
+ account_name prodname;
Can we really not just use eosio::chain::block_header here? Does the
deserialization markedly slow things down or something?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#6913 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ApReUg4phV-JANotT9xjoRslj6Cp-cT6ks5vWSoBgaJpZM4bprOP>
.
--
Steve Strand
|
brianjohnson5972
force-pushed
the
blockLogUtil#6884
branch
from
6dbb07a
to
e5a1a22
Compare
heifner
suggested changes
Apr 29, 2019
programs/eosio-blocklog/main.cpp
Outdated
| bool make_index; | ||
| bool trim_log; | ||
| bool smoke_test; | ||
| bool help; |
There was a problem hiding this comment.
Initialize uint32_t and bool members.
programs/eosio-blocklog/main.cpp
Outdated
| FILE* ind_in; //C style files for reading blocks.log and blocks.index | ||
| //we use low level file IO because it is distinctly faster than C++ filebuf or iostream | ||
| uint64_t index_pos; //filepos in blocks.index for block n, +8 for block n+1 | ||
| uint64_t fpos0, fpos1; //filepos in blocks.log for block n and block n+1 |
There was a problem hiding this comment.
Initialize all these data members.
heifner
approved these changes
May 2, 2019
|
Should this be included in binary packages? Seems like it probably should. But I guess we can address that in a separate pr |
spoonincode
approved these changes
May 2, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added two new options to program eosio-blocklog:
--trim-block-log --last 'blkNum'
This truncates the files blocks.log and blocks.index so last block is 'blkNum'.
Runtime is < 1 second. Nodeos trim block log option can take over 1 hour.
--make-index
This creates blocks.index from blocks.log. Can specify output file if do not want to
overwrite existing blocks.index. Runtime for a recent test with 42e6 blocks in blocks.log
was 80 seconds. Nodeos took 90 minutes to create the same blocks.index file.
I also added BOOST_AUTO_TEST_CASE(reverse_endian_tests) to unittests/misc_tests.cpp. I wrote a slightly smaller and faster implementation of endian_reverse_u64 and endian_reverse_u32 and initially updated fc/bitutil.hpp with this code and added the test to go with it. In the end I decided to not change the endian_reverse code because it was being done the most common way and future compilers might recognize what the intention was and provide highly optimized code. So I removed my slightly faster implementation (for one compiler at one point in time) but left the test in place.
In the future it might be a good idea to rename eosio-blocklog to eosio-nodeos-utils so additional new functions can be added.