* improve for sharding #6958
* improve for sharding #6958
Conversation
Release 1.6.0
- net_plugin security fixes - Additional checktime calls to limit cpu usage - Limit memory usage in producer_plugin Co-Authored-By: Kevin Heifner <heifnerk@objectcomputing.com> Co-Authored-By: Matt Witherspoon <32485495+spoonincode@users.noreply.github.com> Co-authored-by: Kayan <taokayan13@gmail.com>
Consolidated Security Fixes for 1.6.1
Bump version to 1.6.1
Merge 1.6.1 master
The latest homebrew code balks at something in the old 1.67 package file. Fix the package file and store it locally in our repo for now. We try and pin the boost version because boost upgrades invalidate nodeos data files.
Some users (including myself) were seeing llvm@4 unpacked to 4.0.1_1 instead of 4.0.1. Stuff unpacked to the Cellar directory appears to be a kind of implementation detail — /usr/local/opt is the proper place to reference here.
Fix boost 1.67 brew install - 1.6.x
Set proper directory for baked in macOS LLVM_DIR - 1.6.x
Add buildkite step to gather brew files for automatic update
…_16x add noninteractive option for build scripts - 1.6.x
unlinking eveything we install makes no sense -- it means things like cmake aren't in the path any longer like the script expects. So don't do that any more. Unfortuately this old script requires that gettext be force linked. So implement that behavior explictly for now
Don't unlink what we install via brew - 1.6.x
Make sure python-devel is installed for amazon linux builds - 1.6.x
limit assert 1.6.x
Update version to 1.6.2
Merge 1.6.2
When a shutdown signal is handled the sig_set is 1) canceled and 2) destroyed. This means that signals are no longer handled by boost::asio and revert back to default behavior. If someone accidentally hits ctrl-c a second time during a long shutdown they run the risk of leaving the database dirty as that second ctrl-c will be insta-kill. This patch changes the behavior and keeps the sig_set active forever. This means that even after the first handled async_wait() on the sig_set (that starts an appbase quit), additional signals in the set are effectively “blocked” (not posix blocked, but blocked in the sense they are consumed and queued by the sig_set that is not being async_wait()ed on)
…n_17x appbase: Block (queue) exit signals during shutdown - 1.7
there is no tag or branch named "1.6.2", I think it should be "v1.6.2"?
Wrong tag/branch name
- Fix small memory leak in net_plugin. - Add additional deadline checks to transaction authorization.
Signature cpu usage - v1.6.x
Signature cpu usage - v1.7.x
Consolidated Security Fixes for 1.6.3
…mnibus Consolidated Security Fixes for 1.7.0-rc2
Bump version to 1.6.3
Bump version to 1.7.0-rc2
Merge 1.6.3
…avail_ram.py long running test
…ntract Add back integration_test contract; fixes nodeos_under_min_avail_ram_lr_test
Somewhere along the line nodeos picked up a dependency on libicuuc
add libicu as dependency for .deb packages
Bump version to 1.7.0
Merge release 1.7.0 into master
| // 1. enableSharding database (default to EOS) | ||
| // 2. shardCollection: blocks, action_traces, transaction_traces, especially action_traces | ||
| // 3. Use compound index with shard key (default to _id), to improve query performance. | ||
|
|
There was a problem hiding this comment.
Are you saying you would like something added to the mongo_db_plugin for 1 and 2 OR that you merely suggest users should enableSharding on there database and collections?
There was a problem hiding this comment.
I suggest users should do 1 and 2 on production environment.
There was a problem hiding this comment.
Can you update the comment to make that clearer.
| blocks.create_index( bsoncxx::from_json( R"xxx({ "block_num" : 1 })xxx" )); | ||
| blocks.create_index( bsoncxx::from_json( R"xxx({ "block_id" : 1 })xxx" )); | ||
| blocks.create_index( bsoncxx::from_json( R"xxx({ "block_num" : 1, "_id" : 1 })xxx" )); | ||
| blocks.create_index( bsoncxx::from_json( R"xxx({ "block_id" : 1, "_id" : 1 })xxx" )); |
There was a problem hiding this comment.
Do you have any feel for how much larger the indexes are when _id is added?
There was a problem hiding this comment.
use EOS
db.action_traces.stats()
shows:
"indexSizes" : {
"_id_" : 18581565440,
"trx_id_1" : 16938258432,
"block_num_1" : 4512980992,
"block_num_1__id_1" : 21147549696
block_num_1__id_1 is 16,634,568,704 bytes larger than block_num_1, close to "_id_" : 18,581,565,440.
There was a problem hiding this comment.
I wonder if the plugin should even create any indexes and let that be left to the user. However, I'm inclined to accept this PR if you rebase to develop, remove the try-catch changes (or provide some article/test/proof it makes a difference), and update the comment.
Thanks.
| if( !store_transactions ) { | ||
| return; | ||
| } | ||
|
|
||
| try { |
There was a problem hiding this comment.
Why move these statements out of try/catch? Do you see faster performance?
There was a problem hiding this comment.
This plugin itself is CPU-bound. if-return statements are never throw, so it's not necessary in try-block which may have performance costs. (It depends on the compiler)
There was a problem hiding this comment.
I would be surprised if that made any difference but would be interested if you could point me to anything that demonstrates that.
There was a problem hiding this comment.
The difference is tiny. It's more a coding style issue. I'll remove the try-catch changes later.
Change Description
improve for mongodb sharding