This repository has been archived by the owner on Nov 6, 2020. It is now read-only.
Disable parallel verification and skip verifiying already imported txs. #8834
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tomusdrw
added
A0-pleasereview 🤓
ascjones
reviewed
Jun 7, 2018
| .map(|transaction| { | ||
| if self.pool.read().find(&transaction.hash()).is_some() { | ||
| bail!(transaction::Error::AlreadyImported) | ||
| } |
There was a problem hiding this comment.
Was the primary cause of the high CPU usage the parallel verification or verifying already imported txs. Or a combination of both?
There was a problem hiding this comment.
So it seems like a combination of even more things, this brings the load down, but not 100% to the level of the one of 1.10. I'm still figuring out other culprits.
folsen
approved these changes
Jun 8, 2018
| @@ -179,8 +178,14 @@ impl TransactionQueue { | |||
|
|
|||
| let verifier = verifier::Verifier::new(client, options, self.insertion_id.clone()); | |||
| let results = transactions | |||
| .into_par_iter() | |||
| .map(|transaction| verifier.verify_transaction(transaction)) | |||
| .into_iter() | |||
There was a problem hiding this comment.
@tomusdrw Did you ever look at the size of the work for this and how par_iter() distributes it? Like if it's work-stealing it might actually cause more overhead to parallelise if the work-pieces are small enough (In some rayon examples for instance he shows a 0.95x speedup (so it's much worse) on a 4 core machine). It might be worth testing with rayons Chunks iterator to chunk it up into larger work-pieces for each CPU to tackle.
ascjones
approved these changes
Jun 8, 2018
5chdn
added
A8-looksgood 🦄
dvdplm
added a commit
that referenced
this pull request
Jun 11, 2018
* master: Fix subcrate test compile (#8862) network-devp2p: downgrade logging to debug, add target (#8784) Clearing up a comment about the prefix for signing (#8828) Disable parallel verification and skip verifiying already imported txs. (#8834) devp2p: Move UDP socket handling from Discovery to Host. (#8790) Fixed AuthorityRound deadlock on shutdown, closes #8088 (#8803) Specify critical release flag per network (#8821) Fix `deadlock_detection` feature branch compilation (#8824) Use system allocator when profiling memory (#8831) added from and to to Receipt (#8756)
andresilva
pushed a commit
that referenced
this pull request
Jun 18, 2018
…s. (#8834) * Reject transactions that are already in pool without verifying them. * Avoid verifying already imported transactions.
5chdn
pushed a commit
that referenced
this pull request
Jun 19, 2018
* `duration_ns: u64 -> duration: Duration` (#8457) * duration_ns: u64 -> duration: Duration * format on millis {:.2} -> {} * Keep all enacted blocks notify in order (#8524) * Keep all enacted blocks notify in order * Collect is unnecessary * Update ChainNotify to use ChainRouteType * Fix all ethcore fn defs * Wrap the type within ChainRoute * Fix private-tx and sync api * Fix secret_store API * Fix updater API * Fix rpc api * Fix informant api * Eagerly cache enacted/retracted and remove contain_enacted/retracted * Fix indent * tests: should use full expr form for struct constructor * Use into_enacted_retracted to further avoid copy * typo: not a function * rpc/tests: ChainRoute -> ChainRoute::new * Handle removed logs in filter changes and add geth compatibility field (#8796) * Add removed geth compatibility field in log * Fix mocked tests * Add field block hash in PollFilter * Store last block hash info for log filters * Implement canon route * Use canon logs for fetching reorg logs Light client removed logs fetching is disabled. It looks expensive. * Make sure removed flag is set * Address grumbles * Fixed AuthorityRound deadlock on shutdown, closes #8088 (#8803) * CI: Fix docker tags (#8822) * scripts: enable docker builds for beta and stable * scripts: docker latest should be beta not master * scripts: docker latest is master * ethcore: fix ancient block error msg handling (#8832) * Disable parallel verification and skip verifiying already imported txs. (#8834) * Reject transactions that are already in pool without verifying them. * Avoid verifying already imported transactions. * Fix concurrent access to signer queue (#8854) * Fix concurrent access to signer queue * Put request back to the queue if confirmation failed * typo: fix docs and rename functions to be more specific `request_notify` does not need to be public, and it's renamed to `notify_result`. `notify` is renamed to `notify_message`. * Change trace info "Transaction" -> "Request" * Don't allocate in expect_valid_rlp unless necessary (#8867) * don't allocate via format! in case there's no error * fix test? * fixed ipc leak, closes #8774 (#8876) * Add new ovh bootnodes and fix port for foundation bootnode 3.2 (#8886) * Add new ovh bootnodes and fix port for foundation bootnode 3.2 * Remove old bootnodes. * Remove duplicate 1118980bf48b0a3640bdba04e0fe78b1add18e1cd99bf22d53daac1fd9972ad650df52176e7c7d89d1114cfef2bc23a2959aa54998a46afcf7d91809f0855082 * Block 0 is valid in queries (#8891) Early exit for block nr 0 leads to spurious error about pruning: `…your node is running with state pruning…`. Fixes #7547, #8762 * Add ETC Cooperative-run load balanced parity node (#8892) * Minor fix in chain supplier and light provider (#8906) * fix chain supplier increment * fix light provider block_headers * Check whether we need resealing in miner and unwrap has_account in account_provider (#8853) * Remove unused Result wrap in has_account * Check whether we need to reseal for external transactions * Fix reference to has_account interface * typo: missing ) * Refactor duplicates to prepare_and_update_sealing * Fix build * Allow disabling local-by-default for transactions with new config entry (#8882) * Add tx_queue_allow_unknown_local config option - Previous commit messages: dispatcher checks if we have the sender account Add `tx_queue_allow_unknown_local` to MinerOptions Add `tx_queue_allow_unknown_local` to config fix order in MinerOptions to match Configuration add cli flag for tx_queue_allow_unknown_local Update refs to `tx_queue_allow_unknown_local` Add tx_queue_allow_unknown_local to config test revert changes to dispatcher Move tx_queue_allow_unknown_local to `import_own_transaction` Fix var name if statement should return the values derp de derp derp derp semicolons Reset dispatch file to how it was before fix compile issues + change from FLAG to ARG add test and use `into` import MinerOptions, clone the secret Fix tests? Compiler/linter issues fixed Fix linter msg - case of constants IT LIVES refactor to omit yucky explict return update comments Fix based on diff AccountProvider.has_account method * Refactor flag name + don't change import_own_tx behaviour fix arg name Note: force commit to try and get gitlab tests working again 😠 * Add fn to TestMinerService * Avoid race condition from trusted sources - refactor the miner tests a bit to cut down on code reuse - add `trusted` param to dispatch_transaction and import_claimed_local_transaction Add param to `import_claimed_local_transaction` Fix fn sig in tests
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related: #8696