-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PH] Feature performance harness stage 2 development #473
Conversation
…read pool (--chain-threads).
…read pool (--net-threads).
…ad pool (--producer-threads).
[PH] Add support for thread configs
[PH] Update docs for new arguments.
Use namespace alias to simplify and make easier to read. Rename some variables for brevity.
[PH] add database map mode
[PH] log dir add tps target
Also renaming to use Ptb abbreviation in light of the name change.
Using PtbConfig and LoggingConfig objects simplifies test initializer params and encapsulates related config items into an object describing their use. It also allows easy reporting of test configuration in test report and will allow reporting to come for free if additional config arguments are added. Some renaming for efficiency and clarity.
Break out config and logging classes. Update documentation.
…thread pool size. Each plugin's option determines whether to calculate number of worker threads to use in the thread pool with options of: 'none', 'lmax', or 'full'. In 'none' mode, the default, no calculation will be attempted and default configured thread count will be used. In 'lmax' mode, thread count will incrementally be tested until the performance rate ceases to increase with the addition of subsequent threads. In 'full' mode thread count will incrementally be tested from 2..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in 'lmax' mode). Useful for graphing the full performance impact of each available thread. Added log directory to collect results of thread optimization calculations. Added option to opt out of running the tps test. Updated README for new options and to show the updates to the result reports.
…d on performance run calculations.
…figured to run tests.
This will greatly reduce work to bubble up arguments between scripts/modules. Also brings clarity and grouping of arguments by use with group titles and descriptions. Test Helper arguments can now be bubbled up to user scripts via direct access to the ArgumentParser. Performance Test Basic makes use of Test Helper arguments. Perfomance Test makes use of both Performance Test Basic and Test Helper arguments as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please address comment/question regarding Python 3.11.
Edit: this is resolved.
Break up some rather long lines of code.
Combine pieces of argument onto same line. Remove trailing whitespace.
Use variable to make this easier to read and follow.
Newer versions of Python, e.g. 3.11 require use of default_factory in dataclasses. Newer versions of python/numpy no longer require np predicate before float, and can use float directly.
Addressed: 045db81 |
Addressed: 045db81 |
Thanks! |
Moving some functions out of trx_generator since they don't need to be member functions. Addressing a couple missed const qualifiers on arguments.
Add long form for help message. Make sure launch_transaction_generators is added to the module. Remove executable bits from scripts internal to the module.
Since they are now being generated they will always be kept in sync with nodeos arguments and the validation test is no longer needed.
capture_output not added until python 3.7. Ubuntu 18 build still running on older version of python.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Before text=True (3.7) there was universal_newlines=True. Ubuntu 18 build still running on older version of python.
Major features
Develop performance tests to establish thread count recommendations (net, chain, producer) 510
Add options to calculate recommended producer, chain, and net worker thread pool size.
Each plugin's option determines whether to calculate number of worker threads to use in the thread pool with options of:
none
,lmax
, orfull
. Innone
mode, the default, no calculation will be attempted and the configured thread count will be used. Inlmax
mode, thread count will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of subsequent threads. Infull
mode thread count will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered inlmax
mode). Useful for graphing the full performance impact of each available thread.Expand Details Thread Count Recommendations
New Arguments
--calc-producer-threads {none,lmax,full}
Determines whether to calculate number of worker threads to use in producer thread pool ("none", "lmax", or "full").
In "none" mode, the default, no calculation will be attempted and the configured --producer-threads value will be used.
In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)
--calc-chain-threads {none,lmax,full}
Determines whether to calculate number of worker threads to use in chain thread pool ("none", "lmax", or "full").
In "none" mode, the default, no calculation will be attempted and the configured --chain-threads value will be used.
In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)
--calc-net-threads {none,lmax,full}
Determines whether to calculate number of worker threads to use in net thread pool ("none", "lmax", or "full").
In "none" mode, the default, no calculation will be attempted and the configured --net-threads value will be used.
In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)
Added option to opt out of running the tps test.
--skip-tps-test
Determines whether to skip the max TPS measurement tests (default: False)Updated thread configurations based on test recommendations
--producer-threads PRODUCER_THREADS
Number of worker threads in producer thread pool (default: 6)
--chain-threads CHAIN_THREADS
Number of worker threads in controller thread pool (default: 3)
Add Ability to Specify Contract for Performance Tests 598
Use EOSMechanics contract with Performance Test 713
Support configuring transaction type to send in trx generator through JSON transaction description 626
Expand Details JSON Transaction Description Support
JSON Transaction Description Support
Add the ability to configure the type of transaction to generate/send from a JSON transaction description (file, str, etc.)
Added functionality to transaction generator to take input of abi from file, transaction action type, and action data from a string or file json description and generate transactions matching that description.
Transaction Generator now takes additional arguments:
action-name
The action name applied to the provided action data inputaction-data
The path to the json action data file or json action data description string to useabi-file
The path to the contract abi file to use for the supplied transaction action dataTransaction Generator refactor base class
transaction_generator_base
base class.launch_transaction_generators.py
also now supports:action_name
The action name applied to the provided action data inputaction_data
The path to the json action data file or json action data description string to useabi_file
The path to the contract abi file to use for the supplied transaction action dataperformance_test_basic.py
now supports:--user-trx-data-file
Path to userTrxData.jsonOther Changes/Additions:
Rename handler_account to contract_owner_account for more accurate description.
Add integration test exercising the new functionality that builds transactions from abi, action type and action data. Makes use of the performance_test_basic with new argument --user-trx-data-file' and provided example
userTrxData.json
, but generates the transfer transactions from the abi, action name and action description in the test.Added support to
Cluster.py
to allow populating wallet and creating accounts with specific account names for testing purposes (in particular to use with the--user-trx-data-file
where a user would like to have foreknowledge about account names to use in action data).Transaction Generator support generating newaccount transactions 702
Expand Details New Account Transaction Support
Expand transaction generator to support new account trx
'ACCT_PER_TRX'
key word.Additional work:
Additional Work
New Configuration Items
--chain-threads CHAIN_THREADS
Number of worker threads in controller thread pool (default: 2) 469--net-threads NET_THREADS
Number of worker threads in net_plugin thread pool (default: 2) 469--producer-threads PRODUCER_THREADS
Number of worker threads in producer thread pool (default: 2) 469--database-map-mode {mapped,heap,locked}
481Database map mode ("mapped", "heap", or "locked").
In "mapped" mode database is memory mapped as a file.
In "heap" mode database is preloaded in to swappable memory and will use huge pages if available.
In "locked" mode database is preloaded, locked in to memory, and will use huge pages if available. (default: mapped)
--http-max-bytes-in-flight-mb HTTP_MAX_IN_FLIGHT_BYTES
Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited.
429 error response when exceeded. (default: -1) 740
--wasm-runtime RUNTIME
Override default WASM runtime ("eos-vm-jit", "eos-vm")
"eos-vm-jit" : A WebAssembly runtime that compiles WebAssembly code to native x86 code prior to
execution. "eos-vm" : A WebAssembly interpreter. (default: eos-vm-jit) 740
--contracts-console
print contract's output to console (default: False) 740--eos-vm-oc-cache-size-mb CACHE_SIZE_MiB
Maximum size (in MiB) of the EOS VM OC code cache (default: 1024) 740
--eos-vm-oc-compile-threads COMPILE_THREADS
Number of threads to use for EOS VM OC tier-up (default: 1) 740
--non-prods-eos-vm-oc-enable
Enable EOS VM OC tier-up runtime on non producer nodes (default: False) 740
--block-log-retain-blocks BLOCKS_TO_RETAIN
If set to greater than 0, periodically prune the block log to
store only configured number of most recent blocks. If set to 0, no blocks are be written to the block log;
block log file is removed after startup. (default: None) 740
--http-threads HTTP_THREADS
Number of worker threads in http thread pool (default: 2) 740
Override some
chainPlugin
andhttpPlugin
config options for performance test, but not exposed on cli 740maxTransactionTime
(max-transaction-time)abiSerializerMaxTimeMs
(abi-serializer-max-time-ms)httpMaxBytesInFlightMb
(http-max-bytes-in-flight-mb)httpMaxResponseTimeMs
(http-max-response-time-ms)maxClients
(max-clients)resourceMonitorNotShutdownOnThresholdExceeded
(resource-monitor-not-shutdown-on-threshold-exceeded)Cluster updates for config items 740
Previously hardcoded in cluster, now
max-transaction-time
,abi-serializer-max-time-ms
,p2p-max-nodes-per-host
, andmax-clients
can be dynamically configured and passed tonodeos
via cluster without causing failures.Architecture Rework
Make use of ArgumentParser parents and groups. 518
This will greatly reduce work to bubble up arguments between scripts/modules. Also brings clarity and grouping of arguments by use with group titles and descriptions.
Test Helper arguments can now be bubbled up to user scripts via direct access to the ArgumentParser. Performance Test Basic makes use of Test Helper arguments. Perfomance Test makes use of both Performance Test Basic and Test Helper arguments as well.
Provide
*PluginArgs
dataclasses to capture and validatenodeos
config options. 546Each
*PluginArgs
dataclass captures the command line argument and default for each option as specified innodeos
.New test
validate_nodeos_plugin_args.py
usesnodeos
's--help
command to parse out each plugin's available options and defaults to validate current status vs. the*PluginArgs
classes. Any addition, removal, update will cause a test failure with note of what has changed. This will allow easy maintenance going forward.Consequently, quietly defaulted
nodeos
arguments will now have representation in the performance harness test reports.Replace usage of os.path with Path and PurePath 571
Add TransactionGeneratorLauncher to TestHelper Shutdown 731
Fix potential duplicate transactions from transaction generators 757 & 7075447
Report Rework
Added logical cpu count to top level
performance_test.py
report.Add target tps number to test log dir name for ease of locating specific run. 484
currently log directories are timestamp only:
performance_test_basic/2022-11-16_22_26_48
proposing the addition of the tps target to the dir name to make finding a specific run easier:
performance_test_basic/2022-11-16_22_26_48-50000
performance_test_basic/2022-11-16_22_28_25-25000
Log Number of Forks and Forked Blocks 618
Log Dropped Blocks, Dropped Transactions, and Production Windows 633
Additional tracking of block time, and stores it and latency inside as CSV for later consumption. 647
Capture Full Command Line Argument String in Report 720
PRs included
469
475
481
484
510
518
520
546
571
598
610611
618
626
633
647
702
713
720
731
734
740
757
761
762
766
769
771
778