Nosqlbench testing configuration file using dataset #441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maheshrajamani
commented
May 31, 2023
nosqlbench/dataset.txt
Outdated
| @@ -0,0 +1,5 @@ | |||
| {"hash":"3576be49f989ac551af8f7e7b2aad0d4f9dd7b3882e7843839aa173d017cf7fe","ver":2,"vin_sz":1,"vout_sz":1,"size":220,"weight":880,"fee":6468,"relayed_by":"0.0.0.0","lock_time":0,"tx_index":8970844506661621,"double_spend":false,"time":1685054037,"block_index":null,"block_height":null,"inputs":[{"sequence":4294967295,"witness":"","script":"4730440220771152f9eac8397baf82af8711969bdffb4b84db116c3cdf90c15d02699dcbd30220527f2d88855566337933a2180db8ee5455c3da2558f2f79b37dbeb283fd0bbf4014104e62bff7b1af952b1bfd6b68af9e6367d69756807b69f8920a762e1b500d86d821e8d11d3bf9a8bcd88d7a97b6a6aaa945be8e15bd1111478f37c919f807be61f","index":0,"prev_out":{"addr":"1koboutKx17hEXb9mDrF4YjULkdE5faxr","n":0,"script":"76a9140848c944ba84c10a6cb754910a2e8311f49c00b888ac","spending_outpoints":[{"n":0,"tx_index":8970844506661621}],"spent":true,"tx_index":3507365639401646,"type":0,"value":384372}}],"out":[{"type":0,"spent":false,"value":377904,"spending_outpoints":[],"n":0,"tx_index":8970844506661621,"script":"001499c3c085b9b7e31909b4a8ae61a1acc8d63df647","addr":"bc1qn8puppdekl33jzd54zhxrgdvertrmaj8j36jsu"}]} | |||
There was a problem hiding this comment.
Just provided a sample json file for dataset testing.
There was a problem hiding this comment.
cool, maybe name it sample-dataset.txt
ivansenic
suggested changes
Jun 1, 2023
There was a problem hiding this comment.
I would think about adding an extra set of ops which are not bounded to id lookup.. For example, we could define $exists on a field that could be customized, and by default it would be data.. But then I can in the test setup to say data.whatever.smth and thus adapt the search per data set. What do you think?
Other than that it's good, few comments.. Will Request changes so that I can run it locally and test once you update..
nosqlbench/dataset.txt
Outdated
| @@ -0,0 +1,5 @@ | |||
| {"hash":"3576be49f989ac551af8f7e7b2aad0d4f9dd7b3882e7843839aa173d017cf7fe","ver":2,"vin_sz":1,"vout_sz":1,"size":220,"weight":880,"fee":6468,"relayed_by":"0.0.0.0","lock_time":0,"tx_index":8970844506661621,"double_spend":false,"time":1685054037,"block_index":null,"block_height":null,"inputs":[{"sequence":4294967295,"witness":"","script":"4730440220771152f9eac8397baf82af8711969bdffb4b84db116c3cdf90c15d02699dcbd30220527f2d88855566337933a2180db8ee5455c3da2558f2f79b37dbeb283fd0bbf4014104e62bff7b1af952b1bfd6b68af9e6367d69756807b69f8920a762e1b500d86d821e8d11d3bf9a8bcd88d7a97b6a6aaa945be8e15bd1111478f37c919f807be61f","index":0,"prev_out":{"addr":"1koboutKx17hEXb9mDrF4YjULkdE5faxr","n":0,"script":"76a9140848c944ba84c10a6cb754910a2e8311f49c00b888ac","spending_outpoints":[{"n":0,"tx_index":8970844506661621}],"spent":true,"tx_index":3507365639401646,"type":0,"value":384372}}],"out":[{"type":0,"spent":false,"value":377904,"spending_outpoints":[],"n":0,"tx_index":8970844506661621,"script":"001499c3c085b9b7e31909b4a8ae61a1acc8d63df647","addr":"bc1qn8puppdekl33jzd54zhxrgdvertrmaj8j36jsu"}]} | |||
There was a problem hiding this comment.
cool, maybe name it sample-dataset.txt
| description: >2 | ||
| This workload emulates CRUD operations for the Stargate Documents API. | ||
| It requires a data set file, where each line is a single JSON document to be used for writes and updates. | ||
| It requires a data set file (default dataset.txt), where each line is a single JSON document to be used for writes and updates. | ||
| Note that jsonapi_port should reflect the port where the Docs API is exposed (defaults to 8181). | ||
|
|
||
| scenarios: | ||
| default: | ||
| schema: run driver=http tags==block:schema threads==1 cycles==UNDEF | ||
| write: run driver=http tags==name:"write.*" cycles===TEMPLATE(write-cycles,TEMPLATE(docscount,10000000)) threads=auto errors=timer,warn |
There was a problem hiding this comment.
is 10M docs as default meaningful?
There was a problem hiding this comment.
Will make is to 10K similar to fallout, in case file has less records it tries to reload the same data with different id.
ivansenic
approved these changes
Jun 5, 2023
There was a problem hiding this comment.
I did not check the updates, but I did run it locally and it works like a charm.. No errors, all fine..
# HELP command_processor_process_seconds
# TYPE command_processor_process_seconds summary
command_processor_process_seconds_count{command="FindOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 28570.0
command_processor_process_seconds_sum{command="FindOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 1097.928781341
command_processor_process_seconds_count{command="InsertOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 10000.0
command_processor_process_seconds_sum{command="InsertOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 832.656052543
command_processor_process_seconds_count{command="DeleteCollectionCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 1.0
command_processor_process_seconds_sum{command="DeleteCollectionCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 0.015430923
command_processor_process_seconds_count{command="FindCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 42855.0
command_processor_process_seconds_sum{command="FindCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 1990.367078935
command_processor_process_seconds_count{command="DeleteOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 14285.0
command_processor_process_seconds_sum{command="DeleteOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 854.258366268
command_processor_process_seconds_count{command="CreateNamespaceCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 1.0
command_processor_process_seconds_sum{command="CreateNamespaceCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 0.388465388
command_processor_process_seconds_count{command="CreateCollectionCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 1.0
command_processor_process_seconds_sum{command="CreateCollectionCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 2.066756688
command_processor_process_seconds_count{command="UpdateOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 14285.0
command_processor_process_seconds_sum{command="UpdateOneCommand",error="false",error_class="NA",error_code="NA",module="sgv2-jsonapi",tenant="unknown",} 881.17464271
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
No sql bench testing configuration to use data from a dataset file
Checklist