-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document bulk behavior with multiple clients/documents #1666
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this adding this. Basically LGTM, left a few clarification questions.
@@ -758,6 +758,12 @@ Properties | |||
* ``detailed-results`` (optional, defaults to ``false``): Records more detailed meta-data for bulk requests. As it analyzes the corresponding bulk response in more detail, this might incur additional overhead which can skew measurement results. See the section below for the meta-data that are returned. This property must be set to ``true`` for individual bulk request failures to be logged by Rally. | |||
* ``timeout`` (optional, defaults to ``1m``): Defines the `time period that Elasticsearch will wait per action <https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-query-params>`_ until it has finished processing the following operations: automatic index creation, dynamic mapping updates, waiting for active shards. | |||
|
|||
With multiple ``clients``, Rally will split each document from ``corpora`` using as many splits as there are ``clients``. This ensures that the bulk index operations are efficiently parallelized but has the drawback that the ingestion is not done in the order of each document. For example, if ``clients`` is set to 2, one client will index the document starting from the beginning, while the other will index starting from the middle. | |||
|
|||
Additionally, if there are multiple documents or corpora, Rally will try to index all documents in parallel in two ways: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if there are multiple documents or corpora
Isn't this the same thing?
A track can only have one corpora section but I guess you aren't referring to the section here, but rather the plural of corpus, essentially >1 files with documents for Rally to index?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not the same thing. You can have multiple corpora, and multiple documents in each corpora. You can see this clearly when looking at the JSON structure of say geonames:
"corpora": [
{
"name": "geonames",
"base-url": "https://rally-tracks.elastic.co/geonames",
"documents": [
{
"source-file": "documents-2.json.bz2",
"document-count": 11396503,
"compressed-bytes": 265208777,
"uncompressed-bytes": 3547613828
}
]
}
],
I know the elastic/logs track takes advantage of it: https://github.com/elastic/rally-tracks/blob/7df06b34daa50fe4d583d4bfbaea9336ff8cd278/elastic/logs/track.json#L394-L642.
So maybe I should update the doc you linked to as well!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can have multiple corpora, and multiple documents in each corpora.
🤦 ofc I totally forgot about that. I think we are good with the docs as you have them then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is LGTM from me, based on the latest discussion. Further improvements welcome, but it already is an incremental improvements from the current docs and effectively describe this hard to grasp concept.
As suggested by @dliappis in elastic/rally-tracks#378 (comment).