Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document bulk behavior with multiple clients/documents #1666

Merged
merged 2 commits into from
Feb 13, 2023
Merged

Document bulk behavior with multiple clients/documents #1666

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
542722f
Document bulk behavior with multiple clients/documents
pquentin Feb 7, 2023
a310785
Update docs/track.rst
pquentin Feb 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/track.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,6 +758,12 @@ Properties
* ``detailed-results`` (optional, defaults to ``false``): Records more detailed meta-data for bulk requests. As it analyzes the corresponding bulk response in more detail, this might incur additional overhead which can skew measurement results. See the section below for the meta-data that are returned. This property must be set to ``true`` for individual bulk request failures to be logged by Rally.
* ``timeout`` (optional, defaults to ``1m``): Defines the `time period that Elasticsearch will wait per action <https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-query-params>`_ until it has finished processing the following operations: automatic index creation, dynamic mapping updates, waiting for active shards.

With multiple ``clients``, Rally will split each document using as many splits as there are ``clients``. This ensures that the bulk index operations are efficiently parallelized but has the drawback that the ingestion is not done in the order of each document. For example, if ``clients`` is set to 2, one client will index the document starting from the beginning, while the other will index starting from the middle.

Additionally, if there are multiple documents or corpora, Rally will try to index all documents in parallel in two ways:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there are multiple documents or corpora

Isn't this the same thing?

A track can only have one corpora section but I guess you aren't referring to the section here, but rather the plural of corpus, essentially >1 files with documents for Rally to index?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not the same thing. You can have multiple corpora, and multiple documents in each corpora. You can see this clearly when looking at the JSON structure of say geonames:

  "corpora": [
    {
      "name": "geonames",
      "base-url": "https://rally-tracks.elastic.co/geonames",
      "documents": [
        {
          "source-file": "documents-2.json.bz2",
          "document-count": 11396503,
          "compressed-bytes": 265208777,
          "uncompressed-bytes": 3547613828
        }
      ]
    }
  ],

I know the elastic/logs track takes advantage of it: https://github.com/elastic/rally-tracks/blob/7df06b34daa50fe4d583d4bfbaea9336ff8cd278/elastic/logs/track.json#L394-L642.

So maybe I should update the doc you linked to as well!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have multiple corpora, and multiple documents in each corpora.

🤦 ofc I totally forgot about that. I think we are good with the docs as you have them then.


1. Each client will have a different starting point. For example, in a track with 2 corpora and 5 clients, clients 1, 3, and 5 will start with the first corpus and clients 2 and 4 will start with the second corpus.
2. Each client is assigned to multiple documents. Client 1 will start with the first split of the first document of the first corpus. Then it will move on to the first split of the first document of the second corpus, and so on.

The image below shows how Rally behaves with a ``recency`` set to 0.5. Internally, Rally uses the blue function for its calculations but to understand the behavior we will focus on red function (which is just the inverse). Suppose we have already generated ids from 1 to 100 and we are about to simulate an id conflict. Rally will randomly choose a value on the y-axis, e.g. 0.8 which is mapped to 0.1 on the x-axis. This means that in 80% of all cases, Rally will choose an id within the most recent 10%, i.e. between 90 and 100. With 20% probability the id will be between 1 and 89. The closer ``recency`` gets to zero, the "flatter" the red curve gets and the more likely Rally will choose less recent ids.

Expand Down