Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass request-params as is in supported operations #649

Merged
merged 6 commits into from
Feb 26, 2019
Merged

Pass request-params as is in supported operations #649

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
109f4b2
Pass request-params as is in supported operations
dliappis Feb 19, 2019
d6d1076
Fix rst subsection
dliappis Feb 19, 2019
2f0872a
Fix type in request params search runner
dliappis Feb 19, 2019
a0fe476
Address PR comment
dliappis Feb 19, 2019
70639ce
Address PR comment
dliappis Feb 19, 2019
b0d6c41
Address PR comment
dliappis Feb 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/migrate.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,45 @@
Migration Guide
===============

Migrating to Rally 1.1.0
------------------------

``request-params`` in operations are passed as is and not serialized
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

With Rally 1.1.0 any operations supporting the option ``request-params`` property will pass the structure as is without attempting to serialize values.
dliappis marked this conversation as resolved.
Show resolved Hide resolved
Until now, ``request-params`` relied on parameters being supported by the Elasticsearch Python client API calls. This means that for example boolean type parameters
should be specified as strings i.e. ``"true"`` or ``"false"`` rather than ``true/false``.

**Example**

Using ``create-index`` before ``1.1.0``::

{
"name": "create-all-indices",
"operation-type": "create-index",
"settings": {
"index.number_of_shards": 1
},
"request-params": {
"wait_for_active_shards": true
}
}

Using ``create-index`` starting with ``1.1.0``::

{
"name": "create-all-indices",
"operation-type": "create-index",
"settings": {
"index.number_of_shards": 1
},
"request-params": {
"wait_for_active_shards": "true"
}
}


Migrating to Rally 1.0.1
------------------------

Expand Down
34 changes: 19 additions & 15 deletions docs/track.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,11 @@ With the operation type ``search`` you can execute `request body searches <http:
* ``index`` (optional): An `index pattern <https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html>`_ that defines which indices should be targeted by this query. Only needed if the ``index`` section contains more than one index. Otherwise, Rally will automatically derive the index to use. If you have defined multiple indices and want to query all of them, just specify ``"index": "_all"``.
* ``type`` (optional): Defines the type within the specified index for this query. By default, no ``type`` will be used and the query will be performed across all types in the provided index. Also, types have been removed in Elasticsearch 7.0.0 so you must not specify this property if you want to benchmark Elasticsearch 7.0.0 or later.
* ``cache`` (optional): Whether to use the query request cache. By default, Rally will define no value thus the default depends on the benchmark candidate settings and Elasticsearch version.
* ``request-params`` (optional): A structure containing arbitrary request parameters. The supported parameters names are documented in the `Python ES client API docs <http://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.Elasticsearch.search>`_. Parameters that are implicitly set by Rally (e.g. `body` or `request_cache`) are not supported (i.e. you should not try to set them and if so expect unspecified behavior).
* ``request-params`` (optional): A structure containing arbitrary request parameters. The supported parameters names are documented in the `Search URI Request docs <https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html#_parameters_3>`_.

NOTE:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use the RST formatting approach for notes here? I.e.:

.. note::

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't elect to use .. note:: because I was unsure if it can be indented (the note applies for request-params only). It appears this is possible, so I addressed it in 70639ce

1. Parameters that are implicitly set by Rally (e.g. `body` or `request_cache`) are not supported (i.e. you should not try to set them and if so expect unspecified behavior).
2. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).
* ``body`` (mandatory): The query body.
* ``pages`` (optional): Number of pages to retrieve. If this parameter is present, a scroll query will be executed. If you want to retrieve all result pages, use the value "all".
* ``results-per-page`` (optional): Number of documents to retrieve per page for scroll queries.
Expand All @@ -386,7 +390,7 @@ Example::
},
"request-params": {
"_source_include": "some_field",
"analyze_wildcard": false
"analyze_wildcard": "false"
}
}

Expand Down Expand Up @@ -435,7 +439,7 @@ cluster-health

With the operation ``cluster-health`` you can execute the `cluster health API <https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html>`_. It supports the following properties:

* ``request-params`` (optional): A structure containing any request parameters that are allowed by the cluster health API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the cluster health API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).
* ``index`` (optional): The name of the index that should be used to check.

The ``cluster-health`` operation will check whether the expected cluster health and will report a failure if this is not the case. Use ``--on-error`` on the command line to control Rally's behavior in case of such failures.
Expand Down Expand Up @@ -471,13 +475,13 @@ With the operation ``create-index`` you can execute the `create index API <https
If you want it to create all indices that have been declared in the ``indices`` section you can specify the following properties:

* ``settings`` (optional): Allows to specify additional index settings that will be merged with the index settings specified in the body of the index in the ``indices`` section.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create>`__ that are allowed by the create index API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the create index API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

If you want it to create one specific index instead, you can specify the following properties:

* ``index`` (mandatory): One or more names of the indices that should be created. If only one index should be created, you can use a string otherwise this needs to be a list of strings.
* ``body`` (optional): The body for the create index API call.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create>`__ that are allowed by the create index API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the create index API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

**Examples**

Expand All @@ -490,7 +494,7 @@ The following snippet will create all indices that have been defined in the ``in
"index.number_of_shards": 1
},
"request-params": {
"wait_for_active_shards": true
"wait_for_active_shards": "true"
}
}

Expand Down Expand Up @@ -529,13 +533,13 @@ With the operation ``delete-index`` you can execute the `delete index API <https
If you want it to delete all indices that have been declared in the ``indices`` section, you can specify the following properties:

* ``only-if-exists`` (optional, defaults to ``true``): Defines whether an index should only be deleted if it exists.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.delete>`__ that are allowed by the delete index API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the delete index API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

If you want it to delete one specific index (pattern) instead, you can specify the following properties:

* ``index`` (mandatory): One or more names of the indices that should be deleted. If only one index should be deleted, you can use a string otherwise this needs to be a list of strings.
* ``only-if-exists`` (optional, defaults to ``true``): Defines whether an index should only be deleted if it exists.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.delete>`__ that are allowed by the delete index API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the delete index API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

**Examples**

Expand All @@ -555,8 +559,8 @@ With the following snippet we will delete all ``logs-*`` indices::
"only-if-exists": false,
"request-params": {
"expand_wildcards": "all",
"allow_no_indices": true,
"ignore_unavailable": true
"allow_no_indices": "true",
"ignore_unavailable": "true"
}
}

Expand All @@ -571,13 +575,13 @@ If you want it to create index templates that have been declared in the ``templa

* ``template`` (optional): If you specify a template name, only the template with this name will be created.
* ``settings`` (optional): Allows to specify additional settings that will be merged with the settings specified in the body of the index template in the ``templates`` section.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.put_template>`__ that are allowed by the create index template API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the create index template API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

If you want it to create one specific index instead, you can specify the following properties:

* ``template`` (mandatory): The name of the index template that should be created.
* ``body`` (mandatory): The body for the create index template API call.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.put_template>`__ that are allowed by the create index template API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the create index template API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters (see example below).

**Examples**

Expand All @@ -587,7 +591,7 @@ The following snippet will create all index templates that have been defined in
"name": "create-all-templates",
"operation-type": "create-index-template",
"request-params": {
"create": true
"create": "true"
}
}

Expand Down Expand Up @@ -625,15 +629,15 @@ With the operation ``delete-index-template`` you can execute the `delete index t
If you want it to delete all index templates that have been declared in the ``templates`` section, you can specify the following properties:

* ``only-if-exists`` (optional, defaults to ``true``): Defines whether an index template should only be deleted if it exists.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.delete_template>`__ that are allowed by the delete index template API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the delete index template API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters.

If you want it to delete one specific index template instead, you can specify the following properties:

* ``template`` (mandatory): The name of the index that should be deleted.
* ``only-if-exists`` (optional, defaults to ``true``): Defines whether the index template should only be deleted if it exists.
* ``delete-matching-indices`` (optional, defaults to ``false``): Whether to delete indices that match the index template's index pattern.
* ``index-pattern`` (mandatory iff ``delete-matching-indices`` is ``true``): Specifies the index pattern to delete.
* ``request-params`` (optional): A structure containing any `request parameters <https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.delete_template>`__ that are allowed by the delete index template API.
* ``request-params`` (optional): A structure containing any request parameters that are allowed by the delete index template API. Rally will not attempt to serialize the parameters and pass them as is. Always use "true" / "false" strings for boolean parameters.

**Examples**

Expand Down
25 changes: 9 additions & 16 deletions esrally/driver/runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -598,7 +598,7 @@ def request_body_query(self, es, params):
index=params.get("index", "_all"),
doc_type=params.get("type"),
body=mandatory(params, "body", self),
**request_params)
params=request_params)
hits = r["hits"]["total"]
if isinstance(hits, dict):
hits_total = hits["value"]
Expand Down Expand Up @@ -637,7 +637,7 @@ def scroll_query(self, es, params):
scroll="10s",
size=size,
request_cache=cache,
**request_params
params=request_params
)
# This should only happen if we concurrently create an index and start searching
self.scroll_id = r.get("_scroll_id", None)
Expand Down Expand Up @@ -727,9 +727,7 @@ def status(v):
# either the user has defined something or we're good with any count of relocating shards.
expected_relocating_shards = int(request_params.get("wait_for_relocating_shards", sys.maxsize))

# This would not work if the request parameter is not a proper method parameter for the ES client...
# result = es.cluster.health(**request_params)
result = es.transport.perform_request("GET", _make_path("_cluster", "health", index), params=request_params)
result = es.cluster.health(index=index, params=request_params)
cluster_status = result["status"]
relocating_shards = result["relocating_shards"]

Expand Down Expand Up @@ -783,12 +781,7 @@ def __call__(self, es, params):
indices = mandatory(params, "indices", self)
request_params = params.get("request-params", {})
for index, body in indices:
# We don't use es.indices.create() because it doesn't support params
# Ref: https://elasticsearch-py.readthedocs.io/en/master/api.html?highlight=indices%20create#elasticsearch.client.IndicesClient.create
es.transport.perform_request(method="PUT",
url="/{}".format(index),
body=body,
params=request_params)
es.indices.create(index=index, body=body, params=request_params)
return len(indices), "ops"

def __repr__(self, *args, **kwargs):
Expand All @@ -809,11 +802,11 @@ def __call__(self, es, params):

for index_name in indices:
if not only_if_exists:
es.indices.delete(index=index_name, **request_params)
es.indices.delete(index=index_name, params=request_params)
ops += 1
elif only_if_exists and es.indices.exists(index=index_name):
self.logger.info("Index [%s] already exists. Deleting it.", index_name)
es.indices.delete(index=index_name, **request_params)
es.indices.delete(index=index_name, params=request_params)
ops += 1

return ops, "ops"
Expand All @@ -833,7 +826,7 @@ def __call__(self, es, params):
for template, body in templates:
es.indices.put_template(name=template,
body=body,
**request_params)
params=request_params)
return len(templates), "ops"

def __repr__(self, *args, **kwargs):
Expand All @@ -853,11 +846,11 @@ def __call__(self, es, params):

for template_name, delete_matching_indices, index_pattern in template_names:
if not only_if_exists:
es.indices.delete_template(name=template_name, **request_params)
es.indices.delete_template(name=template_name, params=request_params)
ops_count += 1
elif only_if_exists and es.indices.exists_template(template_name):
self.logger.info("Index template [%s] already exists. Deleting it.", template_name)
es.indices.delete_template(name=template_name, **request_params)
es.indices.delete_template(name=template_name, params=request_params)
ops_count += 1
# ensure that we do not provide an empty index pattern by accident
if delete_matching_indices and index_pattern:
Expand Down
Loading