Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to pass parameters via a file #441

Merged
merged 2 commits into from
Mar 15, 2018
Merged

Allow to pass parameters via a file #441

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d4e7f4
Allow to pass parameters via a file
danielmitterdorfer Mar 15, 2018
9e6f4fb
import json at the top
danielmitterdorfer Mar 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/command_line_reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Selects the track that Rally should run. By default the ``geonames`` track is ru

With this parameter you can inject variables into tracks. The supported variables depend on the track and you should check the track JSON file to see which variables can be provided.

It accepts a list of comma-separated key-value pairs. The key-value pairs have to be delimited by a colon.
It accepts a list of comma-separated key-value pairs or a JSON file name. The key-value pairs have to be delimited by a colon.

**Examples**:

Expand Down Expand Up @@ -106,6 +106,14 @@ When we run this track, we can override these defaults:

* ``--track-params="replica_count:1,shard_count:3"`` will set the number of replicas to 1 and the number of shards to 3.
* ``--track-params="replica_count:1"`` will just set the number of replicas to 1 and just keep the default value of 5 shards.
* ``--track-params="params.json"`` will read the track parameters from a JSON file (defined below)

Example JSON file::

{
"replica_count": 1,
"shard_count": 3
}

All track parameters are recorded for each metrics record in the metrics store. Also, when you run ``esrally list races``, it will show all track parameters::

Expand Down Expand Up @@ -167,15 +175,15 @@ Rally will configure Elasticsearch with 4GB of heap (``4gheap``) and enable Java
``car-params``
~~~~~~~~~~~~~~

Allows to override config variables of Elasticsearch.
Allows to override config variables of Elasticsearch. It accepts a list of comma-separated key-value pairs or a JSON file name. The key-value pairs have to be delimited by a colon.

**Example**

::

esrally --car="4gheap" --car-params="data_paths:'/opt/elasticsearch'"

The variables that are exposed depend on the `car's configuration <https://github.com/elastic/rally-teams/tree/master/cars>`__. In addition, Rally implements special handling for the variable ``data_paths`` (by default the values for variable is determined by Rally).
The variables that are exposed depend on the `car's configuration <https://github.com/elastic/rally-teams/tree/master/cars>`__. In addition, Rally implements special handling for the variable ``data_paths`` (by default the value for this variable is determined by Rally).


``elasticsearch-plugins``
Expand All @@ -192,7 +200,7 @@ In this example, Rally will install the ``analysis-icu`` plugin and the ``x-pack
``plugin-params``
~~~~~~~~~~~~~~~~~

Allows to override variables of Elasticsearch plugins.
Allows to override variables of Elasticsearch plugins. It accepts a list of comma-separated key-value pairs or a JSON file name. The key-value pairs have to be delimited by a colon.

Example::

Expand Down
15 changes: 12 additions & 3 deletions esrally/rally.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,15 @@ def dispatch_sub_command(cfg, sub_command):
return False


def to_dict(arg):
if io.has_extension(arg, ".json"):
import json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment: you probably have your reasons, but why not have this at the top? AFAIK pep8 is rather strict about this. I read some literature preferring from json import load for inside-function uses such as this one, if it must be done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to limit the scope of the import so we don't accidentally use this functionality somewhere else. I'm fine with putting it at the top. Mid-term it might make sense to split rally.py though.

with open(io.normalize_path(arg), mode="rt", encoding="utf-8") as f:
return json.load(f)
else:
return kv_to_map(csv_to_list(arg))


def csv_to_list(csv):
if csv is None:
return None
Expand Down Expand Up @@ -684,8 +693,8 @@ def main():
else:
cfg.add(config.Scope.applicationOverride, "mechanic", "repository.name", args.team_repository)
cfg.add(config.Scope.applicationOverride, "mechanic", "car.plugins", csv_to_list(args.elasticsearch_plugins))
cfg.add(config.Scope.applicationOverride, "mechanic", "car.params", kv_to_map(csv_to_list(args.car_params)))
cfg.add(config.Scope.applicationOverride, "mechanic", "plugin.params", kv_to_map(csv_to_list(args.plugin_params)))
cfg.add(config.Scope.applicationOverride, "mechanic", "car.params", to_dict(args.car_params))
cfg.add(config.Scope.applicationOverride, "mechanic", "plugin.params", to_dict(args.plugin_params))
if args.keep_cluster_running:
cfg.add(config.Scope.applicationOverride, "mechanic", "keep.running", True)
# force-preserve the cluster nodes.
Expand Down Expand Up @@ -715,7 +724,7 @@ def main():
chosen_track = args.track if args.track else "geonames"
cfg.add(config.Scope.applicationOverride, "track", "track.name", chosen_track)

cfg.add(config.Scope.applicationOverride, "track", "params", kv_to_map(csv_to_list(args.track_params)))
cfg.add(config.Scope.applicationOverride, "track", "params", to_dict(args.track_params))
cfg.add(config.Scope.applicationOverride, "track", "challenge.name", args.challenge)
cfg.add(config.Scope.applicationOverride, "track", "include.tasks", csv_to_list(args.include_tasks))
cfg.add(config.Scope.applicationOverride, "track", "test.mode.enabled", args.test_mode)
Expand Down