Error can be a string, and shards not always present in response.

[j-bennet]

When there are bulk indexing errors, rally throws an exception Cannot race. Error in load generator, instead of finishing successfully and reporting indexing errors as part of error_rate. This fixes the problem.

Traceback from rally log:

2018-12-01 02:34:42,357 ActorAddr-(T|:57054)/PID:36237 esrally.driver.driver ERROR Could not execute schedule
Traceback (most recent call last):

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 996, in __call__
    total_ops, total_ops_unit, request_meta_data = execute_single(runner, self.es, params, self.abort_on_error)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 1028, in execute_single
    return_value = runner(es, params)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 117, in __call__
    return self.runnable(es['default'], *args[1:])

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 363, in __call__
    stats = self.detailed_stats(params, bulk_size, response) if detailed_results else self.simple_stats(bulk_size, response)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 439, in simple_stats
    self.extract_error_details(error_details, data)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 452, in extract_error_details
    if data.get("error") and data["error"].get("reason"):

AttributeError: 'str' object has no attribute 'get'

2018-12-01 02:34:42,636 ActorAddr-(T|:57050)/PID:36235 esrally.driver.driver ERROR Cannot execute runner [user-defined context-manager enabled runner for [bulk-index]]; most likely due to missing parameters.
Traceback (most recent call last):

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 1028, in execute_single
    return_value = runner(es, params)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 117, in __call__
    return self.runnable(es['default'], *args[1:])

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 363, in __call__
    stats = self.detailed_stats(params, bulk_size, response) if detailed_results else self.simple_stats(bulk_size, response)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 437, in simple_stats
    if data["status"] > 299 or data["_shards"]["failed"] > 0:

KeyError: '_shards'

2018-12-01 02:34:42,637 ActorAddr-(T|:57050)/PID:36235 esrally.driver.driver ERROR Could not execute schedule
Traceback (most recent call last):

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 1028, in execute_single
    return_value = runner(es, params)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 117, in __call__
    return self.runnable(es['default'], *args[1:])

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 363, in __call__
    stats = self.detailed_stats(params, bulk_size, response) if detailed_results else self.simple_stats(bulk_size, response)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/runner.py", line 437, in simple_stats
    if data["status"] > 299 or data["_shards"]["failed"] > 0:

KeyError: '_shards'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 996, in __call__
    total_ops, total_ops_unit, request_meta_data = execute_single(runner, self.es, params, self.abort_on_error)

  File "/Users/irina/.pyenv/versions/3.6.5/envs/benchmarks/lib/python3.6/site-packages/esrally/driver/driver.py", line 1059, in execute_single
    raise exceptions.SystemSetupError(msg)

esrally.exceptions.SystemSetupError: ("Cannot execute [user-defined context-manager enabled runner for [bulk-index]]. Provided parameters are: ['name', 'operation-type', 'corpora', 'bulk-size', 'indices', 'include-in-reporting', 'index', 'type', 'action-metadata-present', 'body', 'bulk-id']. Error: ['_shards'].", None)

[dliappis]

Thank you for this PR!

Seeing that this is something we are already doing in the detailed stats (

rally/esrally/driver/runner.py

Line 413 in abf1623

if data["status"] > 299 or ("_shards" in data and data["_shards"]["failed"] > 0):

), this makes sense.

I left one request to add a test.

[j-bennet]

Test added.

[dliappis]

LGTM thank you very much for this.