Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to specify an indices stats condition #925

Merged

Conversation

danielmitterdorfer
Copy link
Member

With this commit we expand the existing indices-stats operation and
allow to pass a condition based on a path and an expected value. This
has a variety of use cases:

  • Verify that the number of documents in an index matches the expected
    number of documents
  • Wait until an operation has finished (e.g. no ongoing segment merges)
    by specifying additional retry properties.

With this commit we expand the existing `indices-stats` operation and
allow to pass a condition based on a path and an expected value. This
has a variety of use cases:

* Verify that the number of documents in an index matches the expected
number of documents
* Wait until an operation has finished (e.g. no ongoing segment merges)
by specifying additional retry properties.
@danielmitterdorfer danielmitterdorfer added enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc. labels Mar 3, 2020
@danielmitterdorfer danielmitterdorfer added this to the 1.5.0 milestone Mar 3, 2020
@danielmitterdorfer danielmitterdorfer self-assigned this Mar 3, 2020
Copy link
Contributor

@dliappis dliappis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very useful enhancement (and thanks for the debug logging!)

I left a question, but this this is LGTM already.

return return_value
else:
self.logger.debug("%s has returned with an error: %s.", repr(self.delegate), return_value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if it'd be handy to log this more eagerly than only on debug, for troubleshooting purposes and giving a more complete picture of what's going on without having to switch to debug.

Do you think it'd become very chatty?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO it could indeed become a bit chatty because it would add a log entry every time the condition is not yet met.

@danielmitterdorfer danielmitterdorfer merged commit a395a44 into elastic:master Mar 3, 2020
@danielmitterdorfer danielmitterdorfer deleted the indices-stats branch March 3, 2020 09:40
@danielmitterdorfer
Copy link
Member Author

Thanks for your review! :)

danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 3, 2020
With this commit we change all tracks to increase the request timeout of the
`force-merge` task to ensure the force merge finishes before we move on. We also
include another task that waits until all ongoing segment merges finish. Older
Rally versions will entirely rely on the increased request timeout for force
merges but newer Rally versions (every version including elastic/rally#925) will
evaluate the condition and wait accordingly until the system is quiet.

Relates elastic/rally#925
danielmitterdorfer added a commit to danielmitterdorfer/rally that referenced this pull request Mar 4, 2020
With this commit we make the index-stats operation more lenient so it
can check conditions for paths that don't (yet) exist. One use case for
this is to wait until a certain index shows up in the stats.

Relates elastic#925
danielmitterdorfer added a commit that referenced this pull request Mar 5, 2020
With this commit we make the index-stats operation more lenient so it
can check conditions for paths that don't (yet) exist. One use case for
this is to wait until a certain index shows up in the stats.

Relates #925
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants