Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Make use of allow_partial_search_results on search requests of data extractors #40793

Closed
dimitris-athanasiou opened this issue Apr 3, 2019 · 1 comment · Fixed by #55537
Closed

[ML] Make use of allow_partial_search_results on search requests of data extractors #40793

dimitris-athanasiou opened this issue Apr 3, 2019 · 1 comment · Fixed by #55537
Assignees
@benwtrent
Labels
:ml Machine learning >refactoring

Comments

@dimitris-athanasiou
Copy link
Contributor

dimitris-athanasiou commented Apr 3, 2019 •
edited
Loading

When datafeed data extractors run searches they expect no errors on the search responses. Currently, we check for these errors manually. However, there is a feature that allows searches with partial results to fail with an exception instead of responding. We should revisit and simplify data extractors by setting allow_partial_search_results to false in the relevant search requests. Note, an additional benefit doing this is that when the search fails any scroll context gets cleared automatically, reducing the chance of data extractors leaking scroll contexts (see #40772).
@dimitris-athanasiou dimitris-athanasiou self-assigned this Apr 3, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@benwtrent benwtrent mentioned this issue Apr 21, 2020
Merged
benwtrent added a commit that referenced this issue Apr 22, 2020
@benwtrent
[ML] explicitly disallow partial results in datafeed extractors (#55537)
5074f2c 
Instead of doing our own checks against REST status, shard counts, and shard failures, this commit changes all our extractor search requests to set `.setAllowPartialSearchResults(false)`.

- Scrolls are automatically cleared when a search failure occurs with `.setAllowPartialSearchResults(false)` set.
- Code error handling is simplified

closes #40793
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Apr 22, 2020
@benwtrent
[ML] explicitly disallow partial results in datafeed extractors (elas…
d3e2051 
…tic#55537)

Instead of doing our own checks against REST status, shard counts, and shard failures, this commit changes all our extractor search requests to set `.setAllowPartialSearchResults(false)`.

- Scrolls are automatically cleared when a search failure occurs with `.setAllowPartialSearchResults(false)` set.
- Code error handling is simplified

closes elastic#40793
benwtrent added a commit that referenced this issue Apr 22, 2020
@benwtrent
[ML] explicitly disallow partial results in datafeed extractors (#55537…
7c81cd7 
…) (#55585)

Instead of doing our own checks against REST status, shard counts, and shard failures, this commit changes all our extractor search requests to set `.setAllowPartialSearchResults(false)`.

- Scrolls are automatically cleared when a search failure occurs with `.setAllowPartialSearchResults(false)` set.
- Code error handling is simplified

closes #40793
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benwtrent benwtrent

Labels
:ml Machine learning >refactoring
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] explicitly disallow partial results in datafeed extractors benwtrent/elasticsearch
3 participants
@benwtrent @dimitris-athanasiou @elasticmachine