Retry Bulk Request If Bulk Request Item Fails #4488
Labels
Comments
criminosis
added a commit
to criminosis/janusgraph
that referenced
this issue
May 30, 2024
Closes JanusGraph#4488 Signed-off-by: Allan Clements <criminosis@gmail.com>
Merged
9 tasks
criminosis
added a commit
to criminosis/janusgraph
that referenced
this issue
Jun 12, 2024
Retry bulk request if all errors are configured retry error codes Only retry bulk request items that failed Implemented bulk retry test & implemented splitting up a bulk request if large enough Added exception if a bulk request item is too large to send, even as a single item Use HttpStatus.SC_NOT_FOUND instead of 404 literal Added pseudo test to facilitate manual observation of chunking of bulk writes Signed-off-by: Allan Clements <criminosis@gmail.com>
criminosis
added a commit
to criminosis/janusgraph
that referenced
this issue
Jun 12, 2024
Retry bulk request if all errors are configured retry error codes Only retry bulk request items that failed Implemented bulk retry test & implemented splitting up a bulk request if large enough Added exception if a bulk request item is too large to send, even as a single item Use HttpStatus.SC_NOT_FOUND instead of 404 literal Added pseudo test to facilitate manual observation of chunking of bulk writes Added test asserting silent failure of write to ES due to chunk size limit causing the vertex to omitted from the ES mixed index Signed-off-by: Allan Clements <criminosis@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature:
Follow up to #4408. Since implementing that feature I've noticed bulk requests do not get retried. In particular I periodically will have
circuit_breaking_exceptionerror messages mentioned by the thrown IOException here.It appears the bulk API doesn't participate as expected with regards to
ResponseException.It appears it can "successfully" return a Bulk Response, that contains items within it that failed. This PR applies the retry logic originally added in #4408 if all the failures within the bulk response set returned error codes that are configured for retry.
Circuit Breaker Exceptions in particular are coded on the server side as being another type of 429 Too Many Requests that would be appropriate for back pressure to be applied. Operationally some circuit breaker errors (like the parent circuit breaker) are OOM preventative measures to the calling client just saying your request right now may cause an OOM, try again later.
Describe a specific use case for the feature:
This feature would allow bulk request item failures to be reactive to backpressure via error codes.
The text was updated successfully, but these errors were encountered: