Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix retries that timeout hanging forever. #10855

Merged
merged 2 commits into from
Jan 31, 2024

Conversation

larry-safran
Copy link
Contributor

Fixes #10336

Thanks to @turchenkoalex for identifying the underlying problem and fix.

@larry-safran larry-safran requested review from temawi and sergiitk and removed request for temawi January 27, 2024 03:01
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add a test to cover this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out that due to mocks it is difficult.

@larry-safran larry-safran added the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Jan 29, 2024
@grpc-kokoro grpc-kokoro removed kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary labels Jan 29, 2024
@larry-safran larry-safran added the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Jan 30, 2024
@grpc-kokoro grpc-kokoro removed the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Jan 30, 2024
boolean cancelled = retryFuture.cancel(false);
if (cancelled) {
inFlightSubStreams.decrementAndGet();
}
}
if (hedgingFuture != null) {
hedgingFuture.cancel(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks hedging also has the same problem.
A UT would be really helpful in either retry or hedging cases.

@larry-safran larry-safran merged commit c0a9d31 into grpc:master Jan 31, 2024
14 checks passed
@larry-safran larry-safran deleted the retry_timeout branch January 31, 2024 21:15
@larry-safran larry-safran added the TODO:backport PR needs to be backported. Removed after backport complete label Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 1, 2024
larry-safran added a commit that referenced this pull request Feb 5, 2024
* Fix retries that timeout hanging forever. (#10855)

Fixes #10336

* Fix flaky retry tests (#10887)

* Reorder tracing and actually closing listener to eliminate test flakiness
* Use real value rather than mock for flaky test
larry-safran added a commit that referenced this pull request Feb 6, 2024
larry-safran added a commit that referenced this pull request Feb 6, 2024
* Fix retries that timeout hanging forever. (#10855)

Fixes #10336

* Fix flaky retry tests (#10887)
larry-safran added a commit to larry-safran/grpc-java that referenced this pull request Feb 7, 2024
larry-safran added a commit that referenced this pull request Feb 7, 2024
larry-safran added a commit that referenced this pull request Feb 7, 2024
@larry-safran larry-safran removed the TODO:backport PR needs to be backported. Removed after backport complete label Feb 20, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

gRPC retry behave weird when timeout is set along with retry policy
4 participants