Cherry-pick #15590 to 7.x: [Filebeat] Add timeout to GetObjectRequest for s3 input #15901
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add timeout to GetObjectRequest which will cancel the request if it takes too long * Close resp.Body from S3 GetObject API to prevent resource leak * Change aws_api_timeout to api_timeout (cherry picked from commit 86c3e63)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #15590 to 7.x branch. Original message:
Problem we see when using s3 input:
When using s3 input to read logs from S3 bucket, after a while with high amount of logs
read: connection reset by peererror showed up. This error is triggered byreader.ReadStringfunction, thenprocessorKeepAlivefound it's taking too long to runprocessMessage, which is longer than half of the set visibility timeout. SochangeVisibilityTimeoutfunction keep getting called repeatedly.This PR is to add timeout into GetObjectRequest API call by using context pattern to implement timeout logic that will cancel the request if it takes too long. This way, after the default timeout 2 minute is hit, this specific S3 object will be skipped, SQS message will return back to the queue later. So Filebeat can try to read it again later.
I decided to add a config option called
context_timeoutfor s3 input because based on your visibility_timeout value, context_timeout can be as large as half of the visibility_timeout. This will allow users to modify both timeout values when using s3 input or filebeat aws module with larger s3 objects or smaller network bandwidth.closes #15502