Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Add max_number_of_messages config parameter for S3 input #21993

Merged
merged 3 commits into from
Oct 20, 2020
Merged

[Filebeat] Add max_number_of_messages config parameter for S3 input #21993

merged 3 commits into from
Oct 20, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 19, 2020 •
edited
Loading

What does this PR do?

This PR adds max_number_of_messages config option for S3 input to control the number of SQS messages returning from each ReceiveMessage API call.

Why is it important?

This will help to debug performance issues since this controls how many messages returned once and how many log files will be processed in parallel together.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 19, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 19, 2020
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. review Team:Platforms Label for the Integrations - Platforms team labels Oct 19, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 19, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 19, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21993 updated]

  • Start Time: 2020-10-20T14:58:25.091+0000

  • Duration: 64 min 22 sec

Test stats 🧪

Test Results
Failed 0
Passed 1947
Skipped 259
Total 2206

ChrsMark
ChrsMark reviewed Oct 20, 2020
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

Left some minors only. Let me know what you think!

x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/s3/config.go Show resolved Hide resolved
x-pack/filebeat/input/s3/config.go Show resolved Hide resolved
x-pack/filebeat/input/s3/config.go Show resolved Hide resolved
ChrsMark
ChrsMark approved these changes Oct 20, 2020
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Feel free to add the docs suggestions if make sense to you!

@kaiyan-sheng
Copy link
Contributor Author

@ChrsMark Just did, thank you for the review and suggestion!!

@kaiyan-sheng kaiyan-sheng merged commit a10dca7 into elastic:master Oct 20, 2020
@kaiyan-sheng kaiyan-sheng deleted the sqs_message_number branch October 20, 2020 16:32
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 20, 2020
Merged
5 tasks
@kaiyan-sheng kaiyan-sheng added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 20, 2020
v1v added a commit to v1v/beats that referenced this pull request Oct 21, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/beats-tes…
34d2067 
…ter-commit

* upstream/master:
  feat: package aliases for snapshots (elastic#21960)
  [DOC] Add firewall as possible troubleshooting issue (elastic#21743)
  [Filebeat] Add max_number_of_messages config parameter for S3 input (elastic#21993)
  [Elastic Agent] Fix missing elastic_agent event data  (elastic#21994)
  Document auditbeat system process module config (elastic#21766)
  Update links (elastic#22012)
v1v added a commit to v1v/beats that referenced this pull request Oct 21, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-10
91e9d54 
* upstream/master:
  feat: package aliases for snapshots (elastic#21960)
  [DOC] Add firewall as possible troubleshooting issue (elastic#21743)
  [Filebeat] Add max_number_of_messages config parameter for S3 input (elastic#21993)
  [Elastic Agent] Fix missing elastic_agent event data  (elastic#21994)
  Document auditbeat system process module config (elastic#21766)
  Update links (elastic#22012)
  dynamically find librpm (elastic#21936)
  Fix Istio docs (elastic#22019)
  [beats-tester][packaging] store packages in another location (elastic#21903)
  [Kubernetes] Remove redundant dockersock volume mount (elastic#22009)
  [Ingest Manager] Always try snapshot repo for agent upgrade (elastic#21951)
  Azure storage metricset values not inside the metricset name (elastic#21845)
  fix diskio and memory bugs under windows (elastic#21992)
  Fix TestDockerStart flaky test (elastic#21681)
  filebeat: add SSL options to checkpoint module (elastic#19560)
  Stop storing stateless kubernetes keystores (elastic#21880)
  [Elastic Agent] Fix named pipe communication on Windows 7 (elastic#21931)
  [Elastic Agent] Fix index for Agent monitoring to to elastic_agent. (elastic#21932)
@kaiyan-sheng
Copy link
Contributor Author

Follow up: #22057

kaiyan-sheng added a commit that referenced this pull request Oct 22, 2020
@kaiyan-sheng
Cherry-pick #21993 to 7.x: [Filebeat] Add max_number_of_messages conf…
fe4bbd8 
…ig parameter for S3 input (#22035)

* [Filebeat] Add max_number_of_messages config parameter for S3 input (#21993)

(cherry picked from commit a10dca7)
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 22, 2020
Merged
6 tasks
v1v added a commit to v1v/beats that referenced this pull request Oct 22, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-2008
2080cc2 
* upstream/master: (27 commits)
  chore: use ubuntu 18 as linux agent (elastic#22084)
  docs: Prepare Changelog for 7.9.3 (elastic#22073) (elastic#22075)
  docs: Prepare Changelog for 6.8.13 (elastic#22072) (elastic#22079)
  [build][packaging] Add resilience when docker build (elastic#22050)
  Fix the url of reviewdog (elastic#21981)
  revert WSS process reporting for windows (elastic#22055)
  Fix typo (elastic#19585) (elastic#22061)
  [Ingest Manager] Use ML_SYSTEM to detect if agent is running as a service (elastic#21884)
  Prevent log input from sending duplicate messages due to file renaming (elastic#21911)
  feat: package aliases for snapshots (elastic#21960)
  [DOC] Add firewall as possible troubleshooting issue (elastic#21743)
  [Filebeat] Add max_number_of_messages config parameter for S3 input (elastic#21993)
  [Elastic Agent] Fix missing elastic_agent event data  (elastic#21994)
  Document auditbeat system process module config (elastic#21766)
  Update links (elastic#22012)
  dynamically find librpm (elastic#21936)
  Fix Istio docs (elastic#22019)
  [beats-tester][packaging] store packages in another location (elastic#21903)
  [Kubernetes] Remove redundant dockersock volume mount (elastic#22009)
  [Ingest Manager] Always try snapshot repo for agent upgrade (elastic#21951)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
review Team:Platforms Label for the Integrations - Platforms team v7.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kaiyan-sheng @elasticmachine @ChrsMark