Migrate S3 Input to Filebeat Input V2 #20005
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
kaiyan-sheng
added
the
Team:Platforms
botelastic
bot
removed
the
needs_team
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
kaiyan-sheng
changed the title
|
Pinging @elastic/integrations-platforms (Team:Platforms) |
urso
reviewed
Jul 20, 2020
urso
reviewed
Jul 20, 2020
| svcSQS := sqs.New(awsConfig) | ||
| input.sqs = svcSQS |
There was a problem hiding this comment.
Having to modify the object under test for the test is kind of a red flag to me. It should be possible to create and provide a fully initialized object. Is there something missing in the construction of the collector that forces us to do this here?
There was a problem hiding this comment.
@urso I moved the code for creating a full initialized object outside of runTest function. Please let me know if this looks less suspicious for you now :)
urso
reviewed
Jul 20, 2020
urso
reviewed
Jul 20, 2020
urso
reviewed
Jul 20, 2020
kaiyan-sheng
added
the
needs_backport
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
reviewed
Oct 2, 2020
kvch
approved these changes
Oct 2, 2020
3 tasks
kaiyan-sheng
added
v7.10.0
and removed
needs_backport
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 5, 2020
…-matches-found * upstream/master: (21 commits) Skip filestream flaky tests (elastic#21490) Ignore unsupported metrics in the azure module (elastic#21486) Do not run symlink tests on Windows (elastic#21472) Map `cloud.account.id` to azure sub id (elastic#21483) Add support for app_state metricset (elastic#20639) Include original error when metricbeat fails to connect with Kafka (elastic#21484) Prompt only when agent is already enrolled (elastic#21473) Fix leftover delpoyment example (elastic#21474) Bump version to ECS 1.6 in modules without ECS updates (elastic#21455) Clarify input type configuration options (elastic#19284) Increase index pattern size check to 10MiB (elastic#21487) Migrate S3 Input to Filebeat Input V2 (elastic#20005) [libbeat] Add configurable exponential backoff for disk queue write errors (elastic#21493) Revert "Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)" (elastic#21467) Fix format of debug messages in tlscommon (elastic#21482) [CI] Change x-pack/auditbeat build events (comments, labels) (elastic#21463) [CI] changeset from elastic#20603 was not added to CI2.0 (elastic#21464) Add new log file reader for filestream input (elastic#21450) [CI] Send slack message with build status (elastic#21428) Remove duplicated sources url in dependencies report (elastic#21462) ...
kaiyan-sheng
added a commit
that referenced
this pull request
Oct 5, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 5, 2020
* upstream/master: (26 commits) [Ingest Manager] Send updating state (elastic#21461) [Filebeat][New Fileset] Cisco Umbrella support (elastic#21504) [Ingest Manager] Download asc from artifact store specified in spec (elastic#21488) Implementation of fileProspector (elastic#21479) [Metricbeat] Add latency config option into aws module (elastic#20875) Skip filestream flaky tests (elastic#21490) Ignore unsupported metrics in the azure module (elastic#21486) Do not run symlink tests on Windows (elastic#21472) Map `cloud.account.id` to azure sub id (elastic#21483) Add support for app_state metricset (elastic#20639) Include original error when metricbeat fails to connect with Kafka (elastic#21484) Prompt only when agent is already enrolled (elastic#21473) Fix leftover delpoyment example (elastic#21474) Bump version to ECS 1.6 in modules without ECS updates (elastic#21455) Clarify input type configuration options (elastic#19284) Increase index pattern size check to 10MiB (elastic#21487) Migrate S3 Input to Filebeat Input V2 (elastic#20005) [libbeat] Add configurable exponential backoff for disk queue write errors (elastic#21493) Revert "Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)" (elastic#21467) Fix format of debug messages in tlscommon (elastic#21482) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This PR is based on #19756 to finish migrating s3 input to use Filebeat input V2.
What does this PR do?
Move s3 input to input v2 API.
This change splits the internal s3Input into s3Input and s3Collector. The s3Input is responsible for configuration only.
The unit tests have been modified, but the integration tests need some more work.
Why is it important?
Update to v2 input API.
Checklist
- [ ] I have commented my code, particularly in hard-to-understand areas- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Manual testing:
s3:ObjectCreated:*type to your Amazon SQS queue.