Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature - parallelization factor kinesis functionbeat config #20727

Merged
merged 3 commits into from
Aug 27, 2020
Merged

feature - parallelization factor kinesis functionbeat config #20727

merged 3 commits into from
Aug 27, 2020

Conversation

ravinaik1312
Copy link
Contributor

@ravinaik1312 ravinaik1312 commented Aug 20, 2020 •
edited
Loading

What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2020
@cla-checker-service
Copy link

cla-checker-service bot commented Aug 20, 2020 •
edited
Loading

💚 CLA has been signed

@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 20, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [kvch commented: jenkins run tests]

  • Start Time: 2020-08-27T07:56:15.374+0000

  • Duration: 38 min 11 sec

Test stats 🧪

Test Results
Failed 0
Passed 302
Skipped 4
Total 306

@ravinaik1312
Copy link
Contributor Author

@olksdr @gtback Did the CCLA get added for my Github user?

@olksdr
Copy link

olksdr commented Aug 21, 2020

@ravinaik1312 CLA is in our lists. The message #20727 (comment) points to the commit which is wrong. When you look into it it was done with completely different email, I assume with wrong gitconfig - because you first commit does not show the github user either. Last merge commit looks good.
Could you, please, squash the commits and rebase to remove that failed commit from history ?

@andresrc andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label Aug 21, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 21, 2020
@kvch kvch self-assigned this Aug 24, 2020
@kvch
Copy link
Contributor

kvch commented Aug 24, 2020

jenkins run tests

@ravinaik1312 ravinaik1312 force-pushed the feature/parallelization-factor-kinesis-functionbeat-config branch from 38d8345 to 1bebd9a Compare August 25, 2020 00:35
@ravinaik1312 ravinaik1312 force-pushed the feature/parallelization-factor-kinesis-functionbeat-config branch from 1bebd9a to eb893e4 Compare August 25, 2020 00:43
@kvch
Copy link
Contributor

kvch commented Aug 25, 2020

@ravinaik1312 Awesome PR! Could you please add an entry to the changelog?

@kvch kvch added [zube]: In Review needs_backport PR is waiting to be backported to other branches. labels Aug 25, 2020
@ravinaik1312
Copy link
Contributor Author

@ravinaik1312 Awesome PR! Could you please add an entry to the changelog?

Thank you! I've updated the changelog.next.asciidoc, let me know if there are other changelog files that needs to be updated too.

@kvch
Copy link
Contributor

kvch commented Aug 27, 2020

jenkins run tests

@kvch kvch merged commit 98c434a into elastic:master Aug 27, 2020
@gtback
Copy link
Member

gtback commented Aug 27, 2020

Thanks @ravinaik1312 !

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@ravinaik1312 @melchiormoulin
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
ed33c7b 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams. 
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/


## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.


- Closes elastic#16901
kvch pushed a commit to kvch/beats that referenced this pull request Nov 4, 2020
@ravinaik1312 @kvch
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
84dff06 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes elastic#16901

(cherry picked from commit 98c434a)
@kvch kvch mentioned this pull request Nov 4, 2020
Merged
6 tasks
@kvch kvch added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Nov 4, 2020
kvch pushed a commit to kvch/beats that referenced this pull request Nov 4, 2020
@ravinaik1312 @kvch
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
927c49a 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes elastic#16901

(cherry picked from commit 98c434a)
@kvch kvch mentioned this pull request Nov 4, 2020
Merged
6 tasks
@kvch kvch added the v7.10.0 label Nov 4, 2020
kvch added a commit that referenced this pull request Nov 6, 2020
@kvch @ravinaik1312
Add paralelization_factor to Functionbeat Kinesis (#20727) (#22425)
fb31b55 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes #16901

(cherry picked from commit 98c434a)

Co-authored-by: Ravi Naik <ravinaik1312@gmail.com>
kvch added a commit that referenced this pull request Nov 6, 2020
@kvch @ravinaik1312
Add paralelization_factor to Functionbeat Kinesis (#20727) (#22426)
4948f9d 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes #16901

(cherry picked from commit 98c434a)

Co-authored-by: Ravi Naik <ravinaik1312@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kvch kvch kvch approved these changes

Assignees

@kvch kvch

Labels
Team:Services (Deprecated) Label for the former Integrations-Services team v7.10.0 v7.11.0 [zube]: In Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Functionbeat] Allow to specific the parellization factor for Kinesis.
6 participants
@ravinaik1312 @elasticmachine @olksdr @kvch @gtback @andresrc