Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Functionbeat] Allow to specific the parellization factor for Kinesis. #16901

Closed
ph opened this issue Mar 9, 2020 · 3 comments · Fixed by #20727
Closed

[Functionbeat] Allow to specific the parellization factor for Kinesis. #16901

ph opened this issue Mar 9, 2020 · 3 comments · Fixed by #20727
Labels
candidate Candidate to be added to the current iteration enhancement Functionbeat good first issue Indicates a good issue for first-time contributors Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@ph
Copy link
Contributor

ph commented Mar 9, 2020

When you create a function that monitors a Kinesis stream, the way by default that Kinesis operates is to create a lambda per shards. When you have a lot of data this causes the function to lag behind the data.

The solution is to change the parallelization factor and this post. From their documentation:

Now you can specify the number of concurrent batches that Lambda polls from a shard via a Parallelization Factor from 1 (default) to 10. For example, when Parallelization Factor is set to 2, you can have 200 concurrent Lambda invocations at maximum to process 100 Kinesis data shards. This helps scale up the processing throughput when the data volume is volatile and the IteratorAge is high.
@ph ph added Functionbeat Team:Services (Deprecated) Label for the former Integrations-Services team labels Mar 9, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@ph
Copy link
Contributor Author

ph commented Mar 9, 2020

ping @kvch

@andresrc andresrc added [zube]: Inbox enhancement good first issue Indicates a good issue for first-time contributors candidate Candidate to be added to the current iteration [zube]: Ready and removed [zube]: Inbox labels Mar 10, 2020
@ravinaik1312
Copy link
Contributor

ravinaik1312 commented Aug 10, 2020 •
edited
Loading

I am currently working on this, will send in a PR soon. The maintainer can assign this to me.

@ravinaik1312 ravinaik1312 mentioned this issue Aug 20, 2020
Merged
6 tasks
kvch pushed a commit that referenced this issue Aug 27, 2020
@ravinaik1312
Add paralelization_factor to Functionbeat Kinesis (#20727)
98c434a 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams. 
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/


## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.


- Closes #16901
@zube zube bot added [zube]: Done and removed [zube]: Ready labels Aug 27, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this issue Oct 14, 2020
@ravinaik1312 @melchiormoulin
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
ed33c7b 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams. 
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/


## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.


- Closes elastic#16901
kvch pushed a commit to kvch/beats that referenced this issue Nov 4, 2020
@ravinaik1312 @kvch
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
84dff06 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes elastic#16901

(cherry picked from commit 98c434a)
@kvch kvch mentioned this issue Nov 4, 2020
Merged
6 tasks
kvch pushed a commit to kvch/beats that referenced this issue Nov 4, 2020
@ravinaik1312 @kvch
Add paralelization_factor to Functionbeat Kinesis (elastic#20727)
927c49a 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes elastic#16901

(cherry picked from commit 98c434a)
@kvch kvch mentioned this issue Nov 4, 2020
Merged
6 tasks
kvch added a commit that referenced this issue Nov 6, 2020
@kvch @ravinaik1312
Add paralelization_factor to Functionbeat Kinesis (#20727) (#22425)
fb31b55 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes #16901

(cherry picked from commit 98c434a)

Co-authored-by: Ravi Naik <ravinaik1312@gmail.com>
kvch added a commit that referenced this issue Nov 6, 2020
@kvch @ravinaik1312
Add paralelization_factor to Functionbeat Kinesis (#20727) (#22426)
4948f9d 
## What does this PR do?

This PR adds the ability to add the parallelization factor configuration to functionbeat when reading from Kinesis streams.
https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

## Why is it important?

This configuration allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously.

- Closes #16901

(cherry picked from commit 98c434a)

Co-authored-by: Ravi Naik <ravinaik1312@gmail.com>
@zube zube bot removed the [zube]: Done label Nov 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
candidate Candidate to be added to the current iteration enhancement Functionbeat good first issue Indicates a good issue for first-time contributors Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feature - parallelization factor kinesis functionbeat config ravinaik1312/beats
4 participants
@ph @andresrc @ravinaik1312 @elasticmachine