feat(batch): Implementation of base batch processing classes #1588

erikayao93 · 2023-07-07T16:54:38Z

Description of your changes

Added BasePartialProcessor abstract class with basic processing framework
Added BasePartialBatchProcessor abstract class with processing framework specific to SQS, Kinesis, and DynamoDB
Added BatchProcessor class with processing implementation
Added unit tests with 100% coverage on new classes

Related issues, RFCs

Issue number: #1586

Checklist

My changes meet the tenets criteria
I have performed a self-review of my own code
I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my change is effective and works
The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

I have documented the migration process
I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

dreamorosi · 2023-07-07T17:12:13Z

Thank you so much for the PR Erika, we'll be reviewing during the next few days (likely by Monday EOD EMEA time).

Looking forward it this one!

dreamorosi · 2023-07-08T09:39:52Z

@mergify rebase

mergify · 2023-07-08T09:39:57Z

rebase

❌ Base branch update has failed

Git reported the following error:

warning: skipped previously applied commit a3dddbf
hint: use --reapply-cherry-picks to include skipped commits
hint: Disable this message with "git config advice.skippedCherryPicks false"
Rebasing (1/7)
Auto-merging package-lock.json
CONFLICT (content): Merge conflict in package-lock.json
Auto-merging packages/batch/typedoc.json
CONFLICT (add/add): Merge conflict in packages/batch/typedoc.json
error: could not apply 31b2306... chore: init workspace
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 31b2306... chore: init workspace

err-code: 7715D

…ols#1591) * docs: new name * chore: rename e2e files * tests(idempotency): expand integration tests * chore(idempotency): remove unreachable code

dreamorosi

Thank you for the work on this PR, I have left a number of comments but 9/10 are minor ones and at times related to convention/styling.

The implementation is going in the right direction and I look forward to get it merged!

packages/batch/src/constants.ts

packages/batch/src/errors.ts

packages/batch/tests/helpers/factories.ts

packages/batch/tests/helpers/handlers.ts

packages/batch/tests/unit/BatchProcessor.test.ts

* docs: base docs * wip * chore: added paths to snippets tsconfig * chore: added page to docs menu * docs(idempotency): utility docs * highlights * chore: remove CDK mention

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md) - [Commits](npm/node-semver@v5.7.1...v5.7.2) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…s#1595) * chore(idempotency): mark utility as public beta * chore: manually increment version in commons

… layers e2e tests (aws-powertools#1599) * remove parameter from e2e tests * remove parameters from canary stack as well

)

dreamorosi

Thank you for the work on this PR and congrats for landing your this major PR in the repo 🎉

* chore: init workspace * chore: init workspace * Initial base class implementation * Added BatchProcessor implementation, attempted fix for async * Added unit tests * Refactoring unit tests * Lint fix, updated docstrings * Added response and identifier typings * test(idempotency): improve integration tests for utility (#1591) * docs: new name * chore: rename e2e files * tests(idempotency): expand integration tests * chore(idempotency): remove unreachable code * Removed unnecessary type casting * Moved exports for handlers and factories * Updated imports, refactored randomization in factories * Refactored EventType to be const instead of enum * Refactored and added documentation for errors * Removed debugging line * chore(ci): add canary to layer deployment (#1593) * docs(idempotency): write utility docs (#1592) * docs: base docs * wip * chore: added paths to snippets tsconfig * chore: added page to docs menu * docs(idempotency): utility docs * highlights * chore: remove CDK mention * build(internal): bump semver from 5.7.1 to 5.7.2 (#1594) Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md) - [Commits](npm/node-semver@v5.7.1...v5.7.2) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(idempotency): mark the utility ready public beta (#1595) * chore(idempotency): mark utility as public beta * chore: manually increment version in commons * docs(internal): update AWS SDK links to new docs (#1597) * chore(maintenance): remove parameters utility from layer bundling and layers e2e tests (#1599) * remove parameter from e2e tests * remove parameters from canary stack as well * chore(release): v1.11.1 [skip ci] * fix canary deploy in ci with correct workspace name (#1601) * chore: update layer ARN on documentation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Andrea Amorosi <dreamorosi@gmail.com> Co-authored-by: Alexander Schueren <amelnyk@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>

* chore: init workspace * chore: init workspace * feat(batch): Implementation of base batch processing classes (#1588) * chore: init workspace * chore: init workspace * Initial base class implementation * Added BatchProcessor implementation, attempted fix for async * Added unit tests * Refactoring unit tests * Lint fix, updated docstrings * Added response and identifier typings * test(idempotency): improve integration tests for utility (#1591) * docs: new name * chore: rename e2e files * tests(idempotency): expand integration tests * chore(idempotency): remove unreachable code * Removed unnecessary type casting * Moved exports for handlers and factories * Updated imports, refactored randomization in factories * Refactored EventType to be const instead of enum * Refactored and added documentation for errors * Removed debugging line * chore(ci): add canary to layer deployment (#1593) * docs(idempotency): write utility docs (#1592) * docs: base docs * wip * chore: added paths to snippets tsconfig * chore: added page to docs menu * docs(idempotency): utility docs * highlights * chore: remove CDK mention * build(internal): bump semver from 5.7.1 to 5.7.2 (#1594) Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md) - [Commits](npm/node-semver@v5.7.1...v5.7.2) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(idempotency): mark the utility ready public beta (#1595) * chore(idempotency): mark utility as public beta * chore: manually increment version in commons * docs(internal): update AWS SDK links to new docs (#1597) * chore(maintenance): remove parameters utility from layer bundling and layers e2e tests (#1599) * remove parameter from e2e tests * remove parameters from canary stack as well * chore(release): v1.11.1 [skip ci] * fix canary deploy in ci with correct workspace name (#1601) * chore: update layer ARN on documentation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Andrea Amorosi <dreamorosi@gmail.com> Co-authored-by: Alexander Schueren <amelnyk@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com> * feat(batch): Batch processing wrapper function (#1605) * Refactored some types, added function wrapper and base test * Added record check and tests, renamed factories * Refactored type check logic in function * Refactor test to remove error ignore * feat(batch): Implement SQS FIFO processor class (#1606) * Added SQS FIFO processor and unit tests * Added docstring for pbatch processing function * feat(batch): Support for Lambda context access in batch processing (#1609) * Added types and parameter for lambda context, added unit tests * Refactor parameter checking * Added test for malformed context handling * docs: created utility docs * docs: fixed white spaces * feat(batch): add async processor (#1616) * feat(batch): add async processor * tests: improved unit tests * chore: removed docstring + edited test handler * chore: fix typos * docs: added README * chore: added package to beta release * chore: marked package as public * chore: added new batch page to docs * chore: added utility to lerna workspace * chore: added utility to main readme * chore: added utility to CI --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Erika Yao <71943596+erikayao93@users.noreply.github.com> Co-authored-by: Alexander Schueren <amelnyk@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>

dreamorosi and others added 9 commits June 27, 2023 16:46

chore: init workspace

31b2306

chore: init workspace

a3dddbf

Merge branch 'aws-powertools:main' into batch

90d4ce0

Initial base class implementation

d87b895

Added BatchProcessor implementation, attempted fix for async

fdcb59f

Added unit tests

8bb91d5

Refactoring unit tests

8767d6a

Lint fix, updated docstrings

a90e2b6

Merge branch 'aws-powertools:main' into batch

fc32c81

erikayao93 requested a review from a team as a code owner July 7, 2023 16:54

pull-request-size bot added the size/XXL PRs with 1K+ LOC, largely documentation related label Jul 7, 2023

dreamorosi force-pushed the feat/batch branch from a3dddbf to 7f27b7c Compare July 7, 2023 16:57

github-actions bot added the feature PRs that introduce new features or minor changes label Jul 7, 2023

dreamorosi assigned erikayao93 Jul 7, 2023

dreamorosi linked an issue Jul 7, 2023 that may be closed by this pull request

Feature request: implement BasePartialBatchProcessor class #1586

Closed

Added response and identifier typings

d417d71

test(idempotency): improve integration tests for utility (aws-powerto…

863edc8

…ols#1591) * docs: new name * chore: rename e2e files * tests(idempotency): expand integration tests * chore(idempotency): remove unreachable code

dreamorosi reviewed Jul 10, 2023

View reviewed changes

erikayao93 added 3 commits July 10, 2023 16:51

Removed unnecessary type casting

ea50345

Moved exports for handlers and factories

eaee175

Updated imports, refactored randomization in factories

500fbe0

erikayao93 and others added 13 commits July 10, 2023 17:29

Refactored EventType to be const instead of enum

002ef9e

Refactored and added documentation for errors

d9be091

Removed debugging line

604f04d

chore(ci): add canary to layer deployment (aws-powertools#1593)

9f1696b

docs(idempotency): write utility docs (aws-powertools#1592)

3b5ccc1

* docs: base docs * wip * chore: added paths to snippets tsconfig * chore: added page to docs menu * docs(idempotency): utility docs * highlights * chore: remove CDK mention

chore(idempotency): mark the utility ready public beta (aws-powertool…

0fc4f1e

…s#1595) * chore(idempotency): mark utility as public beta * chore: manually increment version in commons

docs(internal): update AWS SDK links to new docs (aws-powertools#1597)

5266d55

chore(maintenance): remove parameters utility from layer bundling and…

16719c3

… layers e2e tests (aws-powertools#1599) * remove parameter from e2e tests * remove parameters from canary stack as well

chore(release): v1.11.1 [skip ci]

3f03c5c

fix canary deploy in ci with correct workspace name (aws-powertools#1601

3251b9c

)

chore: update layer ARN on documentation

7cfac60

Merge branch 'feat/batch' into batch

840208f

pull-request-size bot added size/XL PRs between 500-999 LOC, often PRs that grown with feedback and removed size/XXL PRs with 1K+ LOC, largely documentation related labels Jul 11, 2023

dreamorosi mentioned this pull request Jul 11, 2023

Maintenance: revisit async/sync processing for batch utility #1603

Closed

2 tasks

Merge branch 'aws-powertools:main' into batch

3de49db

pull-request-size bot added size/XXL PRs with 1K+ LOC, largely documentation related and removed size/XL PRs between 500-999 LOC, often PRs that grown with feedback labels Jul 11, 2023

dreamorosi self-requested a review July 11, 2023 16:31

dreamorosi approved these changes Jul 11, 2023

View reviewed changes

dreamorosi merged commit 9d8df3d into aws-powertools:feat/batch Jul 11, 2023

dreamorosi mentioned this pull request Jul 11, 2023

Feature request: allow to access processed messages #1604

Closed

dreamorosi linked an issue Jul 11, 2023 that may be closed by this pull request

Feature request: allow to access processed messages #1604

Closed

dreamorosi mentioned this pull request Jul 19, 2023

feat(batch): add async processor #1616

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(batch): Implementation of base batch processing classes #1588

feat(batch): Implementation of base batch processing classes #1588

erikayao93 commented Jul 7, 2023

dreamorosi commented Jul 7, 2023

dreamorosi commented Jul 8, 2023

mergify bot commented Jul 8, 2023

dreamorosi left a comment

dreamorosi left a comment

feat(batch): Implementation of base batch processing classes #1588

feat(batch): Implementation of base batch processing classes #1588

Conversation

erikayao93 commented Jul 7, 2023

Description of your changes

Related issues, RFCs

Checklist

Breaking change checklist

dreamorosi commented Jul 7, 2023

dreamorosi commented Jul 8, 2023

mergify bot commented Jul 8, 2023

❌ Base branch update has failed

dreamorosi left a comment

Choose a reason for hiding this comment

dreamorosi left a comment

Choose a reason for hiding this comment