Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(batch): add async processor #1616

Merged
merged 3 commits into from
Jul 19, 2023
Merged

feat(batch): add async processor #1616

merged 3 commits into from
Jul 19, 2023

Conversation

dreamorosi
Copy link
Contributor

Description of your changes

This PR introduces a new class and a new high-level function to the Batch Processing utility that allows to process a batch using asynchronous functions. The new class is called AsyncBatchProcessor and uses most of the same logic as the regular BatchProcessor, except that it process the records using an asynchronous handler and it uses Promise.all to run all the promises in parallel. The new companion high-level function, called asyncProcessPartialResponse() follows the same logic as the processPartialResponse() one in that it wraps the class and handles the orchestration before returning any failures.

Related issues, RFCs

Issue number: #1603

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my change is effective and works
  • The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@dreamorosi dreamorosi requested a review from a team as a code owner July 16, 2023 21:40
@dreamorosi dreamorosi self-assigned this Jul 16, 2023
@boring-cyborg boring-cyborg bot added batch This item relates to the Batch Processing Utility dependencies Changes that touch dependencies, e.g. Dependabot, etc. tests PRs that add or change tests labels Jul 16, 2023
@pull-request-size pull-request-size bot added the size/XXL PRs with 1K+ LOC, largely documentation related label Jul 16, 2023
@github-actions github-actions bot added the feature PRs that introduce new features or minor changes label Jul 16, 2023
@dreamorosi dreamorosi linked an issue Jul 16, 2023 that may be closed by this pull request
Maintenance: revisit async/sync processing for batch utility #1603
Closed
2 tasks
erikayao93
erikayao93 reviewed Jul 18, 2023
View reviewed changes
packages/batch/src/BasePartialProcessor.ts Outdated Show resolved Hide resolved
packages/batch/src/BasePartialProcessor.ts Outdated Show resolved Hide resolved
packages/batch/src/constants.ts Show resolved Hide resolved
packages/batch/tests/unit/AsyncBatchProcessor.test.ts Outdated Show resolved Hide resolved
packages/batch/tests/unit/AsyncBatchProcessor.test.ts Outdated Show resolved Hide resolved
packages/batch/tests/unit/SqsFifoPartialProcessor.test.ts Outdated Show resolved Hide resolved
packages/batch/tests/unit/asyncProcessPartialResponse.test.ts Outdated Show resolved Hide resolved
packages/batch/tests/unit/asyncProcessPartialResponse.test.ts Show resolved Hide resolved
packages/batch/tests/unit/asyncProcessPartialResponse.test.ts Show resolved Hide resolved
@dreamorosi dreamorosi removed the dependencies Changes that touch dependencies, e.g. Dependabot, etc. label Jul 19, 2023
@boring-cyborg boring-cyborg bot added the dependencies Changes that touch dependencies, e.g. Dependabot, etc. label Jul 19, 2023
@sonarcloud
Copy link

sonarcloud bot commented Jul 19, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@dreamorosi dreamorosi removed dependencies Changes that touch dependencies, e.g. Dependabot, etc. tests PRs that add or change tests labels Jul 19, 2023
@dreamorosi dreamorosi merged commit 4e6c0d2 into feat/batch Jul 19, 2023
10 checks passed
@dreamorosi dreamorosi deleted the feat/async_batch branch July 19, 2023 17:10
dreamorosi added a commit that referenced this pull request Jul 25, 2023
@dreamorosi @erikayao93
@dependabot @github-actions
feat(batch): add batch processing utility (#1625)
c4e6b19 
* chore: init workspace

* chore: init workspace

* feat(batch): Implementation of base batch processing classes (#1588)

* chore: init workspace

* chore: init workspace

* Initial base class implementation

* Added BatchProcessor implementation, attempted fix for async

* Added unit tests

* Refactoring unit tests

* Lint fix, updated docstrings

* Added response and identifier typings

* test(idempotency): improve integration tests for utility (#1591)

* docs: new name

* chore: rename e2e files

* tests(idempotency): expand integration tests

* chore(idempotency): remove unreachable code

* Removed unnecessary type casting

* Moved exports for handlers and factories

* Updated imports, refactored randomization in factories

* Refactored EventType to be const instead of enum

* Refactored and added documentation for errors

* Removed debugging line

* chore(ci): add canary to layer deployment (#1593)

* docs(idempotency): write utility docs (#1592)

* docs: base docs

* wip

* chore: added paths to snippets tsconfig

* chore: added page to docs menu

* docs(idempotency): utility docs

* highlights

* chore: remove CDK mention

* build(internal): bump semver from 5.7.1 to 5.7.2 (#1594)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(idempotency): mark the utility ready public beta (#1595)

* chore(idempotency): mark utility as public beta

* chore: manually increment version in commons

* docs(internal): update AWS SDK links to new docs (#1597)

* chore(maintenance): remove parameters utility from layer bundling and layers e2e tests (#1599)

* remove parameter from e2e tests

* remove parameters from canary stack as well

* chore(release): v1.11.1 [skip ci]

* fix canary deploy in ci with correct workspace name (#1601)

* chore: update layer ARN on documentation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrea Amorosi <dreamorosi@gmail.com>
Co-authored-by: Alexander Schueren <amelnyk@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>

* feat(batch): Batch processing wrapper function (#1605)

* Refactored some types, added function wrapper and base test

* Added record check and tests, renamed factories

* Refactored type check logic in function

* Refactor test to remove error ignore

* feat(batch): Implement SQS FIFO processor class (#1606)

* Added SQS FIFO processor and unit tests

* Added docstring for pbatch processing function

* feat(batch): Support for Lambda context access in batch processing (#1609)

* Added types and parameter for lambda context, added unit tests

* Refactor parameter checking

* Added test for malformed context handling

* docs: created utility docs

* docs: fixed white spaces

* feat(batch): add async processor (#1616)

* feat(batch): add async processor

* tests: improved unit tests

* chore: removed docstring + edited test handler

* chore: fix typos

* docs: added README

* chore: added package to beta release

* chore: marked package as public

* chore: added new batch page to docs

* chore: added utility to lerna workspace

* chore: added utility to main readme

* chore: added utility to CI

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Erika Yao <71943596+erikayao93@users.noreply.github.com>
Co-authored-by: Alexander Schueren <amelnyk@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Release bot[bot] <aws-devax-open-source@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikayao93 erikayao93 erikayao93 approved these changes

Assignees

@dreamorosi dreamorosi

Labels
batch This item relates to the Batch Processing Utility feature PRs that introduce new features or minor changes size/XXL PRs with 1K+ LOC, largely documentation related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Maintenance: revisit async/sync processing for batch utility
2 participants
@dreamorosi @erikayao93