Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve BatchGet / BatchWrite to iterate over 25 records. #236

Open
mobsense opened this issue Jan 2, 2022 Discussed in #235 · 3 comments
Open

Improve BatchGet / BatchWrite to iterate over 25 records. #236

mobsense opened this issue Jan 2, 2022 Discussed in #235 · 3 comments
Labels
enhancement New feature or request

Comments

@mobsense
Copy link
Contributor

mobsense commented Jan 2, 2022

Discussed in #235

Originally posted by ebisbe January 2, 2022
I have a batch process that does 3 simple things. Query items ( I've limited to 500 ) from DDB, each item is sent to a queue and then deletes each item.
I'd like to be able to do something like:

const Items = Model.find(..., {limit: 500})
let batch = {}
for(const Item of Items){
    // send sqs message
   model.remove(..., {batch})
}
table.batchWrite(batch)

But right now I need to do with batchWrite:

const Items = Model.find(..., {limit: 500})
const operations = []
for(const Item of Items){
    // send sqs message
   operations.push({...})
}
do {
    let batch = {}
    const operationsSlice = operations.slice(0, 24)
    operationsSlice.forEach(async operation => await model.remove(operation, { batch }))
    await table.batchWrite(batch)
} while (actions > 0)

It could be real handy that the batchGet/Write handles the limit imposed by DDB to 25 items by doing the slicing himself. Maybe add an option so the user acknowledges he is sending more than 25 items.

@mobsense mobsense added the enhancement New feature or request label Jan 2, 2022
@mobsense
Copy link
Contributor Author

Merged.

@DenSpirit
Copy link

Hi. Sorry to necrobump, but was this actually merged? I can't see relevant changes in the code or a pull request, and locally batchWrite() still fails with document limit when trying to submit more than 25 documents.

@mobsense
Copy link
Contributor Author

I think you are right. Perhaps got confused with another batch related PR.

I'll reopen and flag.

@mobsense mobsense reopened this Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants