Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Bulk Action on Connection and Engine #24

Closed
alexander-schranz opened this issue Jan 3, 2023 · 3 comments · Fixed by #430
Closed

[Core] Bulk Action on Connection and Engine #24

alexander-schranz opened this issue Jan 3, 2023 · 3 comments · Fixed by #430
Labels
features New feature or request SEAL Core Seal Core related issue

Comments

@alexander-schranz
Copy link
Member

alexander-schranz commented Jan 3, 2023

It should be possible to provide Bulk Action on Connection and Engine.

$engine->bulk(string $index, \Generator $saveDocuments, \Generator $deleteDocuments, bulkSize: 100);

// and

$indexer->bulk(Index $index, \Generator $saveDocuments, \Generator $deleteDocuments, bulkSize: 100);

Engine not supporting bulk action can fallback to basic save and delete methods.

The documents should be able to be aryor \Generator for performance.

A BulkableIndexerInterface should be added so the fallback to the save/delete can be part of the Engine and not every adapter itself need to implement it.

Bulk Action make maybe sense in case of:

Implementing of Reindex Providers #16

@alexander-schranz alexander-schranz added the features New feature or request label Jan 3, 2023
@alexander-schranz alexander-schranz changed the title Bulk Action on Connection and Engine [All] Bulk Action on Connection and Engine Jan 4, 2023
@alexander-schranz alexander-schranz changed the title [All] Bulk Action on Connection and Engine [Core] Bulk Action on Connection and Engine Jan 4, 2023
@alexander-schranz alexander-schranz added the SEAL Core Seal Core related issue label Jan 4, 2023
@zajca
Copy link

zajca commented Mar 25, 2024

I was just playing with loupe adapter and without batch support I was indexing ~5k recors about 35minutes, so I've implemented own reindexing and with 100 batch it did take around 30s. I guess performance will suffer on every adapter when importing more documents, so batch API is must.
I like example of the API you have provided btw. Not sure how many backend support delete and update/create in same batch but I guess that it make sense you do want both operation on dataset.

@alexander-schranz
Copy link
Member Author

@zajca thx for giving SEAL a try and give feedback here. For the adapters not supporting batch or have seperate APIs for update/create and batch I'm planning to fallback to the normal APIs. So they may not have performance improvement but the API from outside for all adapters will be the same and so people using SEAL don't need create different code depending on the Adapters.

@alexander-schranz
Copy link
Member Author

alexander-schranz commented Oct 6, 2024

this is in progress: #430

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
features New feature or request SEAL Core Seal Core related issue
Projects
None yet
2 participants