Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python][c++] C++ reindexer must use a single thread pool shared across all indexer objects #2146

Closed
beroy opened this issue Feb 14, 2024 · 3 comments
Assignees
Labels

Comments

@beroy
Copy link
Collaborator

beroy commented Feb 14, 2024

Is your feature request related to a problem? Please describe.
Currently the C++ Reindexer creates one threadpool per reindxer objects which can lead to further explosion of threads.

Describe the solution you'd like
An alternative solution is to use a single threadpool for all the reindexer objects.
To do this, the following need to done:

  • Allocate a single process level threadpool
  • Placing the indexer object pointer in the threadpool lookup lambda
  • Few small fixes

The API changes:

  • Thread pool should be part of the soma context, which already has a lifecycle.
  • Config and setup can be part of the soma context setup/init. For example, it can be soma.indexer.concurrency
@beroy beroy self-assigned this Feb 14, 2024
@beroy beroy added the priority label Feb 14, 2024
@beroy
Copy link
Collaborator Author

beroy commented Feb 14, 2024

CC @bkmartinjr @johnkerl

@bkmartinjr
Copy link
Member

Fully agree with this. A common case is two indexers used at the same time (obs & var), and it would be much nicer to have a single, shared thread pool for the entire SOMATileDBContext so that we can amortize the cost, have fewer threads, etc.

CC: @thetorpedodog

@johnkerl johnkerl changed the title [Python][C++] C++ Reindexer must use a single thread pool shared across all indexer objects [python][c++] C++ reindexer must use a single thread pool shared across all indexer objects Feb 22, 2024
@beroy
Copy link
Collaborator Author

beroy commented Mar 11, 2024

2148

@beroy beroy closed this as completed Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants