You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently the C++ Reindexer creates one threadpool per reindxer objects which can lead to further explosion of threads.
Describe the solution you'd like
An alternative solution is to use a single threadpool for all the reindexer objects.
To do this, the following need to done:
Allocate a single process level threadpool
Placing the indexer object pointer in the threadpool lookup lambda
Few small fixes
The API changes:
Thread pool should be part of the soma context, which already has a lifecycle.
Config and setup can be part of the soma context setup/init. For example, it can be soma.indexer.concurrency
The text was updated successfully, but these errors were encountered:
Fully agree with this. A common case is two indexers used at the same time (obs & var), and it would be much nicer to have a single, shared thread pool for the entire SOMATileDBContext so that we can amortize the cost, have fewer threads, etc.
johnkerl
changed the title
[Python][C++] C++ Reindexer must use a single thread pool shared across all indexer objects
[python][c++] C++ reindexer must use a single thread pool shared across all indexer objects
Feb 22, 2024
Is your feature request related to a problem? Please describe.
Currently the C++ Reindexer creates one threadpool per reindxer objects which can lead to further explosion of threads.
Describe the solution you'd like
An alternative solution is to use a single threadpool for all the reindexer objects.
To do this, the following need to done:
The API changes:
soma.indexer.concurrency
The text was updated successfully, but these errors were encountered: