Skip to content

Commit

Permalink
add note how to avoid OOM errors to BaseFeaturizer.set_n_jobs() doc str
Browse files Browse the repository at this point in the history
  • Loading branch information
janosh committed Nov 19, 2022
1 parent 1141c53 commit 2d9c499
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions matminer/featurizers/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,14 @@ class BaseFeaturizer(BaseEstimator, TransformerMixin, ABC):
to reproduce the feature, just to get an idea what it is.
"""

def set_n_jobs(self, n_jobs):
"""Set the number of threads for this."""
def set_n_jobs(self, n_jobs: int) -> None:
"""Set the number of concurrent jobs to spawn during featurization.
Args:
n_jobs (int): Number of threads in multiprocessing pool.
Note: It seems multiprocessing can be the cause of out-of-memory (OOM) errors, especially when trying to featurize large structures on HPC nodes with strict memory limits. Using featurizer.set_n_jobs(1) has been known to help as a workaround.
"""
self._n_jobs = n_jobs

@property
Expand Down

0 comments on commit 2d9c499

Please sign in to comment.