Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embedding joblib #843

Merged
merged 3 commits into from
Aug 29, 2022
Merged

Embedding joblib #843

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
edde394
joblib is embedded
valer1435 Aug 25, 2022
6d413e1
remove exception
valer1435 Aug 25, 2022
e0024c4
remove verbose from joblib
valer1435 Aug 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions fedot/core/optimisers/gp_comp/evaluation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
import gc
import multiprocessing
import timeit
from abc import ABC, abstractmethod
from contextlib import closing
from typing import Dict, Optional

import timeit
from datetime import datetime
from random import choice
from typing import Dict, Optional

from joblib import Parallel, delayed
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

может выше убрать вообще импорт и использования multiprocessing

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Но импорт multiprocessing вроде остался. Или он нужен?

import multiprocessing

from fedot.core.dag.graph import Graph
from fedot.core.log import default_log
Expand Down Expand Up @@ -80,15 +82,11 @@ def evaluate_with_cache(self, population: PopulationT) -> Optional[PopulationT]:
def evaluate_population(self, individuals: PopulationT) -> Optional[PopulationT]:
n_jobs = determine_n_jobs(self._n_jobs, self.logger)

if n_jobs == 1:
mapped_evals = map(self.evaluate_single, individuals)
else:
with closing(multiprocessing.Pool(n_jobs)) as pool:
mapped_evals = list(pool.imap_unordered(self.evaluate_single, individuals))

parallel = Parallel(n_jobs=n_jobs, verbose=0, pre_dispatch="2*n_jobs")
eval_inds = parallel(delayed(self.evaluate_single)(ind=ind) for ind in individuals)
# If there were no successful evals then try once again getting at least one,
# even if time limit was reached
successful_evals = list(filter(None, mapped_evals))
successful_evals = list(filter(None, eval_inds))
if not successful_evals:
single = self.evaluate_single(choice(individuals), with_time_limit=False)
if single:
Expand Down