[Retiarii] Grid search, random and evolution strategy (#3377)

docs/en_US/NAS/retiarii/ApiReference.rst

@@ -72,10 +72,16 @@ Oneshot Trainers
 Strategies
 ----------
 
-..  autoclass:: nni.retiarii.strategies.RandomStrategy
+..  autoclass:: nni.retiarii.strategy.Random
     :members:
 
-..  autoclass:: nni.retiarii.strategies.TPEStrategy
+..  autoclass:: nni.retiarii.strategy.GridSearch
+    :members:
+
+..  autoclass:: nni.retiarii.strategy.RegularizedEvolution
+    :members:
+
+..  autoclass:: nni.retiarii.strategy.TPEStrategy
     :members:
 
 Retiarii Experiments

docs/en_US/NAS/retiarii/Tutorial.rst

@@ -167,13 +167,13 @@ In the following table, we listed the available trainers and strategies.
     - TPEStrategy
     - DartsTrainer
   * - Regression
-    - RandomStrategy
+    - Random
     - EnasTrainer
   * - 
-    - 
+    - GridSearch
     - ProxylessTrainer
   * - 
-    - 
+    - RegularizedEvolution
     - SinglePathTrainer (RandomTrainer)
 
 There usage and API document can be found `here <./ApiReference>`__\.
@@ -204,7 +204,7 @@ After all the above are prepared, it is time to start an experiment to do the mo
 
 .. code-block:: python
 
-  exp = RetiariiExperiment(base_model, trainer, applied_mutators, simple_startegy)
+  exp = RetiariiExperiment(base_model, trainer, applied_mutators, simple_strategy)
   exp_config = RetiariiExeConfig('local')
   exp_config.experiment_name = 'mnasnet_search'
   exp_config.trial_concurrency = 2

docs/en_US/NAS/retiarii/WriteStrategy.rst

@@ -3,10 +3,12 @@ Customize A New Strategy
 
 To write a new strategy, you should inherit the base strategy class ``BaseStrategy``, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
 
-Below is a very simple random strategy, the complete code can be found :githublink:`here <nni/retiarii/strategies/random_strategy.py>`.
+Below is a very simple random strategy, which makes the choices completely random.
 
 .. code-block:: python
 
+    from nni.retiarii import Sampler
+
     class RandomSampler(Sampler):
         def choice(self, candidates, mutator, model, index):
             return random.choice(candidates)
@@ -31,6 +33,6 @@ Below is a very simple random strategy, the complete code can be found :githubli
                 else:
                     time.sleep(2)
 
-You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategies/tpe_strategy.py>`.
+You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
 
 After generating a new model, the strategy can use our provided APIs (e.g., ``submit_models``, ``is_stopped_exec``) to submit the model and get its reported results. More APIs can be found in `API References <./ApiReference.rst>`__.

nni/retiarii/execution/base.py

@@ -65,11 +65,11 @@ def _send_trial_callback(self, paramater: dict) -> None:
         if self.resources <= 0:
             _logger.warning('There is no available resource, but trial is submitted.')
         self.resources -= 1
-        _logger.info('on_resource_used: %d', self.resources)
+        _logger.info('Resource used. Remaining: %d', self.resources)
 
     def _request_trial_jobs_callback(self, num_trials: int) -> None:
         self.resources += num_trials
-        _logger.info('on_resource_available: %d', self.resources)
+        _logger.info('New resource available. Remaining: %d', self.resources)
 
     def _trial_end_callback(self, trial_id: int, success: bool) -> None:
         model = self._running_models[trial_id]

nni/retiarii/experiment/pytorch.py

@@ -17,7 +17,7 @@
 from ..integration import RetiariiAdvisor
 from ..mutator import Mutator
 from ..nn.pytorch.mutator import process_inline_mutation
-from ..strategies.strategy import BaseStrategy
+from ..strategy import BaseStrategy
 from ..trainer.interface import BaseOneShotTrainer, BaseTrainer
 from ..utils import get_records
 

nni/retiarii/graph.py

@@ -131,7 +131,7 @@ def fork(self) -> 'Model':
         new_model = Model(_internal=True)
         new_model._root_graph_name = self._root_graph_name
         new_model.graphs = {name: graph._fork_to(new_model) for name, graph in self.graphs.items()}
-        new_model.training_config = copy.deepcopy(self.training_config)
+        new_model.training_config = copy.deepcopy(self.training_config)  # TODO this may be a problem when training config is large
         new_model.history = self.history + [self]
         return new_model
 

nni/retiarii/strategy/__init__.py

@@ -0,0 +1,4 @@
+from .base import BaseStrategy
+from .bruteforce import Random, GridSearch
+from .evolution import RegularizedEvolution
+from .tpe_strategy import TPEStrategy

nni/retiarii/strategy/bruteforce.py

@@ -0,0 +1,115 @@
+import copy
+import itertools
+import logging
+import random
+import time
+from typing import Any, Dict, List
+
+from .. import Sampler, submit_models, query_available_resources
+from .base import BaseStrategy
+from .utils import dry_run_for_search_space, get_targeted_model
+
+_logger = logging.getLogger(__name__)
+
+
+def grid_generator(search_space: Dict[Any, List[Any]], shuffle=True):
+    keys = list(search_space.keys())
+    search_space_values = copy.deepcopy(list(search_space.values()))
+    if shuffle:
+        for values in search_space_values:
+            random.shuffle(values)
+    for values in itertools.product(*search_space_values):
+        yield {key: value for key, value in zip(keys, values)}
+
+
+def random_generator(search_space: Dict[Any, List[Any]], dedup=True, retries=500):
+    keys = list(search_space.keys())
+    history = set()
+    search_space_values = copy.deepcopy(list(search_space.values()))
+    while True:
+        for retry_count in range(retries):
+            selected = [random.choice(v) for v in search_space_values]
+            if not dedup:
+                break
+            selected = tuple(selected)
+            if selected not in history:
+                history.add(selected)
+                break
+            if retry_count + 1 == retries:
+                _logger.info('Random generation has run out of patience. There is nothing to search. Exiting.')
+                return
+        yield {key: value for key, value in zip(keys, selected)}
+
+
+class GridSearch(BaseStrategy):
+    """
+    Traverse the search space and try all the possible combinations one by one.
+
+    Parameters
+    ----------
+    shuffle : bool
+        Shuffle the order in a candidate list, so that they are tried in a random order. Default: true.
+    """
+
+    def __init__(self, shuffle=True):
+        self._polling_interval = 2.
+        self.shuffle = shuffle
+
+    def run(self, base_model, applied_mutators):
+        search_space = dry_run_for_search_space(base_model, applied_mutators)
+        for sample in grid_generator(search_space, shuffle=self.shuffle):
+            _logger.info('New model created. Waiting for resource. %s', str(sample))
+            if query_available_resources() <= 0:
+                time.sleep(self._polling_interval)
+            submit_models(get_targeted_model(base_model, applied_mutators, sample))
+
+
+class _RandomSampler(Sampler):
+    def choice(self, candidates, mutator, model, index):
+        return random.choice(candidates)
+
+
+class Random(BaseStrategy):
+    """
+    Random search on the search space.
+
+    Parameters
+    ----------
+    variational : bool
+        Do not dry run to get the full search space. Used when the search space has variational size or candidates. Default: false.
+    dedup : bool
+        Do not try the same configuration twice. When variational is true, deduplication is not supported. Default: true.
+    """
+
+    def __init__(self, variational=False, dedup=True):
+        self.variational = variational
+        self.dedup = dedup
+        if variational and dedup:
+            raise ValueError('Dedup is not supported in variational mode.')
+        self.random_sampler = _RandomSampler()
+        self._polling_interval = 2.
+
+    def run(self, base_model, applied_mutators):
+        if self.variational:
+            _logger.info('Random search running in variational mode.')
+            sampler = _RandomSampler()
+            for mutator in applied_mutators:
+                mutator.bind_sampler(sampler)
+            while True:
+                avail_resource = query_available_resources()
+                if avail_resource > 0:
+                    model = base_model
+                    for mutator in applied_mutators:
+                        model = mutator.apply(model)
+                    _logger.info('New model created. Applied mutators are: %s', str(applied_mutators))
+                    submit_models(model)
+                else:
+                    time.sleep(self._polling_interval)
+        else:
+            _logger.info('Random search running in fixed size mode. Dedup: %s.', 'on' if self.dedup else 'off')
+            search_space = dry_run_for_search_space(base_model, applied_mutators)
+            for sample in random_generator(search_space, dedup=self.dedup):
+                _logger.info('New model created. Waiting for resource. %s', str(sample))
+                if query_available_resources() <= 0:
+                    time.sleep(self._polling_interval)
+                submit_models(get_targeted_model(base_model, applied_mutators, sample))