DNGO tuner #3479

98may · 2021-03-26T03:57:34Z

DNGO works as a NAS tuner
Scalable Bayesian Optimization Using Deep Neural Networks(DNGO): https://arxiv.org/pdf/1502.05700.pdf
Using DNGO model in https://github.com/automl/pybnn

how to test:
1. make sure codeDir in config.yml is correct
2. nnictl create -c examples/tuners/dngo_tuner/config.yml
search space:
support both numerical and string type(convert string to index as its value so that DNGO can get)
support choice/randint/uniform/quniform/loguniform/qloguniform(by reusing nni.parameter_expressions)

delete "break" for fast trail

examples/trials/mnist-pytorch/mnist.py

ultmaster · 2021-03-29T03:54:02Z

examples/tuners/dngo_tuner/dngo_tuner.py

+            raise ValueError('Unknown key %s and value %s' % (key, val))
+    return chosen_arch
+
+class DngoTuner(Tuner):


@QuanluZhang Is DNGO an example of customized tuner or a built-in tuner?

DNGO is meant to be a built-in tuner. Let me check how to make it a built-in tuner.

ultmaster · 2021-03-29T03:55:07Z

examples/tuners/dngo_tuner/pybnn/bayesian_linear_regression.py

+        return p0
+
+
+class BayesianLinearRegression(BaseModel):


Why can't we use bayesian regression from third-party library (e.g., sklearn) so that we don't have to maintain one on our own?

Can we make pybnn our dependency? https://github.com/automl/pybnn

We do not use the whole https://github.com/automl/pybnn, only the dngo-related part is included.
So I am not sure about "Can we make pybnn our dependency?"

I mean, I can pip install pybnn and import pybnn, rather than copy the whole pybnn into our repo.

ultmaster · 2021-03-29T03:56:52Z

Please add unittests for DNGO tuner.

revert an irrelevant change

98may

I have added the unit test. I will try it myself before update this PR.

examples/trials/mnist-pytorch/mnist.py

98may · 2021-05-14T02:28:32Z

finished the general update, but want to wait to test DNGO tuner with the help of Di Wu's benchmark by comparing its performance with other tuners, so that we can make sure the DNGO tuner works well.

warning has not been ignored, optional for to-do

ultmaster · 2021-05-21T09:46:56Z

nni/runtime/default_config/registered_algorithms.yml

@@ -76,3 +76,7 @@ tuners:
  classArgsValidator: nni.algorithms.hpo.regularized_evolution_tuner.EvolutionClassArgsValidator
  className: nni.algorithms.hpo.regularized_evolution_tuner.RegularizedEvolutionTuner
  source: nni
+  builtinName: DNGOuner


Missing -.

Are you sure this works?

ultmaster · 2021-05-21T09:47:41Z

nni/algorithms/hpo/dngo_tuner/dngo_tuner.py

+from nni.tuner import Tuner
+import nni.parameter_expressions as parameter_expressions
+from torch.distributions import Normal
+from pybnn import DNGO


We should add pybnn into an optional dependency required by DNGO. See how PPO and BOHB does that.

ok, will change it this weekend

ultmaster · 2021-05-21T09:48:45Z

nni/algorithms/hpo/dngo_tuner/dngo_tuner.py

@@ -0,0 +1,117 @@
+import random


I think a __init__.py is needed to make this importable.

ultmaster · 2021-05-21T09:51:04Z

nni/algorithms/hpo/dngo_tuner/search_space.json

+  "batch_size": {"_type":"choice", "_value": [16, 32, 64, 128]},
+  "hidden_size":{"_type":"choice","_value":[128, 256, 512, 1024]},
+  "lr":{"_type":"choice","_value":[0.0001, 0.001, 0.01, 0.1]},
+  "momentum":{"_type":"uniform","_value":[0, 1]}


Does DNGO tuner works on string choices for example ["apple", "banana", "orange"].

yes, technically it supports any string choices. But it's quite meaningless.

But I don't think it's handled in your code.

Have you tested?

ultmaster · 2021-05-24T05:59:04Z

Please fix the lint.

ultmaster · 2021-05-24T12:20:56Z

nni/runtime/default_config/registered_algorithms.yml

@@ -76,3 +76,7 @@ tuners:
  classArgsValidator: nni.algorithms.hpo.regularized_evolution_tuner.EvolutionClassArgsValidator
  className: nni.algorithms.hpo.regularized_evolution_tuner.RegularizedEvolutionTuner
  source: nni
+- builtinName: DNGOuner


Wrong spelling.

ultmaster · 2021-05-24T12:21:27Z

nni/algorithms/hpo/dngo_tuner/search_space.json

+  "batch_size": {"_type":"choice", "_value": [16, 32, 64, 128]},
+  "hidden_size":{"_type":"choice","_value":[128, 256, 512, 1024]},
+  "lr":{"_type":"choice","_value":[0.0001, 0.001, 0.01, 0.1]},
+  "momentum":{"_type":"uniform","_value":[0, 1]}


But I don't think it's handled in your code.

Have you tested?

ultmaster · 2021-05-24T12:21:44Z

nni/algorithms/hpo/dngo_tuner/config.yml

@@ -0,0 +1,18 @@
+authorName: Ayan Mao


Remove your name.

ultmaster · 2021-05-24T12:22:10Z

nni/algorithms/hpo/dngo_tuner/config.yml

+  className: DngoTuner
+  # Any parameter need to pass to your tuner class __init__ constructor
+  # can be specified in this optional classArgs field, for example
+trial:


I think this config should be put into example folder, and should be written in V2 format.

ultmaster · 2021-05-25T04:29:06Z

nni/algorithms/hpo/dngo_tuner/search_space.json

@@ -0,0 +1,6 @@
+{


This file should also be put into examples.

…aster

98may

I see that you have removed example config.yml and the "string" choice for search space, and added UT and docs. Big thanks:)

QuanluZhang · 2021-05-27T12:10:22Z

docs/en_US/Tuner/BuiltinTuner.rst

+
+
+* **optimize_mode** (*'maximize' or 'minimize'*\ ) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **sample_size** (*int, default = 1000*) - Number of samples to select in each iteration. The best one will be picked from the sample.


-> The best one will be picked from the sample as the next trial.

QuanluZhang · 2021-05-27T12:11:30Z

nni/algorithms/hpo/dngo_tuner.py

+
+
+def _random_config(search_space, random_state):
+    chosen_arch = {}


this tuner is not specific for nas, so better to change "arch" to "config"

missed that one. sorry.

AyanMao and others added 9 commits February 20, 2021 12:36

init dngo_darts, ok for tuner+trial

a216727

add readme, optimize config.yml

b6e9501

Merge branch 'master' of git://github.com/microsoft/nni

5ec623b

init dngo_general tuner

02e3e0b

test version done, ready for end-to-end

5ac8213

clean old dngo_darts

eada38d

rich search space option done

d12c1cb

Update mnist.py

2a8483e

delete "break" for fast trail

Update mnist.py

b906af8

SparkSnail mentioned this pull request Mar 26, 2021

NNI 2021 Mar~Apr Iteration Planning #3445

Closed

78 tasks

SparkSnail requested review from ultmaster and QuanluZhang March 26, 2021 08:52

ultmaster reviewed Mar 29, 2021

View reviewed changes

kvartet mentioned this pull request Apr 26, 2021

NNI 2021 May~June Iteration Planning #3581

Closed

49 tasks

ultmaster self-assigned this Apr 28, 2021

ultmaster added the HPO label Apr 28, 2021

update mnist.py

6089a0e

revert an irrelevant change

98may commented Apr 30, 2021

View reviewed changes

examples/trials/mnist-pytorch/mnist.py Outdated Show resolved Hide resolved

Merge branch 'microsoft:master' into master

f770cb6

98may marked this pull request as draft May 14, 2021 02:28

98may and others added 4 commits May 21, 2021 15:44

optimize generate_paramater function in dngo_tuner

74a5d7b

warning has not been ignored, optional for to-do

delete pybnn folder

833f45e

make dngo tuner a built-in tuner

d454288

move dngo_tuner folder into HPO

943e79c

98may marked this pull request as ready for review May 21, 2021 09:20

98may requested a review from ultmaster May 21, 2021 09:20

ultmaster reviewed May 21, 2021

View reviewed changes

better init.py

4fc9ad9

fix pylint

4103dca

ultmaster reviewed May 24, 2021

View reviewed changes

fix typo

4ad1966

ultmaster reviewed May 25, 2021

View reviewed changes

nni/algorithms/hpo/dngo_tuner/search_space.json Outdated

@@ -0,0 +1,6 @@

{

Copy link

Contributor

ultmaster May 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should also be put into examples.

ultmaster added 5 commits May 27, 2021 12:58

Refine tuner impl, add tests and docs

932fd25

Add dependency

05625b0

Merge branch 'master' of https://github.com/microsoft/nni into ayan-m…

2cd3c78

…aster

Update DNGO in dependency

441d6e2

Add requirement for DNGO tuner

b881f2c

ultmaster approved these changes May 27, 2021

View reviewed changes

98may commented May 27, 2021

View reviewed changes

QuanluZhang reviewed May 27, 2021

View reviewed changes

Minor fixes

a821c23

QuanluZhang approved these changes May 28, 2021

View reviewed changes

ultmaster merged commit 580c597 into microsoft:master May 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNGO tuner #3479

DNGO tuner #3479

98may commented Mar 26, 2021

ultmaster Mar 29, 2021

98may Apr 30, 2021

ultmaster Mar 29, 2021

ultmaster Mar 29, 2021

98may Apr 30, 2021

ultmaster Apr 30, 2021

ultmaster commented Mar 29, 2021

98may left a comment •

edited

Loading

98may commented May 14, 2021

ultmaster May 21, 2021

ultmaster May 21, 2021

98may May 21, 2021

ultmaster May 21, 2021

98may May 24, 2021

ultmaster May 21, 2021

98may May 24, 2021

ultmaster May 24, 2021

ultmaster commented May 24, 2021

ultmaster May 24, 2021

ultmaster May 24, 2021

ultmaster May 24, 2021

ultmaster May 24, 2021

QuanluZhang May 25, 2021

ultmaster May 25, 2021

98may left a comment

QuanluZhang May 27, 2021

ultmaster May 27, 2021

QuanluZhang May 27, 2021

ultmaster May 27, 2021



		* optimize_mode ('maximize' or 'minimize'\ ) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
		* sample_size (int, default = 1000) - Number of samples to select in each iteration. The best one will be picked from the sample.



		def _random_config(search_space, random_state):
		chosen_arch = {}

DNGO tuner #3479

DNGO tuner #3479

Conversation

98may commented Mar 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ultmaster commented Mar 29, 2021

98may left a comment • edited Loading

Choose a reason for hiding this comment

98may commented May 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ultmaster commented May 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

98may left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

98may left a comment •

edited

Loading