Add example for EfficientNet and upgrade PyTorch in Dockerfile

deployment/docker/Dockerfile

@@ -68,8 +68,8 @@ RUN python3 -m pip --no-cache-dir install Keras==2.1.6
 #
 # PyTorch
 #
-RUN python3 -m pip --no-cache-dir install torch==0.4.1
-RUN python3 -m pip install torchvision==0.2.1
+RUN python3 -m pip --no-cache-dir install torch==1.2.0
+RUN python3 -m pip install torchvision==0.4.0
 
 #
 # sklearn 0.20.0

examples/trials/efficientnet/.gitignore

@@ -0,0 +1 @@
+EfficientNet-PyTorch

examples/trials/efficientnet/README.md

@@ -0,0 +1,37 @@
+# EfficientNet
+
+[arXiv](https://arxiv.org/abs/1905.11946)
+
+We provide here:
+
+* Search space and tuners for finding the best tuple (alpha, beta, gamma) for EfficientNet-B1 with
+  grid search, as discussed in Section 3.3 in [paper](https://arxiv.org/abs/1905.11946).
+* Search space for EfficientNet transfer learning on datasets other than ImageNet;
+  settings are borrowed from Appendix A.5 from 
+  "[Do Better ImageNet Models Transfer Better?](https://arxiv.org/abs/1805.08974)".
+
+## Instructions
+
+1. Clone our modified version of [EfficientNet-PyTorch](https://github.com/ultmaster/EfficientNet-PyTorch).
+   The modifications were done to adhere to the original [Tensorflow version](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet) 
+   as close as possible (including EMA, label smoothing and etc.); also added are the interactions with NNI.
+   Clone it into `EfficientNet-PyTorch`; then you will see files like `main.py`, `train_imagenet.sh`, as specified
+   in the configuration files.
+2. Resolve all the dependencies following the README.
+3. Run `nnictl create --config config_lr.yml` if you want to do transfer learning;
+   or `nnictl create --config config_net.yml` if you want to find the best EfficientNet-B1.
+   Adjust the training service (PAI/local/remote), batch size in the config files according to your environment.
+
+For training on ImageNet, you should read `EfficientNet-PyTorch/train_imagenet.sh`. Download ImageNet beforehand
+and extract it adhering to [PyTorch format](https://pytorch.org/docs/stable/torchvision/datasets.html#imagenet) 
+and then replace `/mnt/data/imagenet` in with the location of your ImageNet storage. This file should also be a good example
+in case you want to mount ImageNet into the container on OpenPAI.
+
+*Backward compatibility instructions: Grid Search quniform is changed in NNI v1.0. Use `search_net_old.json` if you are 
+still using NNI <= 0.9.* 
+
+## Results
+
+Our experiments show the relationship between acc@1 and alpha, beta, gamma.
+
+![](assets/search_result.png)

examples/trials/efficientnet/config_lr.yml

@@ -0,0 +1,24 @@
+authorName: unknown
+experimentName: lr_efficient_net
+trialConcurrency: 4
+maxExecDuration: 48h
+maxTrialNum: 140
+trainingServicePlatform: pai
+searchSpacePath: search_lr_old.json
+useAnnotation: false
+tuner:
+  builtinTunerName: GridSearch
+trial:
+  codeDir: EfficientNet-PyTorch
+  command: python3 main.py cifar10 -j 4 --epochs 20 -a efficientnet-b0 --pretrained --num-classes 10 --batch-size 64 --optimizer sgd --request-from-nni
+  cpuNum: 4
+  memoryMB: 16384
+  gpuNum: 1
+  shmMB: 8192
+  virtualCluster: nni
+  image: msranni/nni:latest
+nniManagerIp: <nni_manager_ip>
+paiConfig:
+  userName: <username>
+  passWord: <password>
+  host: <host>

examples/trials/efficientnet/config_net.yml

@@ -0,0 +1,28 @@
+authorName: unknown
+experimentName: example_efficient_net
+trialConcurrency: 8
+maxExecDuration: 48h
+maxTrialNum: 100
+trainingServicePlatform: pai
+searchSpacePath: search_net.json
+useAnnotation: false
+tuner:
+  codeDir: .
+  classFileName: tuner.py
+  className: FixedProductTuner
+  classArgs:
+    product: 2
+trial:
+  codeDir: EfficientNet-PyTorch
+  command: sh train_imagenet.sh
+  cpuNum: 4
+  memoryMB: 25000
+  shmMB: 25000
+  gpuNum: 1
+  virtualCluster: nni
+  image: msranni/nni:latest
+nniManagerIp: <nni_manager_ip>
+paiConfig:
+  userName: <username>
+  passWord: <password>
+  host: <host>

examples/trials/efficientnet/search_lr_old.json

@@ -0,0 +1,10 @@
+{
+  "lr": {
+    "_type": "qloguniform",
+    "_value": [0.00001, 0.1, 7]
+  },
+  "wd": {
+    "_type": "qloguniform",
+    "_value": [0.000001, 0.001, 7]
+  }
+}

examples/trials/efficientnet/search_net.json

@@ -0,0 +1,14 @@
+{
+  "alpha": {
+    "_type": "quniform",
+    "_value": [1.0, 2.0, 0.1]
+  },
+  "beta": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 0.1]
+  },
+  "gamma": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 0.1]
+  }
+}

examples/trials/efficientnet/search_net_old.json

@@ -0,0 +1,14 @@
+{
+  "alpha": {
+    "_type": "quniform",
+    "_value": [1.0, 2.0, 11]
+  },
+  "beta": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 11]
+  },
+  "gamma": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 11]
+  }
+}

examples/trials/efficientnet/tuner.py

@@ -0,0 +1,19 @@
+from nni.gridsearch_tuner.gridsearch_tuner import GridSearchTuner
+
+
+class FixedProductTuner(GridSearchTuner):
+
+    def __init__(self, product):
+        super().__init__()
+        self.product = product
+
+    def expand_parameters(self, para):
+        para = super().expand_parameters(para)
+        if all([key in para[0] for key in ["alpha", "beta", "gamma"]]):
+            ret_para = []
+            for p in para:
+                prod = p["alpha"] * (p["beta"] ** 2) * (p["gamma"] ** 2)
+                if abs(prod - self.product) < 0.1:
+                    ret_para.append(p)
+            return ret_para
+        return para