Add example for EfficientNet and upgrade PyTorch in Dockerfile (micro…

…soft#1497)

* Add example for EfficientNet and upgrade PyTorch in Dockerfile

deployment/docker/Dockerfile

@@ -68,8 +68,8 @@ RUN python3 -m pip --no-cache-dir install Keras==2.1.6
 #
 # PyTorch
 #
-RUN python3 -m pip --no-cache-dir install torch==0.4.1
-RUN python3 -m pip install torchvision==0.2.1
+RUN python3 -m pip --no-cache-dir install torch==1.2.0
+RUN python3 -m pip install torchvision==0.4.0
 
 #
 # sklearn 0.20.0

examples/trials/efficientnet/.gitignore

@@ -0,0 +1 @@
+EfficientNet-PyTorch

examples/trials/efficientnet/README.md

@@ -0,0 +1,19 @@
+# EfficientNet
+
+[EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946)
+
+Provided here are: Search space and tuners for finding the best tuple (alpha, beta, gamma) for EfficientNet-B1 with grid search, as discussed in Section 3.3 in [paper](https://arxiv.org/abs/1905.11946).
+
+## Instructions
+
+1. Set your working directory here in this directory.
+2. Run `git clone https://github.com/ultmaster/EfficientNet-PyTorch` to clone this modified version of [EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch). The modifications were done to adhere to the original [Tensorflow version](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet) as close as possible (including EMA, label smoothing and etc.); also added are the part which gets parameters from tuner and reports intermediate/final results. Clone it into `EfficientNet-PyTorch`; the files like `main.py`, `train_imagenet.sh` will appear inside, as specified in the configuration files.
+3. Run `nnictl create --config config_net.yml` to find the best EfficientNet-B1. Adjust the training service (PAI/local/remote), batch size in the config files according to the environment.
+
+For training on ImageNet, read `EfficientNet-PyTorch/train_imagenet.sh`. Download ImageNet beforehand and extract it adhering to [PyTorch format](https://pytorch.org/docs/stable/torchvision/datasets.html#imagenet) and then replace `/mnt/data/imagenet` in with the location of the ImageNet storage. This file should also be a good example to follow for mounting ImageNet into the container on OpenPAI.
+
+## Results
+
+The follow image is a screenshot, demonstrating the relationship between acc@1 and alpha, beta, gamma.
+
+![](assets/search_result.png)

examples/trials/efficientnet/config_net.yml

@@ -0,0 +1,28 @@
+authorName: unknown
+experimentName: example_efficient_net
+trialConcurrency: 8
+maxExecDuration: 48h
+maxTrialNum: 100
+trainingServicePlatform: pai
+searchSpacePath: search_net.json
+useAnnotation: false
+tuner:
+  codeDir: .
+  classFileName: tuner.py
+  className: FixedProductTuner
+  classArgs:
+    product: 2
+trial:
+  codeDir: EfficientNet-PyTorch
+  command: sh train_imagenet.sh
+  cpuNum: 4
+  memoryMB: 25000
+  shmMB: 25000
+  gpuNum: 1
+  virtualCluster: nni
+  image: msranni/nni:latest
+nniManagerIp: <nni_manager_ip>
+paiConfig:
+  userName: <username>
+  passWord: <password>
+  host: <host>

examples/trials/efficientnet/search_net.json

@@ -0,0 +1,14 @@
+{
+  "alpha": {
+    "_type": "quniform",
+    "_value": [1.0, 2.0, 0.1]
+  },
+  "beta": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 0.1]
+  },
+  "gamma": {
+    "_type": "quniform",
+    "_value": [1.0, 1.5, 0.1]
+  }
+}

examples/trials/efficientnet/tuner.py

@@ -0,0 +1,29 @@
+from nni.gridsearch_tuner.gridsearch_tuner import GridSearchTuner
+
+
+class FixedProductTuner(GridSearchTuner):
+    """
+    This tuner is essentially grid search, but it guarantees all the parameters with alpha * beta^2 * gamma^2 is
+    approximately `product`.
+    """
+
+    def __init__(self, product):
+        """
+        :param product: the constant provided, should be 2 in EfficientNet-B1
+        """
+        super().__init__()
+        self.product = product
+
+    def expand_parameters(self, para):
+        """
+        Filter out all qualified parameters
+        """
+        para = super().expand_parameters(para)
+        if all([key in para[0] for key in ["alpha", "beta", "gamma"]]):  # if this is an interested set
+            ret_para = []
+            for p in para:
+                prod = p["alpha"] * (p["beta"] ** 2) * (p["gamma"] ** 2)
+                if abs(prod - self.product) < 0.1:
+                    ret_para.append(p)
+            return ret_para
+        return para