Dev networkmorphism (#413)

* Quick fix nnictl config logic (#289)

* fix nnictl bug

* fix install.sh

* add desc for Dockerfile.build.base

* update document for Dockerfile

* update

* refactor port detect

* update

* refactor NNICTLDOC.md

* add document for pai and nnictl

* add default value for port

* add exception handling in trial_keeper.py

* fix port bug

* fix resume

* fix nnictl resume and fix nnictl stop

* fix document

* update

* refactor nnictl

* update

* update doc

* update

* update nnictl

* fix comment

* revert dockerfile

* update

* update

* update

* fix nnictl error hit

* fix comments

* fix bash-completion

* fix paramiko install

* quick fix resume logic

* update

* quick fix nnictl

* PR merge to 0.3 (#297)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* update doc

* fix links

* fix links

* fix links

* merge

* fix links and doc errors

* merge

* merge

* merge

* merge

* Update README.md (#288)

added License badge

* merge

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* fix doc mistakes and broken links. (#271)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* Update README.md

* Fix misspelling in examples/trials/ga_squad/README.md

* revise the installation cmd to v0.2

* revise to install v0.2

* remove enas readme (#292)

* Fix datastore performance issue (#301)

* Fix nnictl in v0.3 (#299)

Fix old version of config file
fix sklearn requirements
Fix resume log logic

* add basic tuner and trial for network morphism

* Complete basic receive_trial_result() and generate_parameters(). Use onnx  as the intermediate representation ( But it cannot convert to pytorch model )

* add tensorflow cifar10 for network morphism

* add unit test for tuner and its function

* use temporary torch_model

* fix request bug and program can communicate nni

* add basic pickle support for graph and train successful in pytorch

* Update unittest for networkmorphism_tuner

* Network Morphism add multi-gpu trial training support

* Format code with black tool

* change intermediate representation from pickle file to json we defined

* successfully pass the unittest for test_graph_json_transform

* add README for network morphism and it works fine in both Pytorch and Keras.

* separate the original Readme.md in network-morphism into two parts (tuner and trial)

* change the openpai image path

* beautify the file structure of network_morphism and add a fashion_mnist keras example

* pretty the source and add some docstring for funtion in order to pass the pylint.

* remove unused module import and add some docstring

* add some details for the application scenario Network Morphism Tuner

* follow the advice and modify the doc file

* add the config file for each task in the examples trial of network morphism

* change default python interpreter from python to python3

docs/HowToChooseTuner.md

@@ -10,6 +10,7 @@ For now, NNI has supported the following tuner algorithms. Note that NNI install
  - [Batch Tuner](#Batch)
  - [Grid Search](#Grid)
  - [Hyperband](#Hyperband)
+ - [Network Morphism](#NetworkMorphism)
 
  ## Supported tuner algorithms
 
@@ -22,11 +23,11 @@ We will introduce some basic knowledge about the tuner algorithms, suggested sce
 The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.
 The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. 
 This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization][1].
-    
+​    
 _Suggested scenario_: TPE, as a black-box optimization, can be used in various scenarios, and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: TPE
@@ -43,7 +44,7 @@ In [Random Search for Hyper-Parameter Optimization][2] show that Random Search m
 _Suggested scenario_: Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Random
@@ -57,7 +58,7 @@ This simple annealing algorithm begins by sampling from the prior, but tends ove
 _Suggested scenario_: Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Anneal
@@ -74,7 +75,7 @@ Naive Evolution comes from [Large-Scale Evolution of Image Classifiers][3]. It r
 _Suggested scenario_: Its requirement of computation resource is relatively high. Specifically, it requires large inital population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Evolution
@@ -93,7 +94,7 @@ Note that SMAC on nni only supports a subset of the types in [search space spec]
 _Suggested scenario_: Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: SMAC
@@ -110,14 +111,14 @@ Batch tuner allows users to simply provide several configurations (i.e., choices
 _Suggested sceanrio_: If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: BatchTuner
 ```
 
 Note that the search space that BatchTuner supported like:
-```
+```json
 {
     "combine_params":
     {
@@ -133,7 +134,6 @@ Note that the search space that BatchTuner supported like:
 ```
 The search space file including the high-level key `combine_params`. The type of params in search space must be `choice` and the `values` including all the combined-params value.
 
-
 <a name="Grid"></a>
 **Grid Search**
 
@@ -143,7 +143,7 @@ Note that the only acceptable types of search space are `choice`, `quniform`, `q
 _Suggested scenario_: It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: GridSearch
@@ -152,12 +152,12 @@ _Usage_:
 <a name="Hyperband"></a>
 **Hyperband**
 
-[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be refered to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md)
+[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be referred to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md).
 
 _Suggested scenario_: It is suggested when you have limited computation resource but have relatively large search space. It performs good in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   advisor:
     builtinAdvisorName: Hyperband
@@ -171,6 +171,31 @@ _Usage_:
       eta: 3
 ```
 
+<a name="NetworkMorphism"></a>
+**Network Morphism**
+
+[Network Morphism](7) provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md). 
+
+_Suggested scenario_: It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](../examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the cv domain.
+
+_Usage_:
+```yaml
+  # config.yaml
+  tuner:
+    builtinTunerName: NetworkMorphism
+    classArgs:
+      #choice: maximize, minimize
+      optimize_mode: maximize
+      #for now, this tuner only supports cv domain
+      task: cv
+      #input image width
+      input_width: 32
+      #input image channel
+      input_channel: 3
+      #number of classes
+      n_output_node: 10
+```
+
 
 # How to use Assessor that NNI supports?
 
@@ -184,12 +209,12 @@ For now, NNI has supported the following assessor algorithms.
 <a name="Medianstop"></a>
 **Medianstop**
 
-Medianstop is a simple early stopping rule mentioned in the [paper][7]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
+Medianstop is a simple early stopping rule mentioned in the [paper][8]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
 
 _Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
 
 _Usage_:
-```
+```yaml
   assessor:
     builtinAssessorName: Medianstop
     classArgs:
@@ -201,10 +226,11 @@ _Usage_:
       start_step: 5
 ```
 
-  [1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
-  [2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
-  [3]: https://arxiv.org/pdf/1703.01041.pdf
-  [4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
-  [5]: https://github.com/automl/SMAC3
-  [6]: https://arxiv.org/pdf/1603.06560.pdf
-  [7]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
\ No newline at end of file
+[1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
+[2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
+[3]: https://arxiv.org/pdf/1703.01041.pdf
+[4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
+[5]: https://github.com/automl/SMAC3
+[6]: https://arxiv.org/pdf/1603.06560.pdf
+[7]: https://arxiv.org/abs/1806.10282
+[8]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
\ No newline at end of file

examples/trials/ga_squad/README.md

@@ -159,7 +159,7 @@ The trial has a lot of different files, functions and classes. Here we will only
 
 Among those files, `trial.py` and `graph_to_tf.py` is special.
 
-`graph_to_tf.py` has a function named as `graph_to_network`, here is its skelton code:
+`graph_to_tf.py` has a function named as `graph_to_network`, here is its skeleton code:
 
 ```
 def graph_to_network(input1,

examples/trials/network_morphism/FashionMNIST/FashionMNIST_keras.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import argparse
+import logging
+import os
+
+import tensorflow as tf
+import keras
+from keras.callbacks import EarlyStopping, TensorBoard
+from keras.datasets import fashion_mnist
+from keras.optimizers import SGD, Adadelta, Adagrad, Adam, Adamax, RMSprop
+from keras.utils import multi_gpu_model, to_categorical
+import keras.backend.tensorflow_backend as KTF
+
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+# set the logger format
+log_format = "%(asctime)s %(message)s"
+logging.basicConfig(
+    filename="networkmorphism.log",
+    filemode="a",
+    level=logging.INFO,
+    format=log_format,
+    datefmt="%m/%d %I:%M:%S %p",
+)
+# set the logger format
+logger = logging.getLogger("fashion_mnist-network-morphism-keras")
+
+
+# restrict gpu usage background
+config = tf.ConfigProto()
+# pylint: disable=E1101,W0603
+config.gpu_options.allow_growth = True
+sess = tf.Session(config=config)
+
+KTF.set_session(sess)
+
+
+def get_args():
+    """ get args from command line
+    """
+    parser = argparse.ArgumentParser("fashion_mnist")
+    parser.add_argument("--batch_size", type=int, default=128, help="batch size")
+    parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
+    parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
+    parser.add_argument(
+        "--learning_rate", type=float, default=0.001, help="learning rate"
+    )
+    parser.add_argument(
+        "--weight_decay",
+        type=float,
+        default=1e-5,
+        help="weight decay of the learning rate",
+    )
+    return parser.parse_args()
+
+
+trainloader = None
+testloader = None
+net = None
+args = get_args()
+TENSORBOARD_DIR = os.environ["NNI_OUTPUT_DIR"]
+
+
+def build_graph_from_json(ir_model_json):
+    """build model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    logging.debug(graph.operation_history)
+    model = graph.produce_keras_model()
+    return model
+
+
+def parse_rev_args(receive_msg):
+    """ parse reveive msgs to global variable
+    """
+    global trainloader
+    global testloader
+    global net
+
+    # Loading Data
+    logger.debug("Preparing data..")
+
+    (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
+    y_train = to_categorical(y_train, 10)
+    y_test = to_categorical(y_test, 10)
+    x_train = x_train.reshape(x_train.shape+(1,)).astype("float32")
+    x_test = x_test.reshape(x_test.shape+(1,)).astype("float32")
+    x_train /= 255.0
+    x_test /= 255.0
+    trainloader = (x_train, y_train)
+    testloader = (x_test, y_test)
+
+    # Model
+    logger.debug("Building model..")
+    net = build_graph_from_json(receive_msg)
+
+    # parallel model
+    try:
+        available_devices = os.environ["CUDA_VISIBLE_DEVICES"]
+        gpus = len(available_devices.split(","))
+        if gpus > 1:
+            net = multi_gpu_model(net, gpus)
+    except KeyError:
+        logger.debug("parallel model not support in this config settings")
+
+    if args.optimizer == "SGD":
+        optimizer = SGD(lr=args.learning_rate, momentum=0.9, decay=args.weight_decay)
+    if args.optimizer == "Adadelta":
+        optimizer = Adadelta(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adagrad":
+        optimizer = Adagrad(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adam":
+        optimizer = Adam(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adamax":
+        optimizer = Adamax(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "RMSprop":
+        optimizer = RMSprop(lr=args.learning_rate, decay=args.weight_decay)
+
+    # Compile the model
+    net.compile(
+        loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"]
+    )
+    return 0
+
+
+class SendMetrics(keras.callbacks.Callback):
+    """
+    Keras callback to send metrics to NNI framework
+    """
+
+    def on_epoch_end(self, epoch, logs=None):
+        """
+        Run on end of each epoch
+        """
+        if logs is None:
+            logs = dict()
+        logger.debug(logs)
+        nni.report_intermediate_result(logs["acc"])
+
+
+# Training
+def train_eval():
+    """ train and eval the model
+    """
+
+    global trainloader
+    global testloader
+    global net
+
+    (x_train, y_train) = trainloader
+    (x_test, y_test) = testloader
+
+    # train procedure
+    net.fit(
+        x=x_train,
+        y=y_train,
+        batch_size=args.batch_size,
+        validation_data=(x_test, y_test),
+        epochs=args.epochs,
+        shuffle=True,
+        callbacks=[
+            SendMetrics(),
+            EarlyStopping(min_delta=0.001, patience=10),
+            TensorBoard(log_dir=TENSORBOARD_DIR),
+        ],
+    )
+
+    # trial report final acc to tuner
+    _, acc = net.evaluate(x_test, y_test)
+    logger.debug("Final result is: %d", acc)
+    nni.report_final_result(acc)
+
+
+if __name__ == "__main__":
+    try:
+        # trial get next parameter from network morphism tuner
+        RCV_CONFIG = nni.get_next_parameter()
+        logger.debug(RCV_CONFIG)
+        parse_rev_args(RCV_CONFIG)
+        train_eval()
+    except Exception as exception:
+        logger.exception(exception)
+        raise

examples/trials/network_morphism/FashionMNIST/FashionMNIST_pytorch.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import argparse
+import logging
+import sys
+
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+import torchvision
+
+sys.path.append("../")
+from network_morphism import utils
+
+
+# set the logger format
+log_format = "%(asctime)s %(message)s"
+logging.basicConfig(
+    filename="networkmorphism.log",
+    filemode="a",
+    level=logging.INFO,
+    format=log_format,
+    datefmt="%m/%d %I:%M:%S %p",
+)
+# pylint: disable=W0603
+# set the logger format
+logger = logging.getLogger("FashionMNIST-network-morphism")
+
+
+def get_args():
+    """ get args from command line
+    """
+    parser = argparse.ArgumentParser("FashionMNIST")
+    parser.add_argument("--batch_size", type=int, default=128, help="batch size")
+    parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
+    parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
+    parser.add_argument(
+        "--learning_rate", type=float, default=0.001, help="learning rate"
+    )
+    parser.add_argument("--cutout", action="store_true", default=False, help="use cutout")
+    parser.add_argument("--cutout_length", type=int, default=8, help="cutout length")
+    parser.add_argument(
+        "--model_path", type=str, default="./", help="Path to save the destination model"
+    )
+    return parser.parse_args()
+
+
+trainloader = None
+testloader = None
+net = None
+criterion = None
+optimizer = None
+device = "cuda" if torch.cuda.is_available() else "cpu"
+best_acc = 0.0
+args = get_args()
+
+
+def build_graph_from_json(ir_model_json):
+    """build model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    logging.debug(graph.operation_history)
+    model = graph.produce_torch_model()
+    return model
+
+
+def parse_rev_args(receive_msg):
+    """ parse reveive msgs to global variable
+    """
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    # Loading Data
+    logger.debug("Preparing data..")
+
+    raw_train_data = torchvision.datasets.FashionMNIST(
+        root="./data", train=True, download=True
+    )
+
+    dataset_mean, dataset_std = (
+        [raw_train_data.train_data.float().mean() / 255],
+        [raw_train_data.train_data.float().std() / 255],
+    )
+
+    transform_train, transform_test = utils.data_transforms_mnist(
+        args, dataset_mean, dataset_std
+    )
+
+    trainset = torchvision.datasets.FashionMNIST(
+        root="./data", train=True, download=True, transform=transform_train
+    )
+    trainloader = torch.utils.data.DataLoader(
+        trainset, batch_size=args.batch_size, shuffle=True, num_workers=2
+    )
+
+    testset = torchvision.datasets.FashionMNIST(
+        root="./data", train=False, download=True, transform=transform_test
+    )
+    testloader = torch.utils.data.DataLoader(
+        testset, batch_size=args.batch_size, shuffle=False, num_workers=2
+    )
+
+    # Model
+    logger.debug("Building model..")
+    net = build_graph_from_json(receive_msg)
+
+    net = net.to(device)
+    criterion = nn.CrossEntropyLoss()
+
+    if args.optimizer == "SGD":
+        optimizer = optim.SGD(
+            net.parameters(), lr=args.learning_rate, momentum=0.9, weight_decay=5e-4
+        )
+    if args.optimizer == "Adadelta":
+        optimizer = optim.Adadelta(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adagrad":
+        optimizer = optim.Adagrad(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adam":
+        optimizer = optim.Adam(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adamax":
+        optimizer = optim.Adamax(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "RMSprop":
+        optimizer = optim.RMSprop(net.parameters(), lr=args.learning_rate)
+
+    return 0
+
+
+# Training
+def train(epoch):
+    """ train model on each epoch in trainset
+    """
+
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    logger.debug("Epoch: %d", epoch)
+    net.train()
+    train_loss = 0
+    correct = 0
+    total = 0
+
+    for batch_idx, (inputs, targets) in enumerate(trainloader):
+        inputs, targets = inputs.to(device), targets.to(device)
+        optimizer.zero_grad()
+        outputs = net(inputs)
+        loss = criterion(outputs, targets)
+        loss.backward()
+        optimizer.step()
+
+        train_loss += loss.item()
+        _, predicted = outputs.max(1)
+        total += targets.size(0)
+        correct += predicted.eq(targets).sum().item()
+
+        acc = 100.0 * correct / total
+
+        logger.debug(
+            "Loss: %.3f | Acc: %.3f%% (%d/%d)",
+            train_loss / (batch_idx + 1),
+            100.0 * correct / total,
+            correct,
+            total,
+        )
+
+    return acc
+
+
+def test(epoch):
+    """ eval model on each epoch in testset
+    """
+    global best_acc
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    logger.debug("Eval on epoch: %d", epoch)
+    net.eval()
+    test_loss = 0
+    correct = 0
+    total = 0
+    with torch.no_grad():
+        for batch_idx, (inputs, targets) in enumerate(testloader):
+            inputs, targets = inputs.to(device), targets.to(device)
+            outputs = net(inputs)
+            loss = criterion(outputs, targets)
+
+            test_loss += loss.item()
+            _, predicted = outputs.max(1)
+            total += targets.size(0)
+            correct += predicted.eq(targets).sum().item()
+
+            acc = 100.0 * correct / total
+
+            logger.debug(
+                "Loss: %.3f | Acc: %.3f%% (%d/%d)",
+                test_loss / (batch_idx + 1),
+                100.0 * correct / total,
+                correct,
+                total,
+            )
+
+    acc = 100.0 * correct / total
+    if acc > best_acc:
+        best_acc = acc
+    return acc, best_acc
+
+
+if __name__ == "__main__":
+    try:
+        # trial get next parameter from network morphism tuner
+        RCV_CONFIG = nni.get_next_parameter()
+        logger.debug(RCV_CONFIG)
+
+        parse_rev_args(RCV_CONFIG)
+        train_acc = 0.0
+        best_acc = 0.0
+        early_stop = utils.EarlyStopping(mode="max")
+        for ep in range(args.epochs):
+            train_acc = train(ep)
+            test_acc, best_acc = test(ep)
+            nni.report_intermediate_result(test_acc)
+            logger.debug(test_acc)
+            if early_stop.step(test_acc):
+                break
+
+        # trial report best_acc to tuner
+        nni.report_final_result(best_acc)
+    except Exception as exception:
+        logger.exception(exception)
+        raise

examples/trials/network_morphism/FashionMNIST/config.yml

+authorName: default
+experimentName: example_FashionMNIST-network-morphism
+trialConcurrency: 4
+maxExecDuration: 48h
+maxTrialNum: 200
+#choice: local, remote, pai
+trainingServicePlatform: local
+#searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
+  #SMAC (SMAC should be installed through nnictl) 
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    #for now, this tuner only supports cv domain
+    task: cv
+    #input image width
+    input_width: 28
+    #input image channel
+    input_channel: 1
+    #number of classes
+    n_output_node: 10
+trial:
+  command: python3 FashionMNIST_keras.py
+  codeDir: .
+  gpuNum: 1

examples/trials/network_morphism/FashionMNIST/config_pai.yml

+authorName: default
+experimentName: example_FashionMNIST-network-morphism
+trialConcurrency: 1
+maxExecDuration: 24h
+maxTrialNum: 10
+#choice: local, remote, pai
+trainingServicePlatform: pai
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
+  #SMAC (SMAC should be installed through nnictl) 
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    # for now, this tuner only supports cv domain
+    task: cv
+    #input image width
+    input_width: 28
+    #input image channel
+    input_channel: 1
+    #number of classes
+    n_output_node: 10
+trial:
+  command: python3 FashionMNIST_keras.py
+  codeDir: .
+  gpuNum: 1
+  cpuNum: 1
+  memoryMB: 8196
+  #The docker image to run nni job on pai
+  image: msranni/nni:latest
+  #The hdfs directory to store data on pai, format 'hdfs://host:port/directory'
+  dataDir: hdfs://10.10.10.10:9000/username/nni
+  #The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
+  outputDir: hdfs://10.10.10.10:9000/username/nni
+paiConfig:
+  #The username to login pai
+  userName: username
+  #The password to login pai
+  passWord: password
+  #The host of restful server of pai
+  host: 10.10.10.10
\ No newline at end of file

examples/trials/network_morphism/README.md

+# Network Morphism for Automatic Model Architecture Search in NNI
+The Network Morphism is a build-in Tuner using network morphism techniques to search and evaluate the new network architecture. This example shows us how to use it to find good model architectures for deep learning.
+
+## How to run this example?
+
+### 1. Training framework support
+
+The network morphism now is framework-based, and we have not implemented the framework-free methods. The training frameworks which we have supported yet are Pytorch and Keras. If you get familiar with the intermediate JSON format, you can build your own model in your own training framework. In the future, we will change to intermediate format from JSON to ONNX in order to get a [standard intermediate representation spec](https://github.com/onnx/onnx/blob/master/docs/IR.md).
+
+
+### 2. Install the requirements 
+
+```bash
+# install the requirements packages
+cd examples/trials/network_morphism/
+pip install -r requirements.txt
+```
+
+### 3. Update configuration
+
+Modify `examples/trials/network_morphism/cifar10/config.yaml` to fit your own task, note that searchSpacePath is not required in our configuration. Here is the default configuration:
+
+```yaml
+authorName: default
+experimentName: example_cifar10-network-morphism
+trialConcurrency: 1
+maxExecDuration: 48h
+maxTrialNum: 200
+#choice: local, remote, pai
+trainingServicePlatform: local
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
+  #SMAC (SMAC should be installed through nnictl) 
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    #for now, this tuner only supports cv domain
+    task: cv
+    #modify to fit your input image width
+    input_width: 32
+    #modify to fit your input image channel
+    input_channel: 3
+    #modify to fit your number of classes
+    n_output_node: 10
+trial:
+  # your own command here
+  command: python3 cifar10_keras.py
+  codeDir: .
+  gpuNum: 0
+```
+
+In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
+
+`trialConcurrency` is the number of trials running concurrently, which is the number of GPUs you want to use, if you are setting `gpuNum` to 1.
+
+### 4. Call "json\_to\_graph()" function in your own code
+
+Modify your code and call "json\_to\_graph()" function to build a pytorch model or keras model from received json  string. Here is the simple example.
+
+```python
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+def build_graph_from_json(ir_model_json):
+    """build a pytorch model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    model = graph.produce_torch_model()
+    return model
+
+# trial get next parameter from network morphism tuner
+RCV_CONFIG = nni.get_next_parameter()
+# call the function to build pytorch model or keras model
+net = build_graph_from_json(RCV_CONFIG)
+
+# training procedure
+# ....
+
+# report the final accuracy to nni
+nni.report_final_result(best_acc)
+```
+
+### 5. Submit this job 
+
+```bash
+# You can use nni command tool "nnictl" to create the a job which submit to the nni
+# finally you successfully commit a Network Morphism Job to nni
+nnictl create --config config.yaml
+```
+
+## Trial Examples
+
+The trial has some examples which can guide you which located in `examples/trials/network_morphism/`. You can refer to it and modify to your own task. Hope this will help you to build your code.
+
+### FashionMNIST
+
+`Fashion-MNIST` is a dataset of [Zalando](https://jobs.zalando.com/tech/)'s article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. It is a modern image classification dataset widely used to replacing MNIST as a baseline dataset, because the dataset MNIST is too easy and overused.
+
+There are two examples, [FashionMNIST-keras.py](./FashionMNIST/FashionMNIST_keras.py) and [FashionMNIST-pytorch.py](./FashionMNIST/FashionMNIST_pytorch.py). Attention, you should change the `input_width` to 28 and `input_channel` to 1 in `config.yaml ` for this dataset.
+
+### Cifar10
+
+The `CIFAR-10` dataset [Canadian Institute For Advanced Research](https://www.cifar.ca/) is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10 different classes.
+
+There are two examples, [cifar10-keras.py](./cifar10/cifar10_keras.py) and [cifar10-pytorch.py](./cifar10/cifar10_pytorch.py). The value `input_width` is 32 and the value `input_channel` is 3 in `config.yaml ` for this dataset.
\ No newline at end of file

examples/trials/network_morphism/cifar10/cifar10_keras.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import argparse
+import logging
+import os
+
+import tensorflow as tf
+import keras
+from keras.callbacks import EarlyStopping, TensorBoard
+from keras.datasets import cifar10
+from keras.optimizers import SGD, Adadelta, Adagrad, Adam, Adamax, RMSprop
+from keras.utils import multi_gpu_model, to_categorical
+import keras.backend.tensorflow_backend as KTF
+
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+# set the logger format
+log_format = "%(asctime)s %(message)s"
+logging.basicConfig(
+    filename="networkmorphism.log",
+    filemode="a",
+    level=logging.INFO,
+    format=log_format,
+    datefmt="%m/%d %I:%M:%S %p",
+)
+# set the logger format
+logger = logging.getLogger("cifar10-network-morphism-keras")
+
+
+# restrict gpu usage background
+config = tf.ConfigProto()
+# pylint: disable=E1101,W0603
+config.gpu_options.allow_growth = True
+sess = tf.Session(config=config)
+
+KTF.set_session(sess)
+
+
+def get_args():
+    """ get args from command line
+    """
+    parser = argparse.ArgumentParser("cifar10")
+    parser.add_argument("--batch_size", type=int, default=128, help="batch size")
+    parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
+    parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
+    parser.add_argument(
+        "--learning_rate", type=float, default=0.001, help="learning rate"
+    )
+    parser.add_argument(
+        "--weight_decay",
+        type=float,
+        default=1e-5,
+        help="weight decay of the learning rate",
+    )
+    return parser.parse_args()
+
+
+trainloader = None
+testloader = None
+net = None
+args = get_args()
+TENSORBOARD_DIR = os.environ["NNI_OUTPUT_DIR"]
+
+
+def build_graph_from_json(ir_model_json):
+    """build model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    logging.debug(graph.operation_history)
+    model = graph.produce_keras_model()
+    return model
+
+
+def parse_rev_args(receive_msg):
+    """ parse reveive msgs to global variable
+    """
+    global trainloader
+    global testloader
+    global net
+
+    # Loading Data
+    logger.debug("Preparing data..")
+
+    (x_train, y_train), (x_test, y_test) = cifar10.load_data()
+    y_train = to_categorical(y_train, 10)
+    y_test = to_categorical(y_test, 10)
+    x_train = x_train.astype("float32")
+    x_test = x_test.astype("float32")
+    x_train /= 255.0
+    x_test /= 255.0
+    trainloader = (x_train, y_train)
+    testloader = (x_test, y_test)
+
+    # Model
+    logger.debug("Building model..")
+    net = build_graph_from_json(receive_msg)
+
+    # parallel model
+    try:
+        available_devices = os.environ["CUDA_VISIBLE_DEVICES"]
+        gpus = len(available_devices.split(","))
+        if gpus > 1:
+            net = multi_gpu_model(net, gpus)
+    except KeyError:
+        logger.debug("parallel model not support in this config settings")
+
+    if args.optimizer == "SGD":
+        optimizer = SGD(lr=args.learning_rate, momentum=0.9, decay=args.weight_decay)
+    if args.optimizer == "Adadelta":
+        optimizer = Adadelta(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adagrad":
+        optimizer = Adagrad(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adam":
+        optimizer = Adam(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "Adamax":
+        optimizer = Adamax(lr=args.learning_rate, decay=args.weight_decay)
+    if args.optimizer == "RMSprop":
+        optimizer = RMSprop(lr=args.learning_rate, decay=args.weight_decay)
+
+    # Compile the model
+    net.compile(
+        loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"]
+    )
+    return 0
+
+
+class SendMetrics(keras.callbacks.Callback):
+    """
+    Keras callback to send metrics to NNI framework
+    """
+
+    def on_epoch_end(self, epoch, logs=None):
+        """
+        Run on end of each epoch
+        """
+        if logs is None:
+            logs = dict()
+        logger.debug(logs)
+        nni.report_intermediate_result(logs["acc"])
+
+
+# Training
+def train_eval():
+    """ train and eval the model
+    """
+
+    global trainloader
+    global testloader
+    global net
+
+    (x_train, y_train) = trainloader
+    (x_test, y_test) = testloader
+
+    # train procedure
+    net.fit(
+        x=x_train,
+        y=y_train,
+        batch_size=args.batch_size,
+        validation_data=(x_test, y_test),
+        epochs=args.epochs,
+        shuffle=True,
+        callbacks=[
+            SendMetrics(),
+            EarlyStopping(min_delta=0.001, patience=10),
+            TensorBoard(log_dir=TENSORBOARD_DIR),
+        ],
+    )
+
+    # trial report final acc to tuner
+    _, acc = net.evaluate(x_test, y_test)
+    logger.debug("Final result is: %d", acc)
+    nni.report_final_result(acc)
+
+
+if __name__ == "__main__":
+    try:
+        # trial get next parameter from network morphism tuner
+        RCV_CONFIG = nni.get_next_parameter()
+        logger.debug(RCV_CONFIG)
+        parse_rev_args(RCV_CONFIG)
+        train_eval()
+    except Exception as exception:
+        logger.exception(exception)
+        raise

examples/trials/network_morphism/cifar10/cifar10_pytorch.py

+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import argparse
+import logging
+import sys
+
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+import torchvision
+
+sys.path.append("../")
+from network_morphism import utils
+
+# set the logger format
+log_format = "%(asctime)s %(message)s"
+logging.basicConfig(
+    filename="networkmorphism.log",
+    filemode="a",
+    level=logging.INFO,
+    format=log_format,
+    datefmt="%m/%d %I:%M:%S %p",
+)
+# pylint: disable=W0603
+# set the logger format
+logger = logging.getLogger("cifar10-network-morphism-pytorch")
+
+
+def get_args():
+    """ get args from command line
+    """
+    parser = argparse.ArgumentParser("cifar10")
+    parser.add_argument("--batch_size", type=int, default=128, help="batch size")
+    parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
+    parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
+    parser.add_argument(
+        "--learning_rate", type=float, default=0.001, help="learning rate"
+    )
+    parser.add_argument("--cutout", action="store_true", default=False, help="use cutout")
+    parser.add_argument("--cutout_length", type=int, default=8, help="cutout length")
+    parser.add_argument(
+        "--model_path", type=str, default="./", help="Path to save the destination model"
+    )
+    return parser.parse_args()
+
+
+trainloader = None
+testloader = None
+net = None
+criterion = None
+optimizer = None
+device = "cuda" if torch.cuda.is_available() else "cpu"
+best_acc = 0.0
+args = get_args()
+
+
+def build_graph_from_json(ir_model_json):
+    """build model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    logging.debug(graph.operation_history)
+    model = graph.produce_torch_model()
+    return model
+
+
+def parse_rev_args(receive_msg):
+    """ parse reveive msgs to global variable
+    """
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    # Loading Data
+    logger.debug("Preparing data..")
+
+    transform_train, transform_test = utils.data_transforms_cifar10(args)
+
+    trainset = torchvision.datasets.CIFAR10(
+        root="./data", train=True, download=True, transform=transform_train
+    )
+    trainloader = torch.utils.data.DataLoader(
+        trainset, batch_size=args.batch_size, shuffle=True, num_workers=2
+    )
+
+    testset = torchvision.datasets.CIFAR10(
+        root="./data", train=False, download=True, transform=transform_test
+    )
+    testloader = torch.utils.data.DataLoader(
+        testset, batch_size=args.batch_size, shuffle=False, num_workers=2
+    )
+
+    # Model
+    logger.debug("Building model..")
+    net = build_graph_from_json(receive_msg)
+
+    net = net.to(device)
+    criterion = nn.CrossEntropyLoss()
+    if device == "cuda" and torch.cuda.device_count() > 1:
+        net = torch.nn.DataParallel(net)
+
+    if args.optimizer == "SGD":
+        optimizer = optim.SGD(
+            net.parameters(), lr=args.learning_rate, momentum=0.9, weight_decay=5e-4
+        )
+    if args.optimizer == "Adadelta":
+        optimizer = optim.Adadelta(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adagrad":
+        optimizer = optim.Adagrad(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adam":
+        optimizer = optim.Adam(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "Adamax":
+        optimizer = optim.Adamax(net.parameters(), lr=args.learning_rate)
+    if args.optimizer == "RMSprop":
+        optimizer = optim.RMSprop(net.parameters(), lr=args.learning_rate)
+
+
+    return 0
+
+
+# Training
+def train(epoch):
+    """ train model on each epoch in trainset
+    """
+
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    logger.debug("Epoch: %d", epoch)
+    net.train()
+    train_loss = 0
+    correct = 0
+    total = 0
+
+    for batch_idx, (inputs, targets) in enumerate(trainloader):
+        inputs, targets = inputs.to(device), targets.to(device)
+        optimizer.zero_grad()
+        outputs = net(inputs)
+        loss = criterion(outputs, targets)
+        loss.backward()
+        optimizer.step()
+
+        train_loss += loss.item()
+        _, predicted = outputs.max(1)
+        total += targets.size(0)
+        correct += predicted.eq(targets).sum().item()
+
+        acc = 100.0 * correct / total
+
+        logger.debug(
+            "Loss: %.3f | Acc: %.3f%% (%d/%d)",
+            train_loss / (batch_idx + 1),
+            100.0 * correct / total,
+            correct,
+            total,
+        )
+
+    return acc
+
+
+def test(epoch):
+    """ eval model on each epoch in testset
+    """
+    global best_acc
+    global trainloader
+    global testloader
+    global net
+    global criterion
+    global optimizer
+
+    logger.debug("Eval on epoch: %d", epoch)
+    net.eval()
+    test_loss = 0
+    correct = 0
+    total = 0
+    with torch.no_grad():
+        for batch_idx, (inputs, targets) in enumerate(testloader):
+            inputs, targets = inputs.to(device), targets.to(device)
+            outputs = net(inputs)
+            loss = criterion(outputs, targets)
+
+            test_loss += loss.item()
+            _, predicted = outputs.max(1)
+            total += targets.size(0)
+            correct += predicted.eq(targets).sum().item()
+
+            acc = 100.0 * correct / total
+
+            logger.debug(
+                "Loss: %.3f | Acc: %.3f%% (%d/%d)",
+                test_loss / (batch_idx + 1),
+                100.0 * correct / total,
+                correct,
+                total,
+            )
+
+    acc = 100.0 * correct / total
+    if acc > best_acc:
+        best_acc = acc
+    return acc, best_acc
+
+
+if __name__ == "__main__":
+    try:
+        # trial get next parameter from network morphism tuner
+        RCV_CONFIG = nni.get_next_parameter()
+        logger.debug(RCV_CONFIG)
+
+        parse_rev_args(RCV_CONFIG)
+        train_acc = 0.0
+        best_acc = 0.0
+        early_stop = utils.EarlyStopping(mode="max")
+        for ep in range(args.epochs):
+            train_acc = train(ep)
+            test_acc, best_acc = test(ep)
+            nni.report_intermediate_result(test_acc)
+            logger.debug(test_acc)
+            if early_stop.step(test_acc):
+                break
+
+        # trial report best_acc to tuner
+        nni.report_final_result(best_acc)
+    except Exception as exception:
+        logger.exception(exception)
+        raise

examples/trials/network_morphism/cifar10/config.yml

+authorName: default
+experimentName: example_cifar10-network-morphism
+trialConcurrency: 4
+maxExecDuration: 48h
+maxTrialNum: 200
+#choice: local, remote, pai
+trainingServicePlatform: local
+#searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
+  #SMAC (SMAC should be installed through nnictl) 
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    #for now, this tuner only supports cv domain
+    task: cv
+    #input image width
+    input_width: 32
+    #input image channel
+    input_channel: 3
+    #number of classes
+    n_output_node: 10
+trial:
+  command: python3 cifar10_keras.py
+  codeDir: .
+  gpuNum: 1

examples/trials/network_morphism/cifar10/config_pai.yml

+authorName: default
+experimentName: example_cifar10-network-morphism
+trialConcurrency: 1
+maxExecDuration: 24h
+maxTrialNum: 10
+#choice: local, remote, pai
+trainingServicePlatform: pai
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
+  #SMAC (SMAC should be installed through nnictl) 
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    # for now, this tuner only supports cv domain
+    task: cv
+    #input image width
+    input_width: 32
+    #input image channel
+    input_channel: 3
+    #number of classes
+    n_output_node: 10
+trial:
+  command: python3 cifar10_keras.py
+  codeDir: .
+  gpuNum: 1
+  cpuNum: 1
+  memoryMB: 8196
+  #The docker image to run nni job on pai
+  image: msranni/nni:latest
+  #The hdfs directory to store data on pai, format 'hdfs://host:port/directory'
+  dataDir: hdfs://10.10.10.10:9000/username/nni
+  #The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
+  outputDir: hdfs://10.10.10.10:9000/username/nni
+paiConfig:
+  #The username to login pai
+  userName: username
+  #The password to login pai
+  passWord: password
+  #The host of restful server of pai
+  host: 10.10.10.10
\ No newline at end of file

examples/trials/network_morphism/requirements.txt

+numpy==1.14.2
+tensorflow==1.12.0
+torchvision==0.2.1
+Keras==2.2.2
+nni==0.3.0
+torch==0.4.1

examples/trials/network_morphism/utils.py

+"""Some helper functions for PyTorch, including:
+    - get_mean_and_std: calculate the mean and std value of dataset.
+    - msr_init: net parameter initialization.
+    - progress_bar: progress bar mimic xlua.progress.
+"""
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.init as init
+import torchvision.transforms as transforms
+
+
+class EarlyStopping:
+    """ EarlyStopping class to keep NN from overfitting
+    """
+
+    # pylint: disable=E0202
+    def __init__(self, mode="min", min_delta=0, patience=10, percentage=False):
+        self.mode = mode
+        self.min_delta = min_delta
+        self.patience = patience
+        self.best = None
+        self.num_bad_epochs = 0
+        self.is_better = None
+        self._init_is_better(mode, min_delta, percentage)
+
+        if patience == 0:
+            self.is_better = lambda a, b: True
+            self.step = lambda a: False
+
+    def step(self, metrics):
+        """ EarlyStopping step on each epoch
+        Arguments:
+            metrics {float} -- metric value
+        """
+
+        if self.best is None:
+            self.best = metrics
+            return False
+
+        if np.isnan(metrics):
+            return True
+
+        if self.is_better(metrics, self.best):
+            self.num_bad_epochs = 0
+            self.best = metrics
+        else:
+            self.num_bad_epochs += 1
+
+        if self.num_bad_epochs >= self.patience:
+            return True
+
+        return False
+
+    def _init_is_better(self, mode, min_delta, percentage):
+        if mode not in {"min", "max"}:
+            raise ValueError("mode " + mode + " is unknown!")
+        if not percentage:
+            if mode == "min":
+                self.is_better = lambda a, best: a < best - min_delta
+            if mode == "max":
+                self.is_better = lambda a, best: a > best + min_delta
+        else:
+            if mode == "min":
+                self.is_better = lambda a, best: a < best - (best * min_delta / 100)
+            if mode == "max":
+                self.is_better = lambda a, best: a > best + (best * min_delta / 100)
+
+
+class Cutout:
+    """Randomly mask out one or more patches from an image.
+    Args:
+        n_holes (int): Number of patches to cut out of each image.
+        length (int): The length (in pixels) of each square patch.
+    """
+
+    def __init__(self, length):
+        self.length = length
+
+    def __call__(self, img):
+        """
+        Args:
+            img (Tensor): Tensor image of size (C, H, W).
+        Returns:
+            Tensor: Image with n_holes of dimension length x length cut out of it.
+        """
+        h_img, w_img = img.size(1), img.size(2)
+        mask = np.ones((h_img, w_img), np.float32)
+        y_img = np.random.randint(h_img)
+        x_img = np.random.randint(w_img)
+
+        y1_img = np.clip(y_img - self.length // 2, 0, h_img)
+        y2_img = np.clip(y_img + self.length // 2, 0, h_img)
+        x1_img = np.clip(x_img - self.length // 2, 0, w_img)
+        x2_img = np.clip(x_img + self.length // 2, 0, w_img)
+
+        mask[y1_img:y2_img, x1_img:x2_img] = 0.0
+        mask = torch.from_numpy(mask)
+        mask = mask.expand_as(img)
+        img *= mask
+        return img
+
+
+def data_transforms_cifar10(args):
+    """ data_transforms for cifar10 dataset
+    """
+
+    cifar_mean = [0.49139968, 0.48215827, 0.44653124]
+    cifar_std = [0.24703233, 0.24348505, 0.26158768]
+
+    train_transform = transforms.Compose(
+        [
+            transforms.RandomCrop(32, padding=4),
+            transforms.RandomHorizontalFlip(),
+            transforms.ToTensor(),
+            transforms.Normalize(cifar_mean, cifar_std),
+        ]
+    )
+    if args.cutout:
+        train_transform.transforms.append(Cutout(args.cutout_length))
+
+    valid_transform = transforms.Compose(
+        [transforms.ToTensor(), transforms.Normalize(cifar_mean, cifar_std)]
+    )
+    return train_transform, valid_transform
+
+
+def data_transforms_mnist(args, mnist_mean=None, mnist_std=None):
+    """ data_transforms for mnist dataset
+    """
+    if mnist_mean is None:
+        mnist_mean = [0.5]
+
+    if mnist_std is None:
+        mnist_std = [0.5]
+
+    train_transform = transforms.Compose(
+        [
+            transforms.RandomCrop(28, padding=4),
+            transforms.RandomHorizontalFlip(),
+            transforms.ToTensor(),
+            transforms.Normalize(mnist_mean, mnist_std),
+        ]
+    )
+    if args.cutout:
+        train_transform.transforms.append(Cutout(args.cutout_length))
+
+    valid_transform = transforms.Compose(
+        [transforms.ToTensor(), transforms.Normalize(mnist_mean, mnist_std)]
+    )
+    return train_transform, valid_transform
+
+
+def get_mean_and_std(dataset):
+    """Compute the mean and std value of dataset."""
+    dataloader = torch.utils.data.DataLoader(
+        dataset, batch_size=1, shuffle=True, num_workers=2
+    )
+    mean = torch.zeros(3)
+    std = torch.zeros(3)
+    print("==> Computing mean and std..")
+    for inputs, _ in dataloader:
+        for i in range(3):
+            mean[i] += inputs[:, i, :, :].mean()
+            std[i] += inputs[:, i, :, :].std()
+    mean.div_(len(dataset))
+    std.div_(len(dataset))
+    return mean, std
+
+
+def init_params(net):
+    """Init layer parameters."""
+    for module in net.modules():
+        if isinstance(module, nn.Conv2d):
+            init.kaiming_normal(module.weight, mode="fan_out")
+            if module.bias:
+                init.constant(module.bias, 0)
+        elif isinstance(module, nn.BatchNorm2d):
+            init.constant(module.weight, 1)
+            init.constant(module.bias, 0)
+        elif isinstance(module, nn.Linear):
+            init.normal(module.weight, std=1e-3)
+            if module.bias:
+                init.constant(module.bias, 0)

pylintrc

@@ -16,3 +16,6 @@ const-naming-style=any
 
 disable=duplicate-code,
         super-init-not-called
+
+# List of members which are set dynamically and missed by pylint inference
+generated-members=numpy.*,torch.*

src/nni_manager/rest_server/restValidationSchemas.ts

@@ -118,7 +118,7 @@ export namespace ValidationSchemas {
                 checkpointDir: joi.string()
             }),
             tuner: joi.object({
-                builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch'),
+                builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch', 'NetworkMorphism'),
                 codeDir: joi.string(),
                 classFileName: joi.string(),
                 className: joi.string(),