Dev networkmorphism (#413)

* Quick fix nnictl config logic (#289)

* fix nnictl bug

* fix install.sh

* add desc for Dockerfile.build.base

* update document for Dockerfile

* update

* refactor port detect

* update

* refactor NNICTLDOC.md

* add document for pai and nnictl

* add default value for port

* add exception handling in trial_keeper.py

* fix port bug

* fix resume

* fix nnictl resume and fix nnictl stop

* fix document

* update

* refactor nnictl

* update

* update doc

* update

* update nnictl

* fix comment

* revert dockerfile

* update

* update

* update

* fix nnictl error hit

* fix comments

* fix bash-completion

* fix paramiko install

* quick fix resume logic

* update

* quick fix nnictl

* PR merge to 0.3 (#297)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* update doc

* fix links

* fix links

* fix links

* merge

* fix links and doc errors

* merge

* merge

* merge

* merge

* Update README.md (#288)

added License badge

* merge

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* fix doc mistakes and broken links. (#271)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* Update README.md

* Fix misspelling in examples/trials/ga_squad/README.md

* revise the installation cmd to v0.2

* revise to install v0.2

* remove enas readme (#292)

* Fix datastore performance issue (#301)

* Fix nnictl in v0.3 (#299)

Fix old version of config file
fix sklearn requirements
Fix resume log logic

* add basic tuner and trial for network morphism

* Complete basic receive_trial_result() and generate_parameters(). Use onnx  as the intermediate representation ( But it cannot convert to pytorch model )

* add tensorflow cifar10 for network morphism

* add unit test for tuner and its function

* use temporary torch_model

* fix request bug and program can communicate nni

* add basic pickle support for graph and train successful in pytorch

* Update unittest for networkmorphism_tuner

* Network Morphism add multi-gpu trial training support

* Format code with black tool

* change intermediate representation from pickle file to json we defined

* successfully pass the unittest for test_graph_json_transform

* add README for network morphism and it works fine in both Pytorch and Keras.

* separate the original Readme.md in network-morphism into two parts (tuner and trial)

* change the openpai image path

* beautify the file structure of network_morphism and add a fashion_mnist keras example

* pretty the source and add some docstring for funtion in order to pass the pylint.

* remove unused module import and add some docstring

* add some details for the application scenario Network Morphism Tuner

* follow the advice and modify the doc file

* add the config file for each task in the examples trial of network morphism

* change default python interpreter from python to python3

src/sdk/pynni/nni/constants.py

@@ -25,9 +25,9 @@ ModuleName = {
     'Evolution': 'nni.evolution_tuner.evolution_tuner',
     'SMAC': 'nni.smac_tuner.smac_tuner',
     'BatchTuner': 'nni.batch_tuner.batch_tuner',
+    'Medianstop': 'nni.medianstop_assessor.medianstop_assessor',
     'GridSearch': 'nni.gridsearch_tuner.gridsearch_tuner',
-    
-    'Medianstop': 'nni.medianstop_assessor.medianstop_assessor'
+    'NetworkMorphism': 'nni.networkmorphism_tuner.networkmorphism_tuner'
 }
 
 ClassName = {
@@ -38,6 +38,7 @@ ClassName = {
     'SMAC': 'SMACTuner',
     'BatchTuner': 'BatchTuner',
     'GridSearch': 'GridSearchTuner',
+    'NetworkMorphism':'NetworkMorphismTuner',
 
     'Medianstop': 'MedianstopAssessor'
 }

src/sdk/pynni/nni/networkmorphism_tuner/README.md

+# Network Morphism Tuner on NNI
+
+## 1. Intorduction
+
+[Autokeras](https://arxiv.org/abs/1806.10282) is a popular automl tools using Network Morphism. The basic idea of Autokeras is to use Bayesian Regression to estimate the metric of the Neural Network Architecture. Each time, it generates several child networks from father networks. Then it uses a naïve Bayesian regression estimate its metric value from history trained results of network and metric value pair. Next, it chooses the the child which has best estimated performance and adds it to the training queue. Inspired by its work and referring to its [code](https://github.com/jhfjhfj1/autokeras), we implement our Network Morphism method in our NNI platform.
+
+If you want to know about network morphism trial usage, please check [Readme.md](../../../../../examples/trials/network-morphism/README.md) of the trial to get more detail.
+
+## 2. Usage
+
+To use Network Morphism, you should modify the following spec in your `config.yml` file:
+
+```yaml
+tuner:
+  #choice: NetworkMorphism
+  builtinTunerName: NetworkMorphism
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: maximize
+    #for now, this tuner only supports cv domain
+    task: cv
+    #modify to fit your input image width
+    input_width: 32
+    #modify to fit your input image channel
+    input_channel: 3
+    #modify to fit your number of classes
+    n_output_node: 10
+```
+
+
+
+In the training procedure, it generate a JSON file which represent a Network Graph. Users can call "json\_to\_graph()" function to build a pytorch model or keras model from this JSON file.
+
+```python
+import nni
+from nni.networkmorphism_tuner.graph import json_to_graph
+
+def build_graph_from_json(ir_model_json):
+    """build a pytorch model from json representation
+    """
+    graph = json_to_graph(ir_model_json)
+    model = graph.produce_torch_model()
+    return model
+
+# trial get next parameter from network morphism tuner
+RCV_CONFIG = nni.get_next_parameter()
+# call the function to build pytorch model or keras model
+net = build_graph_from_json(RCV_CONFIG)
+
+# training procedure
+# ....
+
+# report the final accuracy to nni
+nni.report_final_result(best_acc)
+```
+
+## 3. File Structure
+
+The tuner has a lot of different files, functions and classes. Here we will only give most of those files a brief introduction:
+
+- `networkmorphism_tuner.py` is a tuner which using network morphism techniques.
+
+- `bayesian.py` is Bayesian method to estimate the metric of unseen model based on the models we have already searched.  
+- `graph.py`  is the meta graph data structure. Class Graph is representing the neural architecture graph of a model.
+  - Graph extracts the neural architecture graph from a model. 
+  - Each node in the graph is a intermediate tensor between layers.
+  - Each layer is an edge in the graph.
+  - Notably, multiple edges may refer to the same layer.
+- `graph_transformer.py` includes some graph transformer to wider, deeper or add a skip-connection into the graph.
+
+- `layers.py`  includes all the layers we use in our model.
+- `layer_transformer.py` includes some layer transformer to wider, deeper or add a skip-connection into the layer.
+- `nn.py` includes the class to generate network class initially.
+- `metric.py` some metric classes including Accuracy and MSE.
+- `utils.py` is the example search network architectures in dataset `cifar10` by using Keras.
+
+## 4. The Network Representation Json Example
+
+Here is an example of the intermediate representation JSON file we defined, which is passed from the tuner to the trial in the architecture search procedure. The example is as follows.
+
+```json
+{
+     "input_shape": [32, 32, 3],
+     "weighted": false,
+     "operation_history": [],
+     "layer_id_to_input_node_ids": {"0": [0],"1": [1],"2": [2],"3": [3],"4": [4],"5": [5],"6": [6],"7": [7],"8": [8],"9": [9],"10": [10],"11": [11],"12": [12],"13": [13],"14": [14],"15": [15],"16": [16]
+     },
+     "layer_id_to_output_node_ids": {"0": [1],"1": [2],"2": [3],"3": [4],"4": [5],"5": [6],"6": [7],"7": [8],"8": [9],"9": [10],"10": [11],"11": [12],"12": [13],"13": [14],"14": [15],"15": [16],"16": [17]
+     },
+     "adj_list": {
+         "0": [[1, 0]],
+         "1": [[2, 1]],
+         "2": [[3, 2]],
+         "3": [[4, 3]],
+         "4": [[5, 4]],
+         "5": [[6, 5]],
+         "6": [[7, 6]],
+         "7": [[8, 7]],
+         "8": [[9, 8]],
+         "9": [[10, 9]],
+         "10": [[11, 10]],
+         "11": [[12, 11]],
+         "12": [[13, 12]],
+         "13": [[14, 13]],
+         "14": [[15, 14]],
+         "15": [[16, 15]],
+         "16": [[17, 16]],
+         "17": []
+     },
+     "reverse_adj_list": {
+         "0": [],
+         "1": [[0, 0]],
+         "2": [[1, 1]],
+         "3": [[2, 2]],
+         "4": [[3, 3]],
+         "5": [[4, 4]],
+         "6": [[5, 5]],
+         "7": [[6, 6]],
+         "8": [[7, 7]],
+         "9": [[8, 8]],
+         "10": [[9, 9]],
+         "11": [[10, 10]],
+         "12": [[11, 11]],
+         "13": [[12, 12]],
+         "14": [[13, 13]],
+         "15": [[14, 14]],
+         "16": [[15, 15]],
+         "17": [[16, 16]]
+     },
+     "node_list": [
+         [0, [32, 32, 3]],
+         [1, [32, 32, 3]],
+         [2, [32, 32, 64]],
+         [3, [32, 32, 64]],
+         [4, [16, 16, 64]],
+         [5, [16, 16, 64]],
+         [6, [16, 16, 64]],
+         [7, [16, 16, 64]],
+         [8, [8, 8, 64]],
+         [9, [8, 8, 64]],
+         [10, [8, 8, 64]],
+         [11, [8, 8, 64]],
+         [12, [4, 4, 64]],
+         [13, [64]],
+         [14, [64]],
+         [15, [64]],
+         [16, [64]],
+         [17, [10]]
+     ],
+     "layer_list": [
+         [0, ["StubReLU", 0, 1]],
+         [1, ["StubConv2d", 1, 2, 3, 64, 3]],
+         [2, ["StubBatchNormalization2d", 2, 3, 64]],
+         [3, ["StubPooling2d", 3, 4, 2, 2, 0]],
+         [4, ["StubReLU", 4, 5]],
+         [5, ["StubConv2d", 5, 6, 64, 64, 3]],
+         [6, ["StubBatchNormalization2d", 6, 7, 64]],
+         [7, ["StubPooling2d", 7, 8, 2, 2, 0]],
+         [8, ["StubReLU", 8, 9]],
+         [9, ["StubConv2d", 9, 10, 64, 64, 3]],
+         [10, ["StubBatchNormalization2d", 10, 11, 64]],
+         [11, ["StubPooling2d", 11, 12, 2, 2, 0]],
+         [12, ["StubGlobalPooling2d", 12, 13]],
+         [13, ["StubDropout2d", 13, 14, 0.25]],
+         [14, ["StubDense", 14, 15, 64, 64]],
+         [15, ["StubReLU", 15, 16]],
+         [16, ["StubDense", 16, 17, 64, 10]]
+     ]
+ }
+```
+
+The definition of each model is a JSON object(also you can consider the model as a DAG graph), where:
+
+- `input_shape` is a list of integers, which does not include the batch axis.
+- `weighted` means whether the weights and biases in the neural network should be included in the graph.
+- `operation_history` is the number of inputs the layer has.
+- `layer_id_to_input_node_ids` is a dictionary instance mapping from layer identifiers to their input nodes identifiers.
+- `layer_id_to_output_node_ids` is a dictionary instance mapping from layer identifiers to their output nodes identifiers
+- `adj_list` is a two dimensional list. The adjacency list of the graph. The first dimension is identified by tensor identifiers. In each edge list, the elements are two-element tuples of (tensor identifier, layer identifier).
+- `reverse_adj_list` is a  A reverse adjacent list in the same format as adj_list.
+- `node_list` is a list of integers. The indices of the list are the identifiers.
+- `layer_list` is a list of stub layers. The indices of the list are the identifiers.
+  - For `StubConv (StubConv1d, StubConv2d, StubConv3d)`, the number follows is its node input id(or id list), node output id, input_channel, filters, kernel_size, stride and padding.
+
+  - For `StubDense`, the number follows is its node input id(or id list), node output id, input_units and units.
+
+  - For `StubBatchNormalization (StubBatchNormalization1d, StubBatchNormalization2d, StubBatchNormalization3d)`, the number follows is its node input id(or id list), node output id and features numbers.
+
+  - For `StubDropout(StubDropout1d, StubDropout2d, StubDropout3d)`, the number follows is its node input id(or id list), node output id and dropout rate.
+
+  - For `StubPooling (StubPooling1d, StubPooling2d, StubPooling3d)`, the number follows is its node input id(or id list), node output id, kernel_size, stride and padding.
+
+  - For else layers, the number follows is its node input id(or id list) and node output id.
+
+## 5. TODO
+
+Next step, we will change the API from fixed network generator to more network operator generator. Besides, we will use ONNX instead of JSON later as the intermediate representation spec in the future.

src/sdk/pynni/nni/networkmorphism_tuner/bayesian.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+import math
+import random
+from copy import deepcopy
+from functools import total_ordering
+from queue import PriorityQueue
+
+import numpy as np
+from scipy.linalg import LinAlgError, cho_solve, cholesky, solve_triangular
+from scipy.optimize import linear_sum_assignment
+from sklearn.metrics.pairwise import rbf_kernel
+
+from nni.networkmorphism_tuner.graph_transformer import transform
+from nni.networkmorphism_tuner.utils import Constant, OptimizeMode
+from nni.networkmorphism_tuner.layers import is_layer
+
+
+def layer_distance(a, b):
+    """The distance between two layers."""
+    # pylint: disable=unidiomatic-typecheck
+    if type(a) != type(b):
+        return 1.0
+    if is_layer(a, "Conv"):
+        att_diff = [
+            (a.filters, b.filters),
+            (a.kernel_size, b.kernel_size),
+            (a.stride, b.stride),
+        ]
+        return attribute_difference(att_diff)
+    if is_layer(a, "Pooling"):
+        att_diff = [
+            (a.padding, b.padding),
+            (a.kernel_size, b.kernel_size),
+            (a.stride, b.stride),
+        ]
+        return attribute_difference(att_diff)
+    return 0.0
+
+
+def attribute_difference(att_diff):
+    ''' The attribute distance.
+    '''
+
+    ret = 0
+    for a_value, b_value in att_diff:
+        if max(a_value, b_value) == 0:
+            ret += 0
+        else:
+            ret += abs(a_value - b_value) * 1.0 / max(a_value, b_value)
+    return ret * 1.0 / len(att_diff)
+
+
+def layers_distance(list_a, list_b):
+    """The distance between the layers of two neural networks."""
+    len_a = len(list_a)
+    len_b = len(list_b)
+    f = np.zeros((len_a + 1, len_b + 1))
+    f[-1][-1] = 0
+    for i in range(-1, len_a):
+        f[i][-1] = i + 1
+    for j in range(-1, len_b):
+        f[-1][j] = j + 1
+    for i in range(len_a):
+        for j in range(len_b):
+            f[i][j] = min(
+                f[i][j - 1] + 1,
+                f[i - 1][j] + 1,
+                f[i - 1][j - 1] + layer_distance(list_a[i], list_b[j]),
+            )
+    return f[len_a - 1][len_b - 1]
+
+
+def skip_connection_distance(a, b):
+    """The distance between two skip-connections."""
+    if a[2] != b[2]:
+        return 1.0
+    len_a = abs(a[1] - a[0])
+    len_b = abs(b[1] - b[0])
+    return (abs(a[0] - b[0]) + abs(len_a - len_b)) / (max(a[0], b[0]) + max(len_a, len_b))
+
+
+def skip_connections_distance(list_a, list_b):
+    """The distance between the skip-connections of two neural networks."""
+    distance_matrix = np.zeros((len(list_a), len(list_b)))
+    for i, a in enumerate(list_a):
+        for j, b in enumerate(list_b):
+            distance_matrix[i][j] = skip_connection_distance(a, b)
+    return distance_matrix[linear_sum_assignment(distance_matrix)].sum() + abs(
+        len(list_a) - len(list_b)
+    )
+
+
+def edit_distance(x, y):
+    """The distance between two neural networks.
+    Args:
+        x: An instance of NetworkDescriptor.
+        y: An instance of NetworkDescriptor
+    Returns:
+        The edit-distance between x and y.
+    """
+
+    ret = layers_distance(x.layers, y.layers)
+    ret += Constant.KERNEL_LAMBDA * skip_connections_distance(
+        x.skip_connections, y.skip_connections
+    )
+    return ret
+
+
+class IncrementalGaussianProcess:
+    """Gaussian process regressor.
+    Attributes:
+        alpha: A hyperparameter.
+    """
+
+    def __init__(self):
+        self.alpha = 1e-10
+        self._distance_matrix = None
+        self._x = None
+        self._y = None
+        self._first_fitted = False
+        self._l_matrix = None
+        self._alpha_vector = None
+
+    @property
+    def kernel_matrix(self):
+        ''' Kernel matric.
+        '''
+        return self._distance_matrix
+
+    def fit(self, train_x, train_y):
+        """ Fit the regressor with more data.
+        Args:
+            train_x: A list of NetworkDescriptor.
+            train_y: A list of metric values.
+        """
+        if self.first_fitted:
+            self.incremental_fit(train_x, train_y)
+        else:
+            self.first_fit(train_x, train_y)
+
+    def incremental_fit(self, train_x, train_y):
+        """ Incrementally fit the regressor. """
+        if not self._first_fitted:
+            raise ValueError("The first_fit function needs to be called first.")
+
+        train_x, train_y = np.array(train_x), np.array(train_y)
+
+        # Incrementally compute K
+        up_right_k = edit_distance_matrix(self._x, train_x)
+        down_left_k = np.transpose(up_right_k)
+        down_right_k = edit_distance_matrix(train_x)
+        up_k = np.concatenate((self._distance_matrix, up_right_k), axis=1)
+        down_k = np.concatenate((down_left_k, down_right_k), axis=1)
+        temp_distance_matrix = np.concatenate((up_k, down_k), axis=0)
+        k_matrix = bourgain_embedding_matrix(temp_distance_matrix)
+        diagonal = np.diag_indices_from(k_matrix)
+        diagonal = (diagonal[0][-len(train_x) :], diagonal[1][-len(train_x) :])
+        k_matrix[diagonal] += self.alpha
+
+        try:
+            self._l_matrix = cholesky(k_matrix, lower=True)  # Line 2
+        except LinAlgError:
+            return self
+
+        self._x = np.concatenate((self._x, train_x), axis=0)
+        self._y = np.concatenate((self._y, train_y), axis=0)
+        self._distance_matrix = temp_distance_matrix
+
+        self._alpha_vector = cho_solve((self._l_matrix, True), self._y)  # Line 3
+
+        return self
+
+    @property
+    def first_fitted(self):
+        ''' if it is firsr fitted
+        '''
+        return self._first_fitted
+
+    def first_fit(self, train_x, train_y):
+        """ Fit the regressor for the first time. """
+        train_x, train_y = np.array(train_x), np.array(train_y)
+
+        self._x = np.copy(train_x)
+        self._y = np.copy(train_y)
+
+        self._distance_matrix = edit_distance_matrix(self._x)
+        k_matrix = bourgain_embedding_matrix(self._distance_matrix)
+        k_matrix[np.diag_indices_from(k_matrix)] += self.alpha
+
+        self._l_matrix = cholesky(k_matrix, lower=True)  # Line 2
+
+        self._alpha_vector = cho_solve((self._l_matrix, True), self._y)  # Line 3
+
+        self._first_fitted = True
+        return self
+
+    def predict(self, train_x):
+        """Predict the result.
+        Args:
+            train_x: A list of NetworkDescriptor.
+        Returns:
+            y_mean: The predicted mean.
+            y_std: The predicted standard deviation.
+        """
+        k_trans = np.exp(-np.power(edit_distance_matrix(train_x, self._x), 2))
+        y_mean = k_trans.dot(self._alpha_vector)  # Line 4 (y_mean = f_star)
+
+        # compute inverse K_inv of K based on its Cholesky
+        # decomposition L and its inverse L_inv
+        l_inv = solve_triangular(self._l_matrix.T, np.eye(self._l_matrix.shape[0]))
+        k_inv = l_inv.dot(l_inv.T)
+        # Compute variance of predictive distribution
+        y_var = np.ones(len(train_x), dtype=np.float)
+        y_var -= np.einsum("ij,ij->i", np.dot(k_trans, k_inv), k_trans)
+
+        # Check if any of the variances is negative because of
+        # numerical issues. If yes: set the variance to 0.
+        y_var_negative = y_var < 0
+        if np.any(y_var_negative):
+            y_var[y_var_negative] = 0.0
+        return y_mean, np.sqrt(y_var)
+
+
+def edit_distance_matrix(train_x, train_y=None):
+    """Calculate the edit distance.
+    Args:
+        train_x: A list of neural architectures.
+        train_y: A list of neural architectures.
+    Returns:
+        An edit-distance matrix.
+    """
+    if train_y is None:
+        ret = np.zeros((train_x.shape[0], train_x.shape[0]))
+        for x_index, x in enumerate(train_x):
+            for y_index, y in enumerate(train_x):
+                if x_index == y_index:
+                    ret[x_index][y_index] = 0
+                elif x_index < y_index:
+                    ret[x_index][y_index] = edit_distance(x, y)
+                else:
+                    ret[x_index][y_index] = ret[y_index][x_index]
+        return ret
+    ret = np.zeros((train_x.shape[0], train_y.shape[0]))
+    for x_index, x in enumerate(train_x):
+        for y_index, y in enumerate(train_y):
+            ret[x_index][y_index] = edit_distance(x, y)
+    return ret
+
+
+def vector_distance(a, b):
+    """The Euclidean distance between two vectors."""
+    a = np.array(a)
+    b = np.array(b)
+    return np.linalg.norm(a - b)
+
+
+def bourgain_embedding_matrix(distance_matrix):
+    """Use Bourgain algorithm to embed the neural architectures based on their edit-distance.
+    Args:
+        distance_matrix: A matrix of edit-distances.
+    Returns:
+        A matrix of distances after embedding.
+    """
+    distance_matrix = np.array(distance_matrix)
+    n = len(distance_matrix)
+    if n == 1:
+        return distance_matrix
+    np.random.seed(123)
+    distort_elements = []
+    r = range(n)
+    k = int(math.ceil(math.log(n) / math.log(2) - 1))
+    t = int(math.ceil(math.log(n)))
+    counter = 0
+    for i in range(0, k + 1):
+        for t in range(t):
+            s = np.random.choice(r, 2 ** i)
+            for j in r:
+                d = min([distance_matrix[j][s] for s in s])
+                counter += len(s)
+                if i == 0 and t == 0:
+                    distort_elements.append([d])
+                else:
+                    distort_elements[j].append(d)
+    return rbf_kernel(distort_elements, distort_elements)
+
+
+class BayesianOptimizer:
+    """ A Bayesian optimizer for neural architectures.
+    Attributes:
+        searcher: The Searcher which is calling the Bayesian optimizer.
+        t_min: The minimum temperature for simulated annealing.
+        metric: An instance of the Metric subclasses.
+        gpr: A GaussianProcessRegressor for bayesian optimization.
+        beta: The beta in acquisition function. (refer to our paper)
+        search_tree: The network morphism search tree.
+    """
+
+    def __init__(self, searcher, t_min, optimizemode, beta=None):
+        self.searcher = searcher
+        self.t_min = t_min
+        self.optimizemode = optimizemode
+        self.gpr = IncrementalGaussianProcess()
+        self.beta = beta if beta is not None else Constant.BETA
+        self.search_tree = SearchTree()
+
+    def fit(self, x_queue, y_queue):
+        """ Fit the optimizer with new architectures and performances.
+        Args:
+            x_queue: A list of NetworkDescriptor.
+            y_queue: A list of metric values.
+        """
+        self.gpr.fit(x_queue, y_queue)
+
+    def generate(self, descriptors):
+        """Generate new architecture.
+        Args:
+            descriptors: All the searched neural architectures.
+        Returns:
+            graph: An instance of Graph. A morphed neural network with weights.
+            father_id: The father node ID in the search tree.
+        """
+        model_ids = self.search_tree.adj_list.keys()
+
+        target_graph = None
+        father_id = None
+        descriptors = deepcopy(descriptors)
+        elem_class = Elem
+        if self.optimizemode is OptimizeMode.Maximize:
+            elem_class = ReverseElem
+
+        # Initialize the priority queue.
+        pq = PriorityQueue()
+        temp_list = []
+        for model_id in model_ids:
+            metric_value = self.searcher.get_metric_value_by_id(model_id)
+            temp_list.append((metric_value, model_id))
+        temp_list = sorted(temp_list)
+        for metric_value, model_id in temp_list:
+            graph = self.searcher.load_model_by_id(model_id)
+            graph.clear_operation_history()
+            graph.clear_weights()
+            pq.put(elem_class(metric_value, model_id, graph))
+
+        t = 1.0
+        t_min = self.t_min
+        alpha = 0.9
+        opt_acq = self._get_init_opt_acq_value()
+        while not pq.empty() and t > t_min:
+            elem = pq.get()
+            if self.optimizemode is OptimizeMode.Maximize:
+                temp_exp = min((elem.metric_value - opt_acq) / t, 1.0)
+            else:
+                temp_exp = min((opt_acq - elem.metric_value) / t, 1.0)
+            ap = math.exp(temp_exp)
+            if ap >= random.uniform(0, 1):
+                for temp_graph in transform(elem.graph):
+                    if contain(descriptors, temp_graph.extract_descriptor()):
+                        continue
+
+                    temp_acq_value = self.acq(temp_graph)
+                    pq.put(elem_class(temp_acq_value, elem.father_id, temp_graph))
+                    descriptors.append(temp_graph.extract_descriptor())
+                    if self._accept_new_acq_value(opt_acq, temp_acq_value):
+                        opt_acq = temp_acq_value
+                        father_id = elem.father_id
+                        target_graph = deepcopy(temp_graph)
+            t *= alpha
+
+        # Did not found a not duplicated architecture
+        if father_id is None:
+            return None, None
+        nm_graph = self.searcher.load_model_by_id(father_id)
+        for args in target_graph.operation_history:
+            getattr(nm_graph, args[0])(*list(args[1:]))
+        return nm_graph, father_id
+
+    def acq(self, graph):
+        ''' estimate the value of generated graph
+        '''
+        mean, std = self.gpr.predict(np.array([graph.extract_descriptor()]))
+        if self.optimizemode is OptimizeMode.Maximize:
+            return mean + self.beta * std
+        return mean - self.beta * std
+
+    def _get_init_opt_acq_value(self):
+        if self.optimizemode is OptimizeMode.Maximize:
+            return -np.inf
+        return np.inf
+
+    def _accept_new_acq_value(self, opt_acq, temp_acq_value):
+        if temp_acq_value > opt_acq and self.optimizemode is OptimizeMode.Maximize:
+            return True
+        if temp_acq_value < opt_acq and not self.optimizemode is OptimizeMode.Maximize:
+            return True
+        return False
+
+    def add_child(self, father_id, model_id):
+        ''' add child to the search tree
+        Arguments:
+            father_id {int} -- father id
+            model_id {int} -- model id
+        '''
+
+        self.search_tree.add_child(father_id, model_id)
+
+
+@total_ordering
+class Elem:
+    """Elements to be sorted according to metric value."""
+
+    def __init__(self, metric_value, father_id, graph):
+        self.father_id = father_id
+        self.graph = graph
+        self.metric_value = metric_value
+
+    def __eq__(self, other):
+        return self.metric_value == other.metric_value
+
+    def __lt__(self, other):
+        return self.metric_value < other.metric_value
+
+
+class ReverseElem(Elem):
+    """Elements to be reversely sorted according to metric value."""
+
+    def __lt__(self, other):
+        return self.metric_value > other.metric_value
+
+
+def contain(descriptors, target_descriptor):
+    """Check if the target descriptor is in the descriptors."""
+    for descriptor in descriptors:
+        if edit_distance(descriptor, target_descriptor) < 1e-5:
+            return True
+    return False
+
+
+class SearchTree:
+    """The network morphism search tree."""
+
+    def __init__(self):
+        self.root = None
+        self.adj_list = {}
+
+    def add_child(self, u, v):
+        ''' add child to search tree itself.
+        Arguments:
+            u {int} -- father id
+            v {int} --  child id
+        '''
+
+        if u == -1:
+            self.root = v
+            self.adj_list[v] = []
+            return
+        if v not in self.adj_list[u]:
+            self.adj_list[u].append(v)
+        if v not in self.adj_list:
+            self.adj_list[v] = []
+
+    def get_dict(self, u=None):
+        """ A recursive function to return the content of the tree in a dict."""
+        if u is None:
+            return self.get_dict(self.root)
+        children = []
+        for v in self.adj_list[u]:
+            children.append(self.get_dict(v))
+        ret = {"name": u, "children": children}
+        return ret

src/sdk/pynni/nni/networkmorphism_tuner/graph_transformer.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+from copy import deepcopy
+
+from random import randrange, sample
+
+from nni.networkmorphism_tuner.graph import NetworkDescriptor
+from nni.networkmorphism_tuner.layers import (
+    StubDense,
+    StubReLU,
+    get_batch_norm_class,
+    get_conv_class,
+    get_dropout_class,
+    get_pooling_class,
+    is_layer,
+)
+from nni.networkmorphism_tuner.utils import Constant
+
+
+def to_wider_graph(graph):
+    ''' wider graph
+    '''
+    weighted_layer_ids = graph.wide_layer_ids()
+    weighted_layer_ids = list(
+        filter(lambda x: graph.layer_list[x].output.shape[-1], weighted_layer_ids)
+    )
+    wider_layers = sample(weighted_layer_ids, 1)
+
+    for layer_id in wider_layers:
+        layer = graph.layer_list[layer_id]
+        if is_layer(layer, "Conv"):
+            n_add = layer.filters
+        else:
+            n_add = layer.units
+
+        graph.to_wider_model(layer_id, n_add)
+    return graph
+
+
+def to_skip_connection_graph(graph):
+    ''' skip connection graph
+    '''
+    # The last conv layer cannot be widen since wider operator cannot be done over the two sides of flatten.
+    weighted_layer_ids = graph.skip_connection_layer_ids()
+    valid_connection = []
+    for skip_type in sorted([NetworkDescriptor.ADD_CONNECT, NetworkDescriptor.CONCAT_CONNECT]):
+        for index_a in range(len(weighted_layer_ids)):
+            for index_b in range(len(weighted_layer_ids))[index_a + 1 :]:
+                valid_connection.append((index_a, index_b, skip_type))
+
+    if not valid_connection:
+        return graph
+    for index_a, index_b, skip_type in sample(valid_connection, 1):
+        a_id = weighted_layer_ids[index_a]
+        b_id = weighted_layer_ids[index_b]
+        if skip_type == NetworkDescriptor.ADD_CONNECT:
+            graph.to_add_skip_model(a_id, b_id)
+        else:
+            graph.to_concat_skip_model(a_id, b_id)
+    return graph
+
+
+def create_new_layer(layer, n_dim):
+    ''' create  new layer for the graph
+    '''
+
+    input_shape = layer.output.shape
+    dense_deeper_classes = [StubDense, get_dropout_class(n_dim), StubReLU]
+    conv_deeper_classes = [get_conv_class(n_dim), get_batch_norm_class(n_dim), StubReLU]
+    if is_layer(layer, "ReLU"):
+        conv_deeper_classes = [get_conv_class(n_dim), get_batch_norm_class(n_dim)]
+        dense_deeper_classes = [StubDense, get_dropout_class(n_dim)]
+    elif is_layer(layer, "Dropout"):
+        dense_deeper_classes = [StubDense, StubReLU]
+    elif is_layer(layer, "BatchNormalization"):
+        conv_deeper_classes = [get_conv_class(n_dim), StubReLU]
+
+    layer_class = None
+    if len(input_shape) == 1:
+        # It is in the dense layer part.
+        layer_class = sample(dense_deeper_classes, 1)[0]
+    else:
+        # It is in the conv layer part.
+        layer_class = sample(conv_deeper_classes, 1)[0]
+
+    if layer_class == StubDense:
+        new_layer = StubDense(input_shape[0], input_shape[0])
+
+    elif layer_class == get_dropout_class(n_dim):
+        new_layer = layer_class(Constant.DENSE_DROPOUT_RATE)
+
+    elif layer_class == get_conv_class(n_dim):
+        new_layer = layer_class(
+            input_shape[-1], input_shape[-1], sample((1, 3, 5), 1)[0], stride=1
+        )
+
+    elif layer_class == get_batch_norm_class(n_dim):
+        new_layer = layer_class(input_shape[-1])
+
+    elif layer_class == get_pooling_class(n_dim):
+        new_layer = layer_class(sample((1, 3, 5), 1)[0])
+
+    else:
+        new_layer = layer_class()
+
+    return new_layer
+
+
+def to_deeper_graph(graph):
+    ''' deeper graph
+    '''
+
+    weighted_layer_ids = graph.deep_layer_ids()
+    if len(weighted_layer_ids) >= Constant.MAX_LAYERS:
+        return None
+
+    deeper_layer_ids = sample(weighted_layer_ids, 1)
+
+    for layer_id in deeper_layer_ids:
+        layer = graph.layer_list[layer_id]
+        new_layer = create_new_layer(layer, graph.n_dim)
+        graph.to_deeper_model(layer_id, new_layer)
+    return graph
+
+
+def legal_graph(graph):
+    '''judge if a graph is legal or not.
+    '''
+
+    descriptor = graph.extract_descriptor()
+    skips = descriptor.skip_connections
+    if len(skips) != len(set(skips)):
+        return False
+    return True
+
+
+def transform(graph):
+    '''core transform function for graph.
+    '''
+
+    graphs = []
+    for _ in range(Constant.N_NEIGHBOURS * 2):
+        random_num = randrange(3)
+        temp_graph = None
+        if random_num == 0:
+            temp_graph = to_deeper_graph(deepcopy(graph))
+        elif random_num == 1:
+            temp_graph = to_wider_graph(deepcopy(graph))
+        elif random_num == 2:
+            temp_graph = to_skip_connection_graph(deepcopy(graph))
+
+        if temp_graph is not None and temp_graph.size() <= Constant.MAX_MODEL_SIZE:
+            graphs.append(temp_graph)
+
+        if len(graphs) >= Constant.N_NEIGHBOURS:
+            break
+
+    return graphs

src/sdk/pynni/nni/networkmorphism_tuner/layer_transformer.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+import numpy as np
+
+from nni.networkmorphism_tuner.layers import (
+    StubDense,
+    StubReLU,
+    get_batch_norm_class,
+    get_conv_class,
+    get_n_dim,
+)
+
+NOISE_RATIO = 1e-4
+
+
+def deeper_conv_block(conv_layer, kernel_size, weighted=True):
+    '''deeper conv layer.
+    '''
+    n_dim = get_n_dim(conv_layer)
+    filter_shape = (kernel_size,) * 2
+    n_filters = conv_layer.filters
+    weight = np.zeros((n_filters, n_filters) + filter_shape)
+    center = tuple(map(lambda x: int((x - 1) / 2), filter_shape))
+    for i in range(n_filters):
+        filter_weight = np.zeros((n_filters,) + filter_shape)
+        index = (i,) + center
+        filter_weight[index] = 1
+        weight[i, ...] = filter_weight
+    bias = np.zeros(n_filters)
+    new_conv_layer = get_conv_class(n_dim)(
+        conv_layer.filters, n_filters, kernel_size=kernel_size
+    )
+    bn = get_batch_norm_class(n_dim)(n_filters)
+
+    if weighted:
+        new_conv_layer.set_weights(
+            (add_noise(weight, np.array([0, 1])), add_noise(bias, np.array([0, 1])))
+        )
+        new_weights = [
+            add_noise(np.ones(n_filters, dtype=np.float32), np.array([0, 1])),
+            add_noise(np.zeros(n_filters, dtype=np.float32), np.array([0, 1])),
+            add_noise(np.zeros(n_filters, dtype=np.float32), np.array([0, 1])),
+            add_noise(np.ones(n_filters, dtype=np.float32), np.array([0, 1])),
+        ]
+        bn.set_weights(new_weights)
+
+    return [StubReLU(), new_conv_layer, bn]
+
+
+def dense_to_deeper_block(dense_layer, weighted=True):
+    '''deeper dense layer.
+    '''
+    units = dense_layer.units
+    weight = np.eye(units)
+    bias = np.zeros(units)
+    new_dense_layer = StubDense(units, units)
+    if weighted:
+        new_dense_layer.set_weights(
+            (add_noise(weight, np.array([0, 1])), add_noise(bias, np.array([0, 1])))
+        )
+    return [StubReLU(), new_dense_layer]
+
+
+def wider_pre_dense(layer, n_add, weighted=True):
+    '''wider previous dense layer.
+    '''
+    if not weighted:
+        return StubDense(layer.input_units, layer.units + n_add)
+
+    n_units2 = layer.units
+
+    teacher_w, teacher_b = layer.get_weights()
+    rand = np.random.randint(n_units2, size=n_add)
+    student_w = teacher_w.copy()
+    student_b = teacher_b.copy()
+
+    # target layer update (i)
+    for i in range(n_add):
+        teacher_index = rand[i]
+        new_weight = teacher_w[teacher_index, :]
+        new_weight = new_weight[np.newaxis, :]
+        student_w = np.concatenate((student_w, add_noise(new_weight, student_w)), axis=0)
+        student_b = np.append(student_b, add_noise(teacher_b[teacher_index], student_b))
+
+    new_pre_layer = StubDense(layer.input_units, n_units2 + n_add)
+    new_pre_layer.set_weights((student_w, student_b))
+
+    return new_pre_layer
+
+
+def wider_pre_conv(layer, n_add_filters, weighted=True):
+    '''wider previous conv layer.
+    '''
+    n_dim = get_n_dim(layer)
+    if not weighted:
+        return get_conv_class(n_dim)(
+            layer.input_channel,
+            layer.filters + n_add_filters,
+            kernel_size=layer.kernel_size,
+        )
+
+    n_pre_filters = layer.filters
+    rand = np.random.randint(n_pre_filters, size=n_add_filters)
+    teacher_w, teacher_b = layer.get_weights()
+
+    student_w = teacher_w.copy()
+    student_b = teacher_b.copy()
+    # target layer update (i)
+    for i in range(len(rand)):
+        teacher_index = rand[i]
+        new_weight = teacher_w[teacher_index, ...]
+        new_weight = new_weight[np.newaxis, ...]
+        student_w = np.concatenate((student_w, new_weight), axis=0)
+        student_b = np.append(student_b, teacher_b[teacher_index])
+    new_pre_layer = get_conv_class(n_dim)(
+        layer.input_channel, n_pre_filters + n_add_filters, layer.kernel_size
+    )
+    new_pre_layer.set_weights(
+        (add_noise(student_w, teacher_w), add_noise(student_b, teacher_b))
+    )
+    return new_pre_layer
+
+
+def wider_next_conv(layer, start_dim, total_dim, n_add, weighted=True):
+    '''wider next conv layer.
+    '''
+    n_dim = get_n_dim(layer)
+    if not weighted:
+        return get_conv_class(n_dim)(layer.input_channel + n_add,
+                                     layer.filters,
+                                     kernel_size=layer.kernel_size,
+                                     stride=layer.stride)
+    n_filters = layer.filters
+    teacher_w, teacher_b = layer.get_weights()
+
+    new_weight_shape = list(teacher_w.shape)
+    new_weight_shape[1] = n_add
+    new_weight = np.zeros(tuple(new_weight_shape))
+
+    student_w = np.concatenate((teacher_w[:, :start_dim, ...].copy(),
+                                add_noise(new_weight, teacher_w),
+                                teacher_w[:, start_dim:total_dim, ...].copy()), axis=1)
+    new_layer = get_conv_class(n_dim)(layer.input_channel + n_add,
+                                      n_filters,
+                                      kernel_size=layer.kernel_size,
+                                      stride=layer.stride)
+    new_layer.set_weights((student_w, teacher_b))
+    return new_layer
+
+
+def wider_bn(layer, start_dim, total_dim, n_add, weighted=True):
+    '''wider batch norm layer.
+    '''
+    n_dim = get_n_dim(layer)
+    if not weighted:
+        return get_batch_norm_class(n_dim)(layer.num_features + n_add)
+
+    weights = layer.get_weights()
+
+    new_weights = [
+        add_noise(np.ones(n_add, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.zeros(n_add, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.zeros(n_add, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.ones(n_add, dtype=np.float32), np.array([0, 1])),
+    ]
+
+    student_w = tuple()
+    for weight, new_weight in zip(weights, new_weights):
+        temp_w = weight.copy()
+        temp_w = np.concatenate(
+            (temp_w[:start_dim], new_weight, temp_w[start_dim:total_dim])
+        )
+        student_w += (temp_w,)
+    new_layer = get_batch_norm_class(n_dim)(layer.num_features + n_add)
+    new_layer.set_weights(student_w)
+    return new_layer
+
+
+def wider_next_dense(layer, start_dim, total_dim, n_add, weighted=True):
+    '''wider next dense layer.
+    '''
+    if not weighted:
+        return StubDense(layer.input_units + n_add, layer.units)
+    teacher_w, teacher_b = layer.get_weights()
+    student_w = teacher_w.copy()
+    n_units_each_channel = int(teacher_w.shape[1] / total_dim)
+
+    new_weight = np.zeros((teacher_w.shape[0], n_add * n_units_each_channel))
+    student_w = np.concatenate(
+        (
+            student_w[:, : start_dim * n_units_each_channel],
+            add_noise(new_weight, student_w),
+            student_w[
+                :, start_dim * n_units_each_channel : total_dim * n_units_each_channel
+            ],
+        ),
+        axis=1,
+    )
+
+    new_layer = StubDense(layer.input_units + n_add, layer.units)
+    new_layer.set_weights((student_w, teacher_b))
+    return new_layer
+
+
+def add_noise(weights, other_weights):
+    '''add noise to the layer.
+    '''
+    w_range = np.ptp(other_weights.flatten())
+    noise_range = NOISE_RATIO * w_range
+    noise = np.random.uniform(-noise_range / 2.0, noise_range / 2.0, weights.shape)
+    return np.add(noise, weights)
+
+
+def init_dense_weight(layer):
+    '''initilize dense layer weight.
+    '''
+    units = layer.units
+    weight = np.eye(units)
+    bias = np.zeros(units)
+    layer.set_weights(
+        (add_noise(weight, np.array([0, 1])), add_noise(bias, np.array([0, 1])))
+    )
+
+
+def init_conv_weight(layer):
+    '''initilize conv layer weight.
+    '''
+    n_filters = layer.filters
+    filter_shape = (layer.kernel_size,) * get_n_dim(layer)
+    weight = np.zeros((n_filters, n_filters) + filter_shape)
+
+    center = tuple(map(lambda x: int((x - 1) / 2), filter_shape))
+    for i in range(n_filters):
+        filter_weight = np.zeros((n_filters,) + filter_shape)
+        index = (i,) + center
+        filter_weight[index] = 1
+        weight[i, ...] = filter_weight
+    bias = np.zeros(n_filters)
+
+    layer.set_weights(
+        (add_noise(weight, np.array([0, 1])), add_noise(bias, np.array([0, 1])))
+    )
+
+
+def init_bn_weight(layer):
+    '''initilize batch norm layer weight.
+    '''
+    n_filters = layer.num_features
+    new_weights = [
+        add_noise(np.ones(n_filters, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.zeros(n_filters, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.zeros(n_filters, dtype=np.float32), np.array([0, 1])),
+        add_noise(np.ones(n_filters, dtype=np.float32), np.array([0, 1])),
+    ]
+    layer.set_weights(new_weights)

src/sdk/pynni/nni/networkmorphism_tuner/networkmorphism_tuner.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+import logging
+import os
+
+
+from nni.tuner import Tuner
+from nni.networkmorphism_tuner.bayesian import BayesianOptimizer
+from nni.networkmorphism_tuner.nn import CnnGenerator, MlpGenerator
+from nni.networkmorphism_tuner.utils import Constant, OptimizeMode
+
+from nni.networkmorphism_tuner.graph import graph_to_json, json_to_graph
+
+logger = logging.getLogger("NetworkMorphism_AutoML")
+
+
+class NetworkMorphismTuner(Tuner):
+    """NetworkMorphismTuner is a tuner which using network morphism techniques."""
+
+    def __init__(
+            self,
+            task="cv",
+            input_width=32,
+            input_channel=3,
+            n_output_node=10,
+            algorithm_name="Bayesian",
+            optimize_mode="maximize",
+            path="model_path",
+            verbose=True,
+            beta=Constant.BETA,
+            t_min=Constant.T_MIN,
+            max_model_size=Constant.MAX_MODEL_SIZE,
+            default_model_len=Constant.MODEL_LEN,
+            default_model_width=Constant.MODEL_WIDTH,
+    ):
+        """ initilizer of the NetworkMorphismTuner.
+        Keyword Arguments:
+            task {str} -- [task mode, such as "cv","common" etc.] (default: {"cv"})
+            input_width {int} -- [input sample shape] (default: {32})
+            input_channel {int} -- [input sample shape] (default: {3})
+            n_output_node {int} -- [output node number] (default: {10})
+            algorithm_name {str} -- [algorithm name used in the network morphism] (default: {"Bayesian"})
+            optimize_mode {str} -- [optimize mode "minimize" or "maximize"] (default: {"minimize"})
+            path {str} -- [default mode path to save the model file] (default: {"model_path"})
+            verbose {bool} -- [verbose to print the log] (default: {True})
+            beta {float} -- [The beta in acquisition function. (refer to our paper)] (default: {Constant.BETA})
+            t_min {float} -- [The minimum temperature for simulated annealing.] (default: {Constant.T_MIN})
+            max_model_size {int} -- [max model size to the graph] (default: {Constant.MAX_MODEL_SIZE})
+            default_model_len {int} -- [default model length] (default: {Constant.MODEL_LEN})
+            default_model_width {int} -- [default model width] (default: {Constant.MODEL_WIDTH})
+        """
+
+        if not os.path.exists(path):
+            os.makedirs(path)
+        self.path = os.path.join(os.getcwd(), path)
+        if task == "cv":
+            self.generators = [CnnGenerator]
+        elif task == "common":
+            self.generators = [MlpGenerator]
+        else:
+            raise NotImplementedError('{} task not supported in List ["cv","common"]')
+
+        self.n_classes = n_output_node
+        self.input_shape = (input_width, input_width, input_channel)
+
+        self.t_min = t_min
+        self.beta = beta
+        self.algorithm_name = algorithm_name
+        self.optimize_mode = OptimizeMode(optimize_mode)
+        self.json = None
+        self.total_data = {}
+        self.verbose = verbose
+        self.model_count = 0
+
+        self.bo = BayesianOptimizer(self, self.t_min, self.optimize_mode, self.beta)
+        self.training_queue = []
+        # self.x_queue = []
+        # self.y_queue = []
+        self.descriptors = []
+        self.history = []
+
+        self.max_model_size = max_model_size
+        self.default_model_len = default_model_len
+        self.default_model_width = default_model_width
+
+        self.search_space = dict()
+
+    def update_search_space(self, search_space):
+        """
+        Update search space definition in tuner by search_space in neural architecture.
+        """
+        self.search_space = search_space
+
+    def generate_parameters(self, parameter_id):
+        """
+        Returns a set of trial neural architecture, as a serializable object.
+        parameter_id : int
+        """
+        if not self.history:
+            self.init_search()
+
+        new_father_id = None
+        generated_graph = None
+        if not self.training_queue:
+            new_father_id, generated_graph = self.generate()
+            new_model_id = self.model_count
+            self.model_count += 1
+            self.training_queue.append((generated_graph, new_father_id, new_model_id))
+            self.descriptors.append(generated_graph.extract_descriptor())
+
+        graph, father_id, model_id = self.training_queue.pop(0)
+
+        # from graph to json
+        json_model_path = os.path.join(self.path, str(model_id) + ".json")
+        json_out = graph_to_json(graph, json_model_path)
+        self.total_data[parameter_id] = (json_out, father_id, model_id)
+
+        return json_out
+
+    def receive_trial_result(self, parameter_id, parameters, value):
+        """ Record an observation of the objective function.
+        Arguments:
+            parameter_id : int
+            parameters : dict of parameters
+            value: final metrics of the trial, including reward
+        Raises:
+            RuntimeError -- Received parameter_id not in total_data.
+        """
+
+        reward = self.extract_scalar_reward(value)
+
+        if parameter_id not in self.total_data:
+            raise RuntimeError("Received parameter_id not in total_data.")
+
+        (_, father_id, model_id) = self.total_data[parameter_id]
+
+        graph = self.bo.searcher.load_model_by_id(model_id)
+
+        # to use the value and graph
+        self.add_model(reward, model_id)
+        self.update(father_id, graph, reward, model_id)
+
+    def init_search(self):
+        """Call the generators to generate the initial architectures for the search."""
+        if self.verbose:
+            logger.info("Initializing search.")
+        for generator in self.generators:
+            graph = generator(self.n_classes, self.input_shape).generate(
+                self.default_model_len, self.default_model_width
+            )
+            model_id = self.model_count
+            self.model_count += 1
+            self.training_queue.append((graph, -1, model_id))
+            self.descriptors.append(graph.extract_descriptor())
+
+        if self.verbose:
+            logger.info("Initialization finished.")
+
+    def generate(self):
+        """Generate the next neural architecture.
+        Returns:
+            other_info: Anything to be saved in the training queue together with the architecture.
+            generated_graph: An instance of Graph.
+        """
+        generated_graph, new_father_id = self.bo.generate(self.descriptors)
+        if new_father_id is None:
+            new_father_id = 0
+            generated_graph = self.generators[0](
+                self.n_classes, self.input_shape
+            ).generate(self.default_model_len, self.default_model_width)
+
+        return new_father_id, generated_graph
+
+    def update(self, other_info, graph, metric_value, model_id):
+        """ Update the controller with evaluation result of a neural architecture.
+        Args:
+            other_info: Anything. In our case it is the father ID in the search tree.
+            graph: An instance of Graph. The trained neural architecture.
+            metric_value: The final evaluated metric value.
+            model_id: An integer.
+        """
+        father_id = other_info
+        self.bo.fit([graph.extract_descriptor()], [metric_value])
+        self.bo.add_child(father_id, model_id)
+
+    def add_model(self, metric_value, model_id):
+        """ Add model to the history, x_queue and y_queue
+
+        Arguments:
+            metric_value: int --metric_value
+            graph: dict -- graph
+            model_id: int -- model_id
+
+        Returns:
+            model dict
+        """
+
+        if self.verbose:
+            logger.info("Saving model.")
+
+        # Update best_model text file
+        ret = {"model_id": model_id, "metric_value": metric_value}
+        self.history.append(ret)
+        if model_id == self.get_best_model_id():
+            file = open(os.path.join(self.path, "best_model.txt"), "w")
+            file.write("best model: " + str(model_id))
+            file.close()
+
+        # descriptor = graph.extract_descriptor()
+        # self.x_queue.append(descriptor)
+        # self.y_queue.append(metric_value)
+        return ret
+
+    def get_best_model_id(self):
+        """ Get the best model_id from history using the metric value
+        Returns:
+            int -- the best model_id
+        """
+
+        if self.optimize_mode is OptimizeMode.Maximize:
+            return max(self.history, key=lambda x: x["metric_value"])["model_id"]
+        return min(self.history, key=lambda x: x["metric_value"])["model_id"]
+
+    def load_model_by_id(self, model_id):
+        """Get the model by model_id
+        Arguments:
+            model_id {int} -- model index
+        Returns:
+            Graph -- the model graph representation
+        """
+
+        with open(os.path.join(self.path, str(model_id) + ".json")) as fin:
+            json_str = fin.read().replace("\n", "")
+
+        load_model = json_to_graph(json_str)
+        return load_model
+
+    def load_best_model(self):
+        """ Get the best model by model id
+        Returns:
+            Graph -- the best model graph representation
+        """
+        return self.load_model_by_id(self.get_best_model_id())
+
+    def get_metric_value_by_id(self, model_id):
+        """ Get the model metric valud by its model_id
+        Arguments:
+            model_id {int} -- model index
+        Returns:
+            float -- the model metric
+        """
+        for item in self.history:
+            if item["model_id"] == model_id:
+                return item["metric_value"]
+        return None

src/sdk/pynni/nni/networkmorphism_tuner/nn.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+from abc import abstractmethod
+
+from nni.networkmorphism_tuner.graph import Graph
+from nni.networkmorphism_tuner.layers import (StubDense, StubDropout1d,
+                                              StubReLU, get_batch_norm_class,
+                                              get_conv_class,
+                                              get_dropout_class,
+                                              get_global_avg_pooling_class,
+                                              get_pooling_class)
+from nni.networkmorphism_tuner.utils import Constant
+
+
+class NetworkGenerator:
+    """The base class for generating a network.
+    It can be used to generate a CNN or Multi-Layer Perceptron.
+    Attributes:
+        n_output_node: Number of output nodes in the network.
+        input_shape: A tuple to represent the input shape.
+    """
+
+    def __init__(self, n_output_node, input_shape):
+        self.n_output_node = n_output_node
+        self.input_shape = input_shape
+
+    @abstractmethod
+    def generate(self, model_len, model_width):
+        pass
+
+
+class CnnGenerator(NetworkGenerator):
+    """A class to generate CNN.
+    Attributes:
+          n_dim: `len(self.input_shape) - 1`
+          conv: A class that represents `(n_dim-1)` dimensional convolution.
+          dropout: A class that represents `(n_dim-1)` dimensional dropout.
+          global_avg_pooling: A class that represents `(n_dim-1)` dimensional Global Average Pooling.
+          pooling: A class that represents `(n_dim-1)` dimensional pooling.
+          batch_norm: A class that represents `(n_dim-1)` dimensional batch normalization.
+    """
+
+    def __init__(self, n_output_node, input_shape):
+        super(CnnGenerator, self).__init__(n_output_node, input_shape)
+        self.n_dim = len(self.input_shape) - 1
+        if len(self.input_shape) > 4:
+            raise ValueError("The input dimension is too high.")
+        if len(self.input_shape) < 2:
+            raise ValueError("The input dimension is too low.")
+        self.conv = get_conv_class(self.n_dim)
+        self.dropout = get_dropout_class(self.n_dim)
+        self.global_avg_pooling = get_global_avg_pooling_class(self.n_dim)
+        self.pooling = get_pooling_class(self.n_dim)
+        self.batch_norm = get_batch_norm_class(self.n_dim)
+
+    def generate(self, model_len=None, model_width=None):
+        """Generates a CNN.
+        Args:
+            model_len: An integer. Number of convolutional layers.
+            model_width: An integer. Number of filters for the convolutional layers.
+        Returns:
+            An instance of the class Graph. Represents the neural architecture graph of the generated model.
+        """
+
+        if model_len is None:
+            model_len = Constant.MODEL_LEN
+        if model_width is None:
+            model_width = Constant.MODEL_WIDTH
+        pooling_len = int(model_len / 4)
+        graph = Graph(self.input_shape, False)
+        temp_input_channel = self.input_shape[-1]
+        output_node_id = 0
+        stride = 1
+        for i in range(model_len):
+            output_node_id = graph.add_layer(StubReLU(), output_node_id)
+            output_node_id = graph.add_layer(
+                self.batch_norm(graph.node_list[output_node_id].shape[-1]), output_node_id
+            )
+            output_node_id = graph.add_layer(
+                self.conv(temp_input_channel, model_width, kernel_size=3, stride=stride),
+                output_node_id,
+            )
+            temp_input_channel = model_width
+            if pooling_len == 0 or ((i + 1) % pooling_len == 0 and i != model_len - 1):
+                output_node_id = graph.add_layer(self.pooling(), output_node_id)
+
+        output_node_id = graph.add_layer(self.global_avg_pooling(), output_node_id)
+        output_node_id = graph.add_layer(
+            self.dropout(Constant.CONV_DROPOUT_RATE), output_node_id
+        )
+        output_node_id = graph.add_layer(
+            StubDense(graph.node_list[output_node_id].shape[0], model_width),
+            output_node_id,
+        )
+        output_node_id = graph.add_layer(StubReLU(), output_node_id)
+        graph.add_layer(StubDense(model_width, self.n_output_node), output_node_id)
+        return graph
+
+
+class MlpGenerator(NetworkGenerator):
+    """A class to generate Multi-Layer Perceptron.
+    """
+
+    def __init__(self, n_output_node, input_shape):
+        """Initialize the instance.
+        Args:
+            n_output_node: An integer. Number of output nodes in the network.
+            input_shape: A tuple. Input shape of the network. If it is 1D, ensure the value is appended by a comma
+                in the tuple.
+        """
+        super(MlpGenerator, self).__init__(n_output_node, input_shape)
+        if len(self.input_shape) > 1:
+            raise ValueError("The input dimension is too high.")
+
+    def generate(self, model_len=None, model_width=None):
+        """Generates a Multi-Layer Perceptron.
+        Args:
+            model_len: An integer. Number of hidden layers.
+            model_width: An integer or a list of integers of length `model_len`. If it is a list, it represents the
+                number of nodes in each hidden layer. If it is an integer, all hidden layers have nodes equal to this
+                value.
+        Returns:
+            An instance of the class Graph. Represents the neural architecture graph of the generated model.
+        """
+        if model_len is None:
+            model_len = Constant.MODEL_LEN
+        if model_width is None:
+            model_width = Constant.MODEL_WIDTH
+        if isinstance(model_width, list) and not len(model_width) == model_len:
+            raise ValueError("The length of 'model_width' does not match 'model_len'")
+        elif isinstance(model_width, int):
+            model_width = [model_width] * model_len
+
+        graph = Graph(self.input_shape, False)
+        output_node_id = 0
+        n_nodes_prev_layer = self.input_shape[0]
+        for width in model_width:
+            output_node_id = graph.add_layer(
+                StubDense(n_nodes_prev_layer, width), output_node_id
+            )
+            output_node_id = graph.add_layer(
+                StubDropout1d(Constant.MLP_DROPOUT_RATE), output_node_id
+            )
+            output_node_id = graph.add_layer(StubReLU(), output_node_id)
+            n_nodes_prev_layer = width
+
+        graph.add_layer(StubDense(n_nodes_prev_layer, self.n_output_node), output_node_id)
+        return graph

src/sdk/pynni/nni/networkmorphism_tuner/utils.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+from enum import Enum, unique
+
+@unique
+class OptimizeMode(Enum):
+    """
+    Oprimize Mode class
+    """
+
+    Minimize = "minimize"
+    Maximize = "maximize"
+
+class Constant:
+    '''Constant for the Tuner.
+    '''
+    MAX_LAYERS = 100
+    N_NEIGHBOURS = 8
+    MAX_MODEL_SIZE = 1 << 24
+    KERNEL_LAMBDA = 1.0
+    BETA = 2.576
+    MLP_MODEL_LEN = 3
+    MLP_MODEL_WIDTH = 5
+    MODEL_LEN = 3
+    MODEL_WIDTH = 64
+    POOLING_KERNEL_SIZE = 2
+    DENSE_DROPOUT_RATE = 0.5
+    CONV_DROPOUT_RATE = 0.25
+    MLP_DROPOUT_RATE = 0.25
+    CONV_BLOCK_DISTANCE = 2
+    BATCH_SIZE = 128
+    T_MIN = 0.0001

src/sdk/pynni/tests/test_multi_phase_tuner.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+# associated documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish, distribute,
+# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# ==================================================================================================
+
+
 import logging
 import random
 from io import BytesIO

src/sdk/pynni/tests/test_tuner.py

@@ -23,7 +23,6 @@ import nni.protocol
 from nni.protocol import CommandType, send, receive
 from nni.tuner import Tuner
 from nni.msg_dispatcher import MsgDispatcher
-
 from io import BytesIO
 import json
 from unittest import TestCase, main

tools/nni_cmd/config_schema.py

@@ -58,6 +58,16 @@ Optional('tuner'): Or({
 },{
     'builtinTunerName': Or('BatchTuner', 'GridSearch'),
     Optional('gpuNum'): And(int, lambda x: 0 <= x <= 99999),
+},{
+    'builtinTunerName': 'NetworkMorphism',
+    'classArgs': {
+        Optional('optimize_mode'): Or('maximize', 'minimize'),
+        Optional('task'): And(str, lambda x: x in ['cv','nlp','common']),
+        Optional('input_width'):  int,
+        Optional('input_channel'):  int,
+        Optional('n_output_node'):  int,
+        },
+    Optional('gpuNum'): And(int, lambda x: 0 <= x <= 99999),
 },{
     'codeDir': os.path.exists,
     'classFileName': str,