doc refactor to master (#648)

* initial commit for document refactor (#533)

* enable sphinx

* fix docs/readme link

* update conf

* use mkdocs to build homepage

* Update mkdocs.yml

use docs/README as main

* delete sphinx file

* add nav to mkdocs config

* fix mkdocs bug

* add requirement for online build

* clean requirements.txt

* add contribution

* change requirements file location

* delete sphinx from gitignore

* mnist examples doc (#566)

* mnist examples doc

* update

* update

* update

* update

* update

* Docs refactor of Tutorial (QuickStart, Tuners, Assessors) (#554)

* Refactor of QuickStart

* fix some typo

* Make new changes based on suggestions

* update successful INFO

* update Tuners

* update Tuners:overview

* Update Tuners.md

* Add Assessor.md

* update

* update

* mkdocs.yml

* Update QuickStart.md

* update

* update

* update

* update

* update

* add diff

* modified QuickStart.md an...

docs/index.rst

+#########################################
+Neural Network Intelligence Documentation
+#########################################
+
+********
+Contents
+********
+
+..  toctree::
+    :caption: Table of Contents
+    :maxdepth: 2
+    :glob:
+
+    Overview
+    GetStarted<QuickStart>
+    Tutorials
+    Examples
+    Reference
+    FAQ
+    Contribution
+    Changelog<RELEASE>
\ No newline at end of file

docs/mnist_examples.md

+# MNIST examples
+
+CNN MNIST classifier for deep learning is similar to `hello world` for programming languages. Thus, we use MNIST as example to introduce different features of NNI. The examples are listed below:
+
+ - [MNIST with NNI API](#mnist)
+ - [MNIST with NNI annotation](#mnist-annotation)
+ - [MNIST in keras](#mnist-keras)
+ - [MNIST -- tuning with batch tuner](#mnist-batch)
+ - [MNIST -- tuning with hyperband](#mnist-hyperband)
+ - [MNIST -- tuning within a nested search space](#mnist-nested)
+ - [distributed MNIST (tensorflow) using kubeflow](#mnist-kubeflow-tf)
+ - [distributed MNIST (pytorch) using kubeflow](#mnist-kubeflow-pytorch)
+
+<a name="mnist"></a>
+**MNIST with NNI API**
+
+This is a simple network which has two convolutional layers, two pooling layers and a fully connected layer. We tune hyperparameters, such as dropout rate, convolution size, hidden size, etc. It can be tuned with most NNI built-in tuners, such as TPE, SMAC, Random. We also provide an exmaple yaml file which enables assessor.
+
+`code directory: examples/trials/mnist/`
+
+<a name="mnist-annotation"></a>
+**MNIST with NNI annotation**
+
+This example is similar to the example above, the only difference is that this example uses NNI annotation to specify search space and report results, while the example above uses NNI apis to receive configuration and report results.
+
+`code directory: examples/trials/mnist-annotation/`
+
+<a name="mnist-keras"></a>
+**MNIST in keras**
+
+This example is implemented in keras. It is also a network for MNIST dataset, with two convolution layers, one pooling layer, and two fully connected layers.
+
+`code directory: examples/trials/mnist-keras/`
+
+<a name="mnist-batch"></a>
+**MNIST -- tuning with batch tuner**
+
+This example is to show how to use batch tuner. Users simply list all the configurations they want to try in the search space file. NNI will try all of them.
+
+`code directory: examples/trials/mnist-batch-tune-keras/`
+
+<a name="mnist-hyperband"></a>
+**MNIST -- tuning with hyperband**
+
+This example is to show how to use hyperband to tune the model. There is one more key `STEPS` in the received configuration for trials to control how long it can run (e.g., number of iterations).
+
+`code directory: examples/trials/mnist-hyperband/`
+
+<a name="mnist-nested"></a>
+**MNIST -- tuning within a nested search space**
+
+This example is to show that NNI also support nested search space. The search space file is an example of how to define nested search space.
+
+`code directory: examples/trials/mnist-cascading-search-space/`
+
+<a name="mnist-kubeflow-tf"></a>
+**distributed MNIST (tensorflow) using kubeflow**
+
+This example is to show how to run distributed training on kubeflow through NNI. Users can simply provide distributed training code and a configure file which specifies the kubeflow mode. For example, what is the command to run ps and what is the command to run worker, and how many resources they consume. This example is implemented in tensorflow, thus, uses kubeflow tensorflow operator.
+
+`code directory: examples/trials/mnist-distributed/`
+
+<a name="mnist-kubeflow-pytorch"></a>
+**distributed MNIST (pytorch) using kubeflow**
+
+Similar to the previous example, the difference is that this example is implemented in pytorch, thus, it uses kubeflow pytorch operator.
+
+`code directory: examples/trials/mnist-distributed-pytorch/`
\ No newline at end of file

docs/requirements.txt

+sphinx==1.8.3
+sphinx-argparse==0.2.5
+sphinx-markdown-tables==0.0.9
+sphinx-rtd-theme==0.4.2
+sphinxcontrib-websupport==1.1.0
+recommonmark==0.5.0
+nni==0.5
\ No newline at end of file

docs/sdk_reference.rst

+###########################
+Python API Reference
+###########################
+
+API for trial code
+------------------------
+..  autofunction:: nni.get_next_parameter
+..  autofunction:: nni.get_current_parameter
+..  autofunction:: nni.report_intermediate_result
+..  autofunction:: nni.report_final_result
+..  autofunction:: nni.get_sequence_id
+
+
+API for tuners
+------------------------
+..  autoclass:: nni.tuner.Tuner
+    :members:
+
+..  autoclass:: nni.hyperopt_tuner.hyperopt_tuner.HyperoptTuner
+    :members:
+
+..  autoclass:: nni.batch_tuner.batch_tuner.BatchTuner
+    :members:
+
+..  autoclass:: nni.evolution_tuner.evolution_tuner.EvolutionTuner
+    :members:
+
+..  autoclass:: nni.gridsearch_tuner.gridsearch_tuner.GridSearchTuner
+    :members:
+
+..  autoclass:: nni.networkmorphism_tuner.networkmorphism_tuner.NetworkMorphismTuner
+    :members:
+
+..  autoclass:: nni.smac_tuner.smac_tuner.SMACTuner
+    :members:
+
+API for assessors
+------------------------
+..  autoclass:: nni.assessor.Assessor
+    :members:
+
+..  autoclass:: nni.curvefitting_assessor.curvefitting_assessor.CurvefittingAssessor
+    :members:
+
+..  autoclass:: nni.medianstop_assessor.medianstop_assessor.MedianstopAssessor
+    :members:
+
+
+API for Advisors
+------------------------
+..  autoclass:: nni.hyperband_advisor.hyperband_advisor.Hyperband
\ No newline at end of file

docs/sklearn_examples.md

+# Scikit-learn in NNI
+[Scikit-learn](https://github.com/scikit-learn/scikit-learn) is a pupular meachine learning tool for data mining and data analysis. It supports many kinds of meachine learning models like LinearRegression, LogisticRegression, DecisionTree, SVM etc. How to make the use of scikit-learn more efficiency is a valuable topic.  
+NNI supports many kinds of tuning algorithms to search the best models and/or hyper-parameters for scikit-learn, and support many kinds of environments like local machine, remote servers and cloud.
+ 
+## 1. How to run the example
+To start using NNI, you should install the nni package, and use the command line tool `nnictl` to start an experiment. For more information about installation and preparing for the environment,  please [refer](GetStarted.md).
+After you installed NNI, you could enter the corresponding folder and start the experiment using following commands:
+```
+nnictl create --config ./config.yml
+```
+
+## 2. Description of the example
+
+
+### 2.1 classification
+This example uses the dataset of digits, which is made up of 1797 8x8 images, and each image is a hand-written digit, the goal is to classify these images into 10 classes.  
+In this example, we use SVC as the model, and choose some parameters of this model, including `"C", "keral", "degree", "gamma" and "coef0"`. For more information of these parameters, please [refer](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).
+
+
+### 2.2 regression
+This example uses the Boston Housing Dataset, this dataset consists of price of houses in various places in Boston and the information such as Crime (CRIM), areas of non-retail business in the town (INDUS), the age of people who own the house (AGE) etc to predict the house price of boston.
+In this example, we tune different kinds of regression models including `"LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"` and some parameters like `"svr_kernel", "knr_weights"`. You could get more details about these models from [here](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning).
+
+## 3. How to write sklearn code using nni
+It is easy to use nni in your sklearn code, there are only a few steps.
+* __step 1__  
+  Prepare a search_space.json to storage your choose spaces. 
+  For example, if you want to choose different models, you may try:
+  ```
+  {
+    "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]}
+  }
+  ```
+  If you want to choose different models and parameters, you could put them together in a search_space.json file.
+  ```
+  {
+    "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]},
+    "svr_kernel": {"_type":"choice","_value":["linear", "poly", "rbf"]},
+    "knr_weights": {"_type":"choice","_value":["uniform", "distance"]}
+  }
+  ```
+  Then you could read these values as a dict from your python code, please get into the step 2.
+* __step 2__  
+  At the beginning of your python code, you should `import nni` to insure the packages works normally.
+  First, you should use `nni.get_next_parameter()` function to get your parameters given by nni. Then you could use these parameters to update your code.
+  For example, if you define your search_space.json like following format:
+  ```
+  {
+    "C": {"_type":"uniform","_value":[0.1, 1]},
+    "keral": {"_type":"choice","_value":["linear", "rbf", "poly", "sigmoid"]},
+    "degree": {"_type":"choice","_value":[1, 2, 3, 4]},
+    "gamma": {"_type":"uniform","_value":[0.01, 0.1]},
+    "coef0 ": {"_type":"uniform","_value":[0.01, 0.1]}
+  }
+  ```
+  You may get a parameter dict like this:
+  ```
+  params = {
+        'C': 1.0,
+        'keral': 'linear',
+        'degree': 3,
+        'gamma': 0.01,
+        'coef0': 0.01
+  }
+  ```
+  Then you could use these variables to write your scikit-learn code.
+* __step 3__  
+  After you finished your training, you could get your own score of the model, like your percision, recall or MSE etc. NNI needs your score to tuner algorithms and generate next group of parameters, please report the score back to NNI and start next trial job.   
+  You just need to use `nni.report_final_result(score)` to communitate with NNI after you process your scikit-learn code. Or if you have multiple scores in the steps of training, you could also report them back to NNI using `nni.report_intemediate_result(score)`. Note, you may not report intemediate result of your job, but you must report back your final result.

docs/training_services.rst

+Introduction to NNI Training Services
+=====================================
+
+..  toctree::
+    Local<tutorial_1_CR_exp_local_api>
+    Remote<RemoteMachineMode>
+    PAI<PAIMode>
+    Kubeflow<KubeflowMode>
+    FrameworkController Mode<FrameworkControllerMode>
\ No newline at end of file

docs/tuners.rst

+#################
+Tuners
+#################
+
+Overview
+-----------------
+
+NNI provides an easy way to adopt an approach to set up parameter tuning algorithms, we call them **Tuner**.
+
+Tuner receives the result from `Trial` as a matrix to evaluate the performance of a specific parameters/architecture configures. And tuner sends next hyper-parameter or architecture configure to Trial.
+
+In NNI, we support two approaches to set the tuner: first is directly use builtin tuner provided by nni sdk, second is customize a tuner file by yourself. We also have Advisor that combines the functinality of Tuner & Assessor.
+
+For details, please refer to the following tutorials:
+
+..  toctree::
+    Builtin Tuners<Builtin_Tuner>
+    Customized Tuners<Customize_Tuner>
+    Customized Advisor<Customize_Advisor>
\ No newline at end of file

examples/trials/mnist/config_assessor.yml

@@ -2,7 +2,7 @@ authorName: default
 experimentName: example_mnist
 trialConcurrency: 1
 maxExecDuration: 1h
-maxTrialNum: 20
+maxTrialNum: 50
 #choice: local, remote
 trainingServicePlatform: local
 searchSpacePath: search_space.json
@@ -17,10 +17,12 @@ tuner:
     optimize_mode: maximize
 assessor:
   #choice: Medianstop, Curvefitting
-  builtinAssessorName: Medianstop
+  builtinAssessorName: Curvefitting
   classArgs:
     #choice: maximize, minimize
     optimize_mode: maximize
+    epoch_num: 20
+    threshold: 0.9
 trial:
   command: python3 mnist.py
   codeDir: .

examples/trials/mnist/mnist_before.py

+"""A deep MNIST classifier using convolutional layers."""
+import argparse
+import logging
+import math
+import tempfile
+import tensorflow as tf
+
+from tensorflow.examples.tutorials.mnist import input_data
+
+FLAGS = None
+
+logger = logging.getLogger('mnist_AutoML')
+
+
+class MnistNetwork(object):
+    '''
+    MnistNetwork is for initializing and building basic network for mnist.
+    '''
+
+    def __init__(self,
+                 channel_1_num,
+                 channel_2_num,
+                 conv_size,
+                 hidden_size,
+                 pool_size,
+                 learning_rate,
+                 x_dim=784,
+                 y_dim=10):
+        self.channel_1_num = channel_1_num
+        self.channel_2_num = channel_2_num
+        self.conv_size = conv_size
+        self.hidden_size = hidden_size
+        self.pool_size = pool_size
+        self.learning_rate = learning_rate
+        self.x_dim = x_dim
+        self.y_dim = y_dim
+
+        self.images = tf.placeholder(
+            tf.float32, [None, self.x_dim], name='input_x')
+        self.labels = tf.placeholder(
+            tf.float32, [None, self.y_dim], name='input_y')
+        self.keep_prob = tf.placeholder(tf.float32, name='keep_prob')
+
+        self.train_step = None
+        self.accuracy = None
+
+    def build_network(self):
+        '''
+        Building network for mnist
+        '''
+
+        # Reshape to use within a convolutional neural net.
+        # Last dimension is for "features" - there is only one here, since images are
+        # grayscale -- it would be 3 for an RGB image, 4 for RGBA, etc.
+        with tf.name_scope('reshape'):
+            try:
+                input_dim = int(math.sqrt(self.x_dim))
+            except:
+                print(
+                    'input dim cannot be sqrt and reshape. input dim: ' + str(self.x_dim))
+                logger.debug(
+                    'input dim cannot be sqrt and reshape. input dim: %s', str(self.x_dim))
+                raise
+            x_image = tf.reshape(self.images, [-1, input_dim, input_dim, 1])
+
+        # First convolutional layer - maps one grayscale image to 32 feature maps.
+        with tf.name_scope('conv1'):
+            w_conv1 = weight_variable(
+                [self.conv_size, self.conv_size, 1, self.channel_1_num])
+            b_conv1 = bias_variable([self.channel_1_num])
+            h_conv1 = tf.nn.relu(conv2d(x_image, w_conv1) + b_conv1)
+
+        # Pooling layer - downsamples by 2X.
+        with tf.name_scope('pool1'):
+            h_pool1 = max_pool(h_conv1, self.pool_size)
+
+        # Second convolutional layer -- maps 32 feature maps to 64.
+        with tf.name_scope('conv2'):
+            w_conv2 = weight_variable([self.conv_size, self.conv_size,
+                                       self.channel_1_num, self.channel_2_num])
+            b_conv2 = bias_variable([self.channel_2_num])
+            h_conv2 = tf.nn.relu(conv2d(h_pool1, w_conv2) + b_conv2)
+
+        # Second pooling layer.
+        with tf.name_scope('pool2'):
+            h_pool2 = max_pool(h_conv2, self.pool_size)
+
+        # Fully connected layer 1 -- after 2 round of downsampling, our 28x28 image
+        # is down to 7x7x64 feature maps -- maps this to 1024 features.
+        last_dim = int(input_dim / (self.pool_size * self.pool_size))
+        with tf.name_scope('fc1'):
+            w_fc1 = weight_variable(
+                [last_dim * last_dim * self.channel_2_num, self.hidden_size])
+            b_fc1 = bias_variable([self.hidden_size])
+
+        h_pool2_flat = tf.reshape(
+            h_pool2, [-1, last_dim * last_dim * self.channel_2_num])
+        h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, w_fc1) + b_fc1)
+
+        # Dropout - controls the complexity of the model, prevents co-adaptation of features.
+        with tf.name_scope('dropout'):
+            h_fc1_drop = tf.nn.dropout(h_fc1, self.keep_prob)
+
+        # Map the 1024 features to 10 classes, one for each digit
+        with tf.name_scope('fc2'):
+            w_fc2 = weight_variable([self.hidden_size, self.y_dim])
+            b_fc2 = bias_variable([self.y_dim])
+            y_conv = tf.matmul(h_fc1_drop, w_fc2) + b_fc2
+
+        with tf.name_scope('loss'):
+            cross_entropy = tf.reduce_mean(
+                tf.nn.softmax_cross_entropy_with_logits(labels=self.labels, logits=y_conv))
+        with tf.name_scope('adam_optimizer'):
+            self.train_step = tf.train.AdamOptimizer(
+                self.learning_rate).minimize(cross_entropy)
+
+        with tf.name_scope('accuracy'):
+            correct_prediction = tf.equal(
+                tf.argmax(y_conv, 1), tf.argmax(self.labels, 1))
+            self.accuracy = tf.reduce_mean(
+                tf.cast(correct_prediction, tf.float32))
+
+
+def conv2d(x_input, w_matrix):
+    """conv2d returns a 2d convolution layer with full stride."""
+    return tf.nn.conv2d(x_input, w_matrix, strides=[1, 1, 1, 1], padding='SAME')
+
+
+def max_pool(x_input, pool_size):
+    """max_pool downsamples a feature map by 2X."""
+    return tf.nn.max_pool(x_input, ksize=[1, pool_size, pool_size, 1],
+                          strides=[1, pool_size, pool_size, 1], padding='SAME')
+
+
+def weight_variable(shape):
+    """weight_variable generates a weight variable of a given shape."""
+    initial = tf.truncated_normal(shape, stddev=0.1)
+    return tf.Variable(initial)
+
+
+def bias_variable(shape):
+    """bias_variable generates a bias variable of a given shape."""
+    initial = tf.constant(0.1, shape=shape)
+    return tf.Variable(initial)
+
+
+def main(params):
+    '''
+    Main function, build mnist network, run and send result to NNI.
+    '''
+    # Import data
+    mnist = input_data.read_data_sets(params['data_dir'], one_hot=True)
+    print('Mnist download data down.')
+    logger.debug('Mnist download data down.')
+
+    # Create the model
+    # Build the graph for the deep net
+    mnist_network = MnistNetwork(channel_1_num=params['channel_1_num'],
+                                 channel_2_num=params['channel_2_num'],
+                                 conv_size=params['conv_size'],
+                                 hidden_size=params['hidden_size'],
+                                 pool_size=params['pool_size'],
+                                 learning_rate=params['learning_rate'])
+    mnist_network.build_network()
+    logger.debug('Mnist build network done.')
+
+    # Write log
+    graph_location = tempfile.mkdtemp()
+    logger.debug('Saving graph to: %s', graph_location)
+    train_writer = tf.summary.FileWriter(graph_location)
+    train_writer.add_graph(tf.get_default_graph())
+
+    test_acc = 0.0
+    with tf.Session() as sess:
+        sess.run(tf.global_variables_initializer())
+        for i in range(params['batch_num']):
+            batch = mnist.train.next_batch(params['batch_size'])
+            mnist_network.train_step.run(feed_dict={mnist_network.images: batch[0],
+                                                    mnist_network.labels: batch[1],
+                                                    mnist_network.keep_prob: 1 - params['dropout_rate']}
+                                         )
+
+            if i % 100 == 0:
+                test_acc = mnist_network.accuracy.eval(
+                    feed_dict={mnist_network.images: mnist.test.images,
+                               mnist_network.labels: mnist.test.labels,
+                               mnist_network.keep_prob: 1.0})
+
+                logger.debug('test accuracy %g', test_acc)
+                logger.debug('Pipe send intermediate result done.')
+
+        test_acc = mnist_network.accuracy.eval(
+            feed_dict={mnist_network.images: mnist.test.images,
+                       mnist_network.labels: mnist.test.labels,
+                       mnist_network.keep_prob: 1.0})
+
+        logger.debug('Final result is %g', test_acc)
+        logger.debug('Send final result done.')
+
+def get_params():
+    ''' Get parameters from command line '''
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--data_dir", type=str, default='/tmp/tensorflow/mnist/input_data', help="data directory")
+    parser.add_argument("--dropout_rate", type=float, default=0.5, help="dropout rate")
+    parser.add_argument("--channel_1_num", type=int, default=32)
+    parser.add_argument("--channel_2_num", type=int, default=64)
+    parser.add_argument("--conv_size", type=int, default=5)
+    parser.add_argument("--pool_size", type=int, default=2)
+    parser.add_argument("--hidden_size", type=int, default=1024)
+    parser.add_argument("--learning_rate", type=float, default=1e-4)
+    parser.add_argument("--batch_num", type=int, default=2000)
+    parser.add_argument("--batch_size", type=int, default=32)
+
+    args, _ = parser.parse_known_args()
+    return args
+
+if __name__ == '__main__':
+    try:
+        params = vars(get_params())
+        main(params)
+    except Exception as exception:
+        logger.exception(exception)
+        raise