Merge pull request #126 from Microsoft/master

merge master

Merge pull request #126 from Microsoft/master
merge master
1d845267 · SparkSnail · GitHub · 40391ec2 · a6c2ac2b · 1d845267
Unverified Commit 1d845267 authored Feb 20, 2019 by SparkSnail Committed by GitHub Feb 20, 2019
20 changed files
--- a/docs/Makefile
+++ b/docs/Makefile
--- a/docs/NNICTLDOC.md
+++ b/docs/NNICTLDOC.md
--- a/docs/Overview.md
+++ b/docs/Overview.md
--- a/docs/PAIMode.md
+++ b/docs/PAIMode.md
@@ -50,7 +50,7 @@ Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial con
    * Required key. Should be positive number based on your trial program's memory requirement
 * image
    * Required key. In pai mode, your trial program will be scheduled by OpenPAI to run in [Docker container](https://www.docker.com/). This key is used to specify the Docker image used to create the container in which your trial will run.
-    * We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
+    * We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/docker/Dockerfile). You can either use this image directly in your config file, or build your own image based on it.
 * dataDir
    * Optional key. It specifies the HDFS data direcotry for trial to download data. The format should be something like hdfs://{your HDFS host}:9000/{your data directory}
 * outputDir 
@@ -62,20 +62,20 @@ nnictl create --config exp_pai.yml
 ```
 to start the experiment in pai mode. NNI will create OpenPAI job for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`. 
 You can see jobs created by NNI in the OpenPAI cluster's web portal, like:
-![](./img/nni_pai_joblist.jpg)
+![](../img/nni_pai_joblist.jpg)
 Notice: In pai mode, NNIManager will start a rest server and listen on a port which is your NNI WebUI's port plus 1. For example, if your WebUI port is `8080`, the rest server will listen on `8081`, to receive metrics from trial job running in Kubernetes. So you should `enable 8081` TCP port in your firewall rule to allow incoming traffic. 
 Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information. 
 Expand a trial information in trial list view, click the logPath link like:
-![](./img/nni_webui_joblist.jpg)
+![](../img/nni_webui_joblist.jpg)
 And you will be redirected to HDFS web portal to browse the output files of that trial in HDFS:
-![](./img/nni_trial_hdfs_output.jpg)
+![](../img/nni_trial_hdfs_output.jpg)
 You can see there're three fils in output folder: stderr, stdout, and trial.log
 If you also want to save trial's other output into HDFS, like model files, you can use environment variable `NNI_OUTPUT_DIR` in your trial code to save your own output files, and NNI SDK will copy all the files in `NNI_OUTPUT_DIR` from trial's container to HDFS. 
-Any problems when using NNI in pai mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni).
+Any problems when using NNI in pai mode, please create issues on [NNI github repo](https://github.com/Microsoft/nni).
--- a/docs/QuickStart.md
+++ b/docs/QuickStart.md
@@ -183,28 +183,28 @@ Click the tab "Overview".
 Information about this experiment will be shown in the WebUI, including the experiment trial profile and search space message. NNI also support `download these information and parameters` through the **Download** button. You can download the experiment result anytime in the middle for the running or at the end of the execution, etc.
-![](./img/QuickStart1.png)
+![](../img/QuickStart1.png)
 Top 10 trials will be listed in the Overview page, you can browse all the trials in "Trials Detail" page.
-![](./img/QuickStart2.png)
+![](../img/QuickStart2.png)
 #### View trials detail page
 Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
-![](./img/QuickStart3.png)
+![](../img/QuickStart3.png)
 Click the tab "Hyper Parameter" to see the parallel graph.
 * You can select the percentage to see top trials.
 * Choose two axis to swap its positions
-![](./img/QuickStart4.png)
+![](../img/QuickStart4.png)
 Click the tab "Trial Duration" to see the bar graph.
-![](./img/QuickStart5.png)
+![](../img/QuickStart5.png)
 Below is the status of the all trials. Specifically:
@@ -213,11 +213,11 @@ Below is the status of the all trials. Specifically:
 * Kill: you can kill a job that status is running.
 * Support to search for a specific trial.
-![](./img/QuickStart6.png)
+![](../img/QuickStart6.png)
 * Intermediate Result Grap
-![](./img/QuickStart7.png)
+![](../img/QuickStart7.png)
 ## Related Topic

--- a/docs/RELEASE.md
+++ b/docs/RELEASE.md
--- a/docs/Reference.rst
+++ b/docs/Reference.rst
--- a/docs/RemoteMachineMode.md
+++ b/docs/RemoteMachineMode.md
--- a/docs/SQuAD_evolution_examples.md
+++ b/docs/SQuAD_evolution_examples.md
 # Automatic Model Architecture Search for Reading Comprehension
 This example shows us how to use Genetic Algorithm to find good model architectures for Reading Comprehension.
 ## 1. Search Space
 Since attention and recurrent neural network (RNN) have been proven effective in Reading Comprehension.
 We conclude the search space as follow:
 1. IDENTITY (Effectively means keep training).
 2. INSERT-RNN-LAYER (Inserts a LSTM. Comparing the performance of GRU and LSTM in our experiment, we decided to use LSTM here.)
 3. REMOVE-RNN-LAYER
 4. INSERT-ATTENTION-LAYER(Inserts an attention layer.)
 5. REMOVE-ATTENTION-LAYER
 6. ADD-SKIP (Identity between random layers).
 7. REMOVE-SKIP (Removes random skip).
-![](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad/ga_squad.png)
+![](../../examples/trials/ga_squad/ga_squad.png)
 ### New version
 Also we have another version which time cost is less and performance is better. We will release soon.
 ## 2. How to run this example in local?
 ### 2.1 Use downloading script to download data
 Execute the following command to download needed files
 using the downloading script:
 ```bash
 chmod +x ./download.sh
 ./download.sh
 ```
 Or Download manually
 1. download "dev-v1.1.json" and "train-v1.1.json" in https://rajpurkar.github.io/SQuAD-explorer/
 ```bash
 wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
 wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
 ```
 2. download "glove.840B.300d.txt" in https://nlp.stanford.edu/projects/glove/
 ```bash
 wget http://nlp.stanford.edu/data/glove.840B.300d.zip
 unzip glove.840B.300d.zip
 ```
 ### 2.2 Update configuration
 Modify `nni/examples/trials/ga_squad/config.yml`, here is the default configuration:
 ```yaml
 authorName: default
 experimentName: example_ga_squad
 trialConcurrency: 1
 maxExecDuration: 1h
 maxTrialNum: 1
 #choice: local, remote
 trainingServicePlatform: local
 #choice: true, false
 useAnnotation: false
 tuner:
  codeDir: ~/nni/examples/tuners/ga_customer_tuner
  classFileName: customer_tuner.py
  className: CustomerTuner
  classArgs:
    optimize_mode: maximize
 trial:
  command: python3 trial.py
  codeDir: ~/nni/examples/trials/ga_squad
  gpuNum: 0
 ```
 In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
 ### 2.3 submit this job
 ```bash
 nnictl create --config ~/nni/examples/trials/ga_squad/config.yml
 ```
 ## 3 Run this example on OpenPAI
 Due to the memory limitation of upload, we only upload the source code and complete the data download and training on OpenPAI. This experiment requires sufficient memory that `memoryMB >= 32G`, and the training may last for several hours.
 ### 3.1 Update configuration
 Modify `nni/examples/trials/ga_squad/config_pai.yml`, here is the default configuration:
 ```yaml
 authorName: default
 experimentName: example_ga_squad
 trialConcurrency: 1
 maxExecDuration: 1h
 maxTrialNum: 10
 #choice: local, remote, pai
 trainingServicePlatform: pai
 #choice: true, false
 useAnnotation: false
 #Your nni_manager ip
 nniManagerIp: 10.10.10.10
 tuner:
  codeDir: https://github.com/Microsoft/nni/tree/master/examples/tuners/ga_customer_tuner
  classFileName: customer_tuner.py
  className: CustomerTuner
  classArgs:
    optimize_mode: maximize
 trial:
  command: chmod +x ./download.sh && ./download.sh && python3 trial.py
  codeDir: .
  gpuNum: 0
  cpuNum: 1
  memoryMB: 32869
  #The docker image to run nni job on OpenPAI
  image: msranni/nni:latest
  #The hdfs directory to store data on OpenPAI, format 'hdfs://host:port/directory'
  dataDir: hdfs://10.10.10.10:9000/username/nni
  #The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
  outputDir: hdfs://10.10.10.10:9000/username/nni
 paiConfig:
  #The username to login OpenPAI
  userName: username
  #The password to login OpenPAI
  passWord: password
  #The host of restful server of OpenPAI
  host: 10.10.10.10
 ```
 Please change the default value to your personal account and machine information. Including `nniManagerIp`, `dataDir`, `outputDir`, `userName`, `passWord` and `host`.
 In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
 `trialConcurrency` is the number of trials running concurrently, which is the number of GPUs you want to use, if you are setting `gpuNum` to 1.
 ### 3.2 submit this job
 ```bash
 nnictl create --config ~/nni/examples/trials/ga_squad/config_pai.yml
 ```
 ## 4. Technical details about the trial
 ### 4.1 How does it works
 The evolution-algorithm based architecture for question answering has two different parts just like any other examples: the trial and the tuner.
 ### 4.2 The trial
 The trial has a lot of different files, functions and classes. Here we will only give most of those files a brief introduction:
 * `attention.py` contains an implementation for attention mechanism in Tensorflow.
 * `data.py` contains functions for data preprocessing.
 * `evaluate.py` contains the evaluation script.
 * `graph.py` contains the definition of the computation graph.
 * `rnn.py` contains an implementation for GRU in Tensorflow.
 * `train_model.py` is a wrapper for the whole question answering model.
 Among those files, `trial.py` and `graph_to_tf.py` are special.
 `graph_to_tf.py` has a function named as `graph_to_network`, here is its skeleton code:
 ```python
 def graph_to_network(input1,
                     input2,
                     input1_lengths,
                     input2_lengths,
                     graph,
                     dropout_rate,
                     is_training,
                     num_heads=1,
                     rnn_units=256):
    topology = graph.is_topology()
    layers = dict()
    layers_sequence_lengths = dict()
    num_units = input1.get_shape().as_list()[-1]
    layers[0] = input1*tf.sqrt(tf.cast(num_units, tf.float32)) + \
        positional_encoding(input1, scale=False, zero_pad=False)
    layers[1] = input2*tf.sqrt(tf.cast(num_units, tf.float32))
    layers[0] = dropout(layers[0], dropout_rate, is_training)
    layers[1] = dropout(layers[1], dropout_rate, is_training)
    layers_sequence_lengths[0] = input1_lengths
    layers_sequence_lengths[1] = input2_lengths
    for _, topo_i in enumerate(topology):
        if topo_i == '|':
            continue
        if graph.layers[topo_i].graph_type == LayerType.input.value:
            # ......
        elif graph.layers[topo_i].graph_type == LayerType.attention.value:
            # ......
        # More layers to handle
 ```
 As we can see, this function is actually a compiler, that converts the internal model DAG configuration (which will be introduced in the `Model configuration format` section) `graph`, to a Tensorflow computation graph.
 ```python
 topology = graph.is_topology()
 ```
 performs topological sorting on the internal graph representation, and the code inside the loop:
 ```python
 for _, topo_i in enumerate(topology):
 ```
 performs actually conversion that maps each layer to a part in Tensorflow computation graph.
 ### 4.3 The tuner
 The tuner is much more simple than the trial. They actually share the same `graph.py`. Besides, the tuner has a `customer_tuner.py`, the most important class in which is `CustomerTuner`:
 ```python
 class CustomerTuner(Tuner):
    # ......
    def generate_parameters(self, parameter_id):
        """Returns a set of trial graph config, as a serializable object.
        parameter_id : int
        """
        if len(self.population) <= 0:
            logger.debug("the len of poplution lower than zero.")
            raise Exception('The population is empty')
        pos = -1
        for i in range(len(self.population)):
            if self.population[i].result == None:
                pos = i
                break
        if pos != -1:
            indiv = copy.deepcopy(self.population[pos])
            self.population.pop(pos)
            temp = json.loads(graph_dumps(indiv.config))
        else:
            random.shuffle(self.population)
            if self.population[0].result > self.population[1].result:
                self.population[0] = self.population[1]
            indiv = copy.deepcopy(self.population[0])
            self.population.pop(1)
            indiv.mutation()
            graph = indiv.config
            temp =  json.loads(graph_dumps(graph))
    # ......
 ```
 As we can see, the overloaded method `generate_parameters` implements a pretty naive mutation algorithm. The code lines:
 ```python
            if self.population[0].result > self.population[1].result:
                self.population[0] = self.population[1]
            indiv = copy.deepcopy(self.population[0])
 ```
 controls the mutation process. It will always take two random individuals in the population, only keeping and mutating the one with better result.
 ### 4.4 Model configuration format
 Here is an example of the model configuration, which is passed from the tuner to the trial in the architecture search procedure.
 ```json
 {
    "max_layer_num": 50,
    "layers": [
        {
            "input_size": 0,
            "type": 3,
            "output_size": 1,
            "input": [],
            "size": "x",
            "output": [4, 5],
            "is_delete": false
        },
        {
            "input_size": 0,
            "type": 3,
            "output_size": 1,
            "input": [],
            "size": "y",
            "output": [4, 5],
            "is_delete": false
        },
        {
            "input_size": 1,
            "type": 4,
            "output_size": 0,
            "input": [6],
            "size": "x",
            "output": [],
            "is_delete": false
        },
        {
            "input_size": 1,
            "type": 4,
            "output_size": 0,
            "input": [5],
            "size": "y",
            "output": [],
            "is_delete": false
        },
        {"Comment": "More layers will be here for actual graphs."}
    ]
 }
 ```
 Every model configuration will have a "layers" section, which is a JSON list of layer definitions. The definition of each layer is also a JSON object, where:
 * `type` is the type of the layer. 0, 1, 2, 3, 4 corresponds to attention, self-attention, RNN, input and output layer respectively.
 * `size` is the length of the output. "x", "y" correspond to document length / question length, respectively.
 * `input_size` is the number of inputs the layer has.
 * `input` is the indices of layers taken as input of this layer.
 * `output` is the indices of layers use this layer's output as their input.
 * `is_delete` means whether the layer is still available.
--- a/docs/SearchSpaceSpec.md
+++ b/docs/SearchSpaceSpec.md
--- a/docs/SetupNNIDeveloperEnvironment.md
+++ b/docs/SetupNNIDeveloperEnvironment.md
--- a/docs/Trials.md
+++ b/docs/Trials.md
@@ -61,7 +61,7 @@ searchSpacePath: /path/to/your/search_space.json
 You can refer to [here](ExperimentConfig.md) for more information about how to set up experiment configurations.
-*Please refer to [here](sdk_reference.md) for more APIs (e.g., `nni.get_sequence_id()`) provided by NNI.
+*Please refer to [here](https://nni.readthedocs.io/en/latest/sdk_reference.html) for more APIs (e.g., `nni.get_sequence_id()`) provided by NNI.
 <a name="nni-annotation"></a>

--- a/docs/Tutorials.rst
+++ b/docs/Tutorials.rst
--- a/docs/WebUI.md
+++ b/docs/WebUI.md
@@ -7,16 +7,16 @@ Click the tab "Overview".
 * See the experiment trial profile and search space message.
 * Support to download the experiment result.
-![](./img/webui-img/over1.png)
+![](../img/webui-img/over1.png)
 * See good performance trials.
-![](./img/webui-img/over2.png)
+![](../img/webui-img/over2.png)
 ## View job default metric
 Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
-![](./img/accuracy.png)
+![](../img/accuracy.png)
 ## View hyper parameter
@@ -25,13 +25,13 @@ Click the tab "Hyper Parameter" to see the parallel graph.
 * You can select the percentage to see top trials.
 * Choose two axis to swap its positions
-![](./img/hyperPara.png)
+![](../img/hyperPara.png)
 ## View Trial Duration
 Click the tab "Trial Duration" to see the bar graph.
-![](./img/trial_duration.png)
+![](../img/trial_duration.png)
 ## View trials status 
@@ -39,15 +39,15 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically:
 * Trial detail: trial's id, trial's duration, start time, end time, status, accuracy and search space file.
-![](./img/webui-img/detail-local.png)
+![](../img/webui-img/detail-local.png)
 * If you run on OpenPAI or Kubeflow platform, you can also see the hdfsLog.
-![](./img/webui-img/detail-pai.png)
+![](../img/webui-img/detail-pai.png)
 * Kill: you can kill a job that status is running.
 * Support to search for a specific trial.
 * Intermediate Result Graph.
-![](./img/intermediate.png)
+![](../img/intermediate.png)
--- a/docs/advanced.rst
+++ b/docs/advanced.rst
--- a/docs/assessors.rst
+++ b/docs/assessors.rst
@@ -8,7 +8,7 @@ Here is an experimental result of MNIST after using 'Curvefitting' Assessor in '
 *Implemented code directory: config_assessor.yml <https://github.com/Microsoft/nni/blob/master/examples/trials/mnist/config_assessor.yml>*
-..  image:: ./img/Assessor.png
+..  image:: ../img/Assessor.png
 Like Tuners, users can either use built-in Assessors, or customize an Assessor on their own. Please refer to the following tutorials for detail:

--- a/docs/cifar10_examples.md
+++ b/docs/cifar10_examples.md
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -44,6 +44,7 @@ extensions = [
    'sphinx.ext.mathjax',
    'sphinx_markdown_tables',
    'sphinxarg.ext',
+    'sphinx.ext.napoleon',
 ]
 # Add any paths that contain templates here, relative to this directory.
@@ -107,7 +108,7 @@ html_theme_options = {
 #
 # html_sidebars = {}
-html_logo = './img/nni_logo_dark.png'
+html_logo = '../img/nni_logo_dark.png'
 # -- Options for HTMLHelp output ---------------------------------------------

--- a/docs/gbdt_example.md
+++ b/docs/gbdt_example.md
 # GBDT in nni
 Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion as other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. 
 Gradient boosting decision tree has many popular implementations, such as [lightgbm](https://github.com/Microsoft/LightGBM), [xgboost](https://github.com/dmlc/xgboost), and [catboost](https://github.com/catboost/catboost), etc. GBDT is a great tool for solving the problem of traditional machine learning problem. Since GBDT is a robust algorithm, it could use in many domains. The better hyper-parameters for GBDT, the better performance you could achieve.
 NNI is a great platform for tuning hyper-parameters, you could try various builtin search algorithm in nni and run multiple trials concurrently. 
 ## 1. Search Space in GBDT
 There are many hyper-parameters in GBDT, but what kind of parameters will affect the performance or speed? Based on some practical experience, some suggestion here(Take lightgbm as example):
 > * For better accuracy
 * `learning_rate`. The range of `learning rate` could be [0.001, 0.9].
 * `num_leaves`. `num_leaves` is related to `max_depth`, you don't have to tune both of them.
 * `bagging_freq`. `bagging_freq` could be [1, 2, 4, 8, 10]
 * `num_iterations`. May larger if underfitting.
 > * For speed up
 * `bagging_fraction`. The range of `bagging_fraction` could be [0.7, 1.0].
 * `feature_fraction`. The range of `feature_fraction` could be [0.6, 1.0].
 * `max_bin`.
 > * To avoid overfitting
 * `min_data_in_leaf`. This depends on your dataset.
 * `min_sum_hessian_in_leaf`. This depend on your dataset.
 * `lambda_l1` and `lambda_l2`.
 * `min_gain_to_split`.
 * `num_leaves`.
 Reference link:
 [lightgbm](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) and [autoxgoboost](https://github.com/ja-thomas/autoxgboost/blob/master/poster_2018.pdf)
 ## 2. Task description
 Now we come back to our example "auto-gbdt" which run in lightgbm and nni. The data including [train data](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/data/regression.train) and [test data](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/data/regression.train).
 Given the features and label in train data, we train a GBDT regression model and use it to predict.
 ## 3. How to run in nni
 ### 3.1 Prepare your trial code
 You need to prepare a basic code as following:
 ```python
 ...
 def get_default_parameters():
    ...
    return params
 def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
    '''
    Load or create dataset
    '''
    ...
    return lgb_train, lgb_eval, X_test, y_test
 def run(lgb_train, lgb_eval, params, X_test, y_test):
    # train
    gbm = lgb.train(params,
                    lgb_train,
                    num_boost_round=20,
                    valid_sets=lgb_eval,
                    early_stopping_rounds=5)
    # predict
    y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
    # eval
    rmse = mean_squared_error(y_test, y_pred) ** 0.5
    print('The rmse of prediction is:', rmse)
 if __name__ == '__main__':
    lgb_train, lgb_eval, X_test, y_test = load_data()
    PARAMS = get_default_parameters()
    # train
    run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
 ```
 ### 3.2 Prepare your search space.
 If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `bagging_freq`, you could write a [search_space.json](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/search_space.json) as follow:
 ```json
 {
    "num_leaves":{"_type":"choice","_value":[31, 28, 24, 20]},
    "learning_rate":{"_type":"choice","_value":[0.01, 0.05, 0.1, 0.2]},
    "bagging_fraction":{"_type":"uniform","_value":[0.7, 1.0]},
    "bagging_freq":{"_type":"choice","_value":[1, 2, 4, 8, 10]}
 }
 ```
 More support variable type you could reference [here](SearchSpaceSpec.md).
 ### 3.3 Add SDK of nni into your code.
 ```diff
 +import nni
 ...
 def get_default_parameters():
    ...
    return params
 def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
    '''
    Load or create dataset
    '''
    ...
    return lgb_train, lgb_eval, X_test, y_test
 def run(lgb_train, lgb_eval, params, X_test, y_test):
    # train
    gbm = lgb.train(params,
                    lgb_train,
                    num_boost_round=20,
                    valid_sets=lgb_eval,
                    early_stopping_rounds=5)
    # predict
    y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
    # eval
    rmse = mean_squared_error(y_test, y_pred) ** 0.5
    print('The rmse of prediction is:', rmse)
 +   nni.report_final_result(rmse)
 if __name__ == '__main__':
    lgb_train, lgb_eval, X_test, y_test = load_data()
 +   RECEIVED_PARAMS = nni.get_next_parameter()
    PARAMS = get_default_parameters()
 +   PARAMS.update(RECEIVED_PARAMS)
    PARAMS = get_default_parameters()
    PARAMS.update(RECEIVED_PARAMS)
    # train
    run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
 ```
 ### 3.4 Write a config file and run it.
 In the config file, you could set some settings including:
 * Experiment setting: `trialConcurrency`, `maxExecDuration`, `maxTrialNum`, `trial gpuNum`, etc.
 * Platform setting: `trainingServicePlatform`, etc.
 * Path seeting: `searchSpacePath`, `trial codeDir`, etc.
 * Algorithm setting: select `tuner` algorithm, `tuner optimize_mode`, etc.
 An config.yml as follow:
 ```yaml
 authorName: default
 experimentName: example_auto-gbdt
 trialConcurrency: 1
 maxExecDuration: 10h
 maxTrialNum: 10
 #choice: local, remote, pai
 trainingServicePlatform: local
 searchSpacePath: search_space.json
 #choice: true, false
 useAnnotation: false
 tuner:
  #choice: TPE, Random, Anneal, Evolution, BatchTuner
  #SMAC (SMAC should be installed through nnictl)
  builtinTunerName: TPE
  classArgs:
    #choice: maximize, minimize
    optimize_mode: minimize
 trial:
  command: python3 main.py
  codeDir: .
  gpuNum: 0
 ```
 Run this experiment with command as follow:
 ```bash
 nnictl create --config ./config.yml
 ```
\ No newline at end of file
--- a/docs/index.rst
+++ b/docs/index.rst