Merge pull request #121 from Microsoft/master

merge master

docs/gbdt_example.md

+# GBDT in nni
+Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion as other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. 
+
+Gradient boosting decision tree has many popular implementations, such as [lightgbm](https://github.com/Microsoft/LightGBM), [xgboost](https://github.com/dmlc/xgboost), and [catboost](https://github.com/catboost/catboost), etc. GBDT is a great tool for solving the problem of traditional machine learning problem. Since GBDT is a robust algorithm, it could use in many domains. The better hyper-parameters for GBDT, the better performance you could achieve.
+
+NNI is a great platform for tuning hyper-parameters, you could try various builtin search algorithm in nni and run multiple trials concurrently. 
+
+## 1. Search Space in GBDT
+There are many hyper-parameters in GBDT, but what kind of parameters will affect the performance or speed? Based on some practical experience, some suggestion here(Take lightgbm as example):
+
+> * For better accuracy
+* `learning_rate`. The range of `learning rate` could be [0.001, 0.9].
+
+* `num_leaves`. `num_leaves` is related to `max_depth`, you don't have to tune both of them.
+
+* `bagging_freq`. `bagging_freq` could be [1, 2, 4, 8, 10]
+
+* `num_iterations`. May larger if underfitting.
+
+> * For speed up
+* `bagging_fraction`. The range of `bagging_fraction` could be [0.7, 1.0].
+
+* `feature_fraction`. The range of `feature_fraction` could be [0.6, 1.0].
+
+* `max_bin`.
+
+> * To avoid overfitting
+* `min_data_in_leaf`. This depends on your dataset.
+
+* `min_sum_hessian_in_leaf`. This depend on your dataset.
+
+* `lambda_l1` and `lambda_l2`.
+
+* `min_gain_to_split`.
+
+* `num_leaves`.
+
+Reference link:
+[lightgbm](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) and [autoxgoboost](https://github.com/ja-thomas/autoxgboost/blob/master/poster_2018.pdf)
+
+## 2. Task description
+Now we come back to our example "auto-gbdt" which run in lightgbm and nni. The data including [train data](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/data/regression.train) and [test data](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/data/regression.train).
+Given the features and label in train data, we train a GBDT regression model and use it to predict.
+
+## 3. How to run in nni
+
+### 3.1 Prepare your trial code
+
+You need to prepare a basic code as following:
+
+```python
+...
+
+def get_default_parameters():
+    ...
+    return params
+
+
+def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
+    '''
+    Load or create dataset
+    '''
+    ...
+
+    return lgb_train, lgb_eval, X_test, y_test
+
+def run(lgb_train, lgb_eval, params, X_test, y_test):
+    # train
+    gbm = lgb.train(params,
+                    lgb_train,
+                    num_boost_round=20,
+                    valid_sets=lgb_eval,
+                    early_stopping_rounds=5)
+    # predict
+    y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
+
+    # eval
+    rmse = mean_squared_error(y_test, y_pred) ** 0.5
+    print('The rmse of prediction is:', rmse)
+
+if __name__ == '__main__':
+    lgb_train, lgb_eval, X_test, y_test = load_data()
+
+    PARAMS = get_default_parameters()
+    # train
+    run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
+```
+
+### 3.2 Prepare your search space.
+If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `bagging_freq`, you could write a [search_space.json](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/search_space.json) as follow:
+
+```json
+{
+    "num_leaves":{"_type":"choice","_value":[31, 28, 24, 20]},
+    "learning_rate":{"_type":"choice","_value":[0.01, 0.05, 0.1, 0.2]},
+    "bagging_fraction":{"_type":"uniform","_value":[0.7, 1.0]},
+    "bagging_freq":{"_type":"choice","_value":[1, 2, 4, 8, 10]}
+}
+```
+
+More support variable type you could reference [here](https://github.com/Microsoft/nni/blob/master/docs/SearchSpaceSpec.md).
+
+### 3.3 Add SDK of nni into your code.
+
+```diff
++import nni
+...
+
+def get_default_parameters():
+    ...
+    return params
+
+
+def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
+    '''
+    Load or create dataset
+    '''
+    ...
+
+    return lgb_train, lgb_eval, X_test, y_test
+
+def run(lgb_train, lgb_eval, params, X_test, y_test):
+    # train
+    gbm = lgb.train(params,
+                    lgb_train,
+                    num_boost_round=20,
+                    valid_sets=lgb_eval,
+                    early_stopping_rounds=5)
+    # predict
+    y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
+
+    # eval
+    rmse = mean_squared_error(y_test, y_pred) ** 0.5
+    print('The rmse of prediction is:', rmse)
++   nni.report_final_result(rmse)
+
+if __name__ == '__main__':
+    lgb_train, lgb_eval, X_test, y_test = load_data()
++   RECEIVED_PARAMS = nni.get_next_parameter()
+    PARAMS = get_default_parameters()
++   PARAMS.update(RECEIVED_PARAMS)
+    PARAMS = get_default_parameters()
+    PARAMS.update(RECEIVED_PARAMS)
+
+    # train
+    run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
+```
+
+### 3.4 Write a config file and run it.
+
+In the config file, you could set some settings including:
+
+* Experiment setting: `trialConcurrency`, `maxExecDuration`, `maxTrialNum`, `trial gpuNum`, etc.
+* Platform setting: `trainingServicePlatform`, etc.
+* Path seeting: `searchSpacePath`, `trial codeDir`, etc.
+* Algorithm setting: select `tuner` algorithm, `tuner optimize_mode`, etc.
+
+An config.yml as follow:
+
+```yaml
+authorName: default
+experimentName: example_auto-gbdt
+trialConcurrency: 1
+maxExecDuration: 10h
+maxTrialNum: 10
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner
+  #SMAC (SMAC should be installed through nnictl)
+  builtinTunerName: TPE
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: minimize
+trial:
+  command: python3 main.py
+  codeDir: .
+  gpuNum: 0
+```
+
+Run this experiment with command as follow:
+
+```bash
+nnictl create --config ./config.yml
+```
\ No newline at end of file

docs/index.rst

+#########################################
+Neural Network Intelligence Documentation
+#########################################
+
+********
+Contents
+********
+
+..  toctree::
+    :caption: Table of Contents
+    :maxdepth: 2
+    :titlesonly:
+
+    Overview
+    QuickStart<QuickStart>
+    Tutorials
+    Examples
+    Reference
+    FAQ
+    Contribution
+    Changelog<RELEASE>
\ No newline at end of file

docs/mnist_examples.md

+# MNIST examples
+
+CNN MNIST classifier for deep learning is similar to `hello world` for programming languages. Thus, we use MNIST as example to introduce different features of NNI. The examples are listed below:
+
+ - [MNIST with NNI API](#mnist)
+ - [MNIST with NNI annotation](#mnist-annotation)
+ - [MNIST in keras](#mnist-keras)
+ - [MNIST -- tuning with batch tuner](#mnist-batch)
+ - [MNIST -- tuning with hyperband](#mnist-hyperband)
+ - [MNIST -- tuning within a nested search space](#mnist-nested)
+ - [distributed MNIST (tensorflow) using kubeflow](#mnist-kubeflow-tf)
+ - [distributed MNIST (pytorch) using kubeflow](#mnist-kubeflow-pytorch)
+
+<a name="mnist"></a>
+**MNIST with NNI API**
+
+This is a simple network which has two convolutional layers, two pooling layers and a fully connected layer. We tune hyperparameters, such as dropout rate, convolution size, hidden size, etc. It can be tuned with most NNI built-in tuners, such as TPE, SMAC, Random. We also provide an exmaple YAML file which enables assessor.
+
+`code directory: examples/trials/mnist/`
+
+<a name="mnist-annotation"></a>
+**MNIST with NNI annotation**
+
+This example is similar to the example above, the only difference is that this example uses NNI annotation to specify search space and report results, while the example above uses NNI apis to receive configuration and report results.
+
+`code directory: examples/trials/mnist-annotation/`
+
+<a name="mnist-keras"></a>
+**MNIST in keras**
+
+This example is implemented in keras. It is also a network for MNIST dataset, with two convolution layers, one pooling layer, and two fully connected layers.
+
+`code directory: examples/trials/mnist-keras/`
+
+<a name="mnist-batch"></a>
+**MNIST -- tuning with batch tuner**
+
+This example is to show how to use batch tuner. Users simply list all the configurations they want to try in the search space file. NNI will try all of them.
+
+`code directory: examples/trials/mnist-batch-tune-keras/`
+
+<a name="mnist-hyperband"></a>
+**MNIST -- tuning with hyperband**
+
+This example is to show how to use hyperband to tune the model. There is one more key `STEPS` in the received configuration for trials to control how long it can run (e.g., number of iterations).
+
+`code directory: examples/trials/mnist-hyperband/`
+
+<a name="mnist-nested"></a>
+**MNIST -- tuning within a nested search space**
+
+This example is to show that NNI also support nested search space. The search space file is an example of how to define nested search space.
+
+`code directory: examples/trials/mnist-cascading-search-space/`
+
+<a name="mnist-kubeflow-tf"></a>
+**distributed MNIST (tensorflow) using kubeflow**
+
+This example is to show how to run distributed training on kubeflow through NNI. Users can simply provide distributed training code and a configure file which specifies the kubeflow mode. For example, what is the command to run ps and what is the command to run worker, and how many resources they consume. This example is implemented in tensorflow, thus, uses kubeflow tensorflow operator.
+
+`code directory: examples/trials/mnist-distributed/`
+
+<a name="mnist-kubeflow-pytorch"></a>
+**distributed MNIST (pytorch) using kubeflow**
+
+Similar to the previous example, the difference is that this example is implemented in pytorch, thus, it uses kubeflow pytorch operator.
+
+`code directory: examples/trials/mnist-distributed-pytorch/`
\ No newline at end of file

docs/multiPhase.md

 ## Create multi-phase experiment
 
 Typically each trial job gets single set of configuration (e.g. hyper parameters) from tuner and do some kind of experiment, let's say train a model with that hyper parameter and reports its result to tuner. Sometimes you may want to train multiple models within one trial job to share information between models or saving system resource by creating less trial jobs, for example:
+
 1. Train multiple models sequentially in one trial job, so that later models can leverage the weights or other information of prior models and may use different hyper parameters.
 2. Train large amount of models on limited system resource, combine multiple models together to save system resource to create large amount of trial jobs.
 3. Any other scenario that you would like to train multiple models with different hyper parameters in one trial job, be aware that if you allocate multiple GPUs to a trial job and you train multiple models concurrently within on trial job, you need to allocate GPU resource properly by your trial code.
@@ -13,25 +14,24 @@ To use multi-phase experiment, please follow below steps:
 
 1. Implement nni.multi_phase.MultiPhaseTuner. For example, this [ENAS tuner](https://github.com/countif/enas_nni/blob/master/nni/examples/tuners/enas/nni_controller_ptb.py) is a multi-phase Tuner which implements nni.multi_phase.MultiPhaseTuner. While implementing your MultiPhaseTuner, you may want to use the trial_job_id parameter of generate_parameters method to generate hyper parameters for each trial job.
 
-2. Set ```multiPhase``` field to ```true```, and configure your tuner implemented in step 1 as customized tuner in configuration file, for example:
+1. Set `multiPhase` field to `true`, and configure your tuner implemented in step 1 as customized tuner in configuration file, for example:
 
-```yml
-...
-multiPhase: true
-tuner:
+    ```yaml
+    ...
+    multiPhase: true
+    tuner:
       codeDir: tuners/enas
       classFileName: nni_controller_ptb.py
       className: ENASTuner
       classArgs:
         say_hello: "hello"
-...
-```
-
+    ...
+    ```
 
-3. Invoke nni.get_next_parameter() API for multiple times as needed in a trial, for example:
+1. Invoke nni.get_next_parameter() API for multiple times as needed in a trial, for example:
 
-```python
-for i in range(5):
+    ```python
+    for i in range(5):
         # get parameter from tuner
         tuner_param = nni.get_next_parameter()
 
@@ -40,4 +40,4 @@ for i in range(5):
         # report final result somewhere for the parameter retrieved above
         nni.report_final_result()
         # ...
-```
+    ```

docs/requirements.txt

+sphinx==1.8.3
+sphinx-argparse==0.2.5
+sphinx-markdown-tables==0.0.9
+sphinx-rtd-theme==0.4.2
+sphinxcontrib-websupport==1.1.0
+recommonmark==0.5.0
+nni==0.5
\ No newline at end of file

docs/sdk_reference.rst

+###########################
+Python API Reference
+###########################
+
+Trial
+------------------------
+..  autofunction:: nni.get_next_parameter
+..  autofunction:: nni.get_current_parameter
+..  autofunction:: nni.report_intermediate_result
+..  autofunction:: nni.report_final_result
+..  autofunction:: nni.get_sequence_id
+
+
+Tuner
+------------------------
+..  autoclass:: nni.tuner.Tuner
+    :members:
+
+..  autoclass:: nni.hyperopt_tuner.hyperopt_tuner.HyperoptTuner
+    :members:
+
+..  autoclass:: nni.evolution_tuner.evolution_tuner.EvolutionTuner
+    :members:
+
+..  autoclass:: nni.gridsearch_tuner.gridsearch_tuner.GridSearchTuner
+    :members:
+
+..  autoclass:: nni.smac_tuner.smac_tuner.SMACTuner
+    :members:
+
+Assessor
+------------------------
+..  autoclass:: nni.assessor.Assessor
+    :members:
+
+..  autoclass:: nni.curvefitting_assessor.curvefitting_assessor.CurvefittingAssessor
+    :members:
+
+..  autoclass:: nni.medianstop_assessor.medianstop_assessor.MedianstopAssessor
+    :members:
+
+
+Advisor
+------------------------
+..  autoclass:: nni.hyperband_advisor.hyperband_advisor.Hyperband
\ No newline at end of file

docs/sklearn_examples.md

+# Scikit-learn in NNI
+
+[Scikit-learn](https://github.com/scikit-learn/scikit-learn) is a pupular meachine learning tool for data mining and data analysis. It supports many kinds of meachine learning models like LinearRegression, LogisticRegression, DecisionTree, SVM etc. How to make the use of scikit-learn more efficiency is a valuable topic.  
+NNI supports many kinds of tuning algorithms to search the best models and/or hyper-parameters for scikit-learn, and support many kinds of environments like local machine, remote servers and cloud.
+
+## 1. How to run the example
+
+To start using NNI, you should install the nni package, and use the command line tool `nnictl` to start an experiment. For more information about installation and preparing for the environment,  please [refer](QuickStart.md).
+After you installed NNI, you could enter the corresponding folder and start the experiment using following commands:
+
+```bash
+nnictl create --config ./config.yml
+```
+
+## 2. Description of the example
+
+### 2.1 classification
+
+This example uses the dataset of digits, which is made up of 1797 8x8 images, and each image is a hand-written digit, the goal is to classify these images into 10 classes.  
+In this example, we use SVC as the model, and choose some parameters of this model, including `"C", "keral", "degree", "gamma" and "coef0"`. For more information of these parameters, please [refer](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).
+
+### 2.2 regression
+
+This example uses the Boston Housing Dataset, this dataset consists of price of houses in various places in Boston and the information such as Crime (CRIM), areas of non-retail business in the town (INDUS), the age of people who own the house (AGE) etc to predict the house price of boston.
+In this example, we tune different kinds of regression models including `"LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"` and some parameters like `"svr_kernel", "knr_weights"`. You could get more details about these models from [here](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning).
+
+## 3. How to write sklearn code using nni
+
+It is easy to use nni in your sklearn code, there are only a few steps.
+
+* __step 1__
+
+  Prepare a search_space.json to storage your choose spaces.
+  For example, if you want to choose different models, you may try:
+
+  ```json
+  {
+    "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]}
+  }
+  ```
+
+  If you want to choose different models and parameters, you could put them together in a search_space.json file.
+
+  ```json
+  {
+    "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]},
+    "svr_kernel": {"_type":"choice","_value":["linear", "poly", "rbf"]},
+    "knr_weights": {"_type":"choice","_value":["uniform", "distance"]}
+  }
+  ```
+
+  Then you could read these values as a dict from your python code, please get into the step 2.
+* __step 2__  
+  At the beginning of your python code, you should `import nni` to insure the packages works normally.
+  First, you should use `nni.get_next_parameter()` function to get your parameters given by nni. Then you could use these parameters to update your code.
+  For example, if you define your search_space.json like following format:
+
+  ```json
+  {
+    "C": {"_type":"uniform","_value":[0.1, 1]},
+    "keral": {"_type":"choice","_value":["linear", "rbf", "poly", "sigmoid"]},
+    "degree": {"_type":"choice","_value":[1, 2, 3, 4]},
+    "gamma": {"_type":"uniform","_value":[0.01, 0.1]},
+    "coef0 ": {"_type":"uniform","_value":[0.01, 0.1]}
+  }
+  ```
+
+  You may get a parameter dict like this:
+
+  ```python
+  params = {
+        'C': 1.0,
+        'keral': 'linear',
+        'degree': 3,
+        'gamma': 0.01,
+        'coef0': 0.01
+  }
+  ```
+
+  Then you could use these variables to write your scikit-learn code.
+* __step 3__  
+  After you finished your training, you could get your own score of the model, like your percision, recall or MSE etc. NNI needs your score to tuner algorithms and generate next group of parameters, please report the score back to NNI and start next trial job.  
+  You just need to use `nni.report_final_result(score)` to communitate with NNI after you process your scikit-learn code. Or if you have multiple scores in the steps of training, you could also report them back to NNI using `nni.report_intemediate_result(score)`. Note, you may not report intemediate result of your job, but you must report back your final result.

docs/training_services.rst

+Introduction to NNI Training Services
+=====================================
+
+..  toctree::
+    Local<tutorial_1_CR_exp_local_api>
+    Remote<RemoteMachineMode>
+    OpenPAI<PAIMode>
+    Kubeflow<KubeflowMode>
+    FrameworkController<FrameworkControllerMode>
\ No newline at end of file

docs/tuners.rst

+#################
+Tuners
+#################
+
+NNI provides an easy way to adopt an approach to set up parameter tuning algorithms, we call them **Tuner**.
+
+Tuner receives metrics from `Trial` to evaluate the performance of a specific parameters/architecture configures. And tuner sends next hyper-parameter or architecture configure to Trial.
+
+In NNI, we support two approaches to set the tuner: first is directly use builtin tuner provided by nni sdk, second is customize a tuner file by yourself. We also have Advisor that combines the functinality of Tuner & Assessor.
+
+For details, please refer to the following tutorials:
+
+..  toctree::
+    Builtin Tuners<Builtin_Tuner>
+    Customized Tuners<Customize_Tuner>
+    Customized Advisor<Customize_Advisor>
\ No newline at end of file

docs/tutorial_1_CR_exp_local_api.md

@@ -73,7 +73,7 @@ To run an experiment in NNI, you only needed:
 
 * Provide a runnable trial
 * Provide or choose a tuner
-* Provide a yaml experiment configure file
+* Provide a YAML experiment configure file
 * (optional) Provide or choose an assessor
 
 **Prepare trial**: 
@@ -83,18 +83,18 @@ Let's use a simple trial example, e.g. mnist, provided by NNI. After you install
 
       python ~/nni/examples/trials/mnist-annotation/mnist.py
 
-This command will be filled in the yaml configure file below. Please refer to [here](./howto_1_WriteTrial.md) for how to write your own trial.
+This command will be filled in the YAML configure file below. Please refer to [here](Trials.md) for how to write your own trial.
 
-**Prepare tuner**: NNI supports several popular automl algorithms, including Random Search, Tree of Parzen Estimators (TPE), Evolution algorithm etc. Users can write their own tuner (refer to [here](./howto_2_CustomizedTuner.md)), but for simplicity, here we choose a tuner provided by NNI as below:
+**Prepare tuner**: NNI supports several popular automl algorithms, including Random Search, Tree of Parzen Estimators (TPE), Evolution algorithm etc. Users can write their own tuner (refer to [here](Customize_Tuner.md)), but for simplicity, here we choose a tuner provided by NNI as below:
 
       tuner:
         builtinTunerName: TPE
         classArgs:
           optimize_mode: maximize
 
-*builtinTunerName* is used to specify a tuner in NNI, *classArgs* are the arguments pass to the tuner (the spec of builtin tuners can be found [here]()), *optimization_mode* is to indicate whether you want to maximize or minimize your trial's result.
+*builtinTunerName* is used to specify a tuner in NNI, *classArgs* are the arguments pass to the tuner (the spec of builtin tuners can be found [here](Builtin_Tuner.md)), *optimization_mode* is to indicate whether you want to maximize or minimize your trial's result.
 
-**Prepare configure file**: Since you have already known which trial code you are going to run and which tuner you are going to use, it is time to prepare the yaml configure file. NNI provides a demo configure file for each trial example, `cat ~/nni/examples/trials/mnist-annotation/config.yml` to see it. Its content is basically shown below:
+**Prepare configure file**: Since you have already known which trial code you are going to run and which tuner you are going to use, it is time to prepare the YAML configure file. NNI provides a demo configure file for each trial example, `cat ~/nni/examples/trials/mnist-annotation/config.yml` to see it. Its content is basically shown below:
 
 ```
 authorName: your_name
@@ -124,7 +124,7 @@ trial:
   gpuNum: 0
 ``` 
 
-Here *useAnnotation* is true because this trial example uses our python annotation (refer to [here](../tools/annotation/README.md) for details). For trial, we should provide *trialCommand* which is the command to run the trial, provide *trialCodeDir* where the trial code is. The command will be executed in this directory. We should also provide how many GPUs a trial requires.
+Here *useAnnotation* is true because this trial example uses our python annotation (refer to [here](AnnotationSpec.md) for details). For trial, we should provide *trialCommand* which is the command to run the trial, provide *trialCodeDir* where the trial code is. The command will be executed in this directory. We should also provide how many GPUs a trial requires.
 
 With all these steps done, we can run the experiment with the following command: