Merge pull request #130 from Microsoft/master

merge master

deployment/pypi/setup.py

@@ -63,7 +63,7 @@ setuptools.setup(
         'psutil',
         'requests',
         'astor',
-        'pyhdfs',
+        'PythonWebHDFS',
         'hyperopt',
         'json_tricks',
         'numpy',

docs/en_US/Builtin_Assessors.md

@@ -4,10 +4,14 @@ NNI provides state-of-the-art tuning algorithm in our builtin-assessors and make
 
 Note: Click the **Assessor's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Assessor's installation requirements, suggested scenario and using example.
 
+Currently we support the following Assessors:
+* [Medianstop](medianstopAssessor.md)
+* [Curvefitting](curvefittingAssessor.md)
+
 |Assessor|Brief Introduction of Algorithm|
 |---|---|
-|[Medianstop](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/medianstop_assessor/README.md) [(Usage)](#MedianStop)|Medianstop is a simple early stopping rule. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S. [Reference Paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf)|
-|[Curvefitting](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/curvefitting_assessor/README.md) [(Usage)](#Curvefitting)|Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve. [Reference Paper](http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf)|
+|__Medianstop__[(Usage)](#MedianStop)|Medianstop is a simple early stopping rule. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S. [Reference Paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf)|
+|__Curvefitting__[(Usage)](#Curvefitting)|Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve. [Reference Paper](http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf)|
 
 ## Usage of Builtin Assessors
 

docs/en_US/Builtin_Tuner.md

@@ -4,18 +4,30 @@ NNI provides state-of-the-art tuning algorithm as our builtin-tuners and makes t
 
 Note: Click the **Tuner's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Tuner's installation requirements, suggested scenario and using example.
 
+Currently we support the following algorithms:
+* [TPE](hyperoptTuner.md)
+* [Random Search](hyperoptTuner.md)
+* [Anneal](hyperoptTuner.md)
+* [Naive Evolution](evolutionTuner.md)
+* [SMAC](smacTuner.md)
+* [Batch tuner](batchTuner.md)
+* [Grid Search](gridsearchTuner.md)
+* [Hyperband](hyperbandAdvisor.md)
+* [Network Morphism](networkmorphismTuner.md)
+* [Metis Tuner](metisTuner.md)
+
 |Tuner|Brief Introduction of Algorithm|
 |---|---|
-|[TPE](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/hyperopt_tuner/README.md) [(Usage)](#TPE)|The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. [Reference Paper](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf)|
-|[Random Search](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/hyperopt_tuner/README.md) [(Usage)](#Random)|In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters. [Reference Paper](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)|
-|[Anneal](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/hyperopt_tuner/README.md) [(Usage)](#Anneal)|This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.|
-|[Naive Evolution](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/evolution_tuner/README.md) [(Usage)](#Evolution)|Naive Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naive Evolution requires many trials to works, but it's very simple and easy to expand new features. [Reference paper](https://arxiv.org/pdf/1703.01041.pdf)|
-|[SMAC](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/smac_tuner/README.md) [(Usage)](#SMAC)|SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by nni is a wrapper on the SMAC3 Github repo. Notice, SMAC need to be installed by `nnictl package` command. [Reference Paper,](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) [Github Repo](https://github.com/automl/SMAC3)|
-|[Batch tuner](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/batch_tuner/README.md) [(Usage)](#Batch)|Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.|
-|[Grid Search](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/gridsearch_tuner/README.md) [(Usage)](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, qloguniform. The number q in quniform and qloguniform has special meaning (different from the spec in search space spec). It means the number of values that will be sampled evenly from the range low and high.|
-|[Hyperband](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/hyperband_advisor/README.md) [(Usage)](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of STEPs to find out promising one, then further training those promising ones to select several more promising one.[Reference Paper](https://arxiv.org/pdf/1603.06560.pdf)|
-|[Network Morphism](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/networkmorphism_tuner/README.md) [(Usage)](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train. [Reference Paper](https://arxiv.org/abs/1806.10282)|
-|[Metis Tuner](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/metis_tuner/README.md) [(Usage)](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. [Reference Paper](https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/)|
+|__TPE__ [(Usage)](#TPE)|The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. [Reference Paper](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf)|
+|__Random Search__ [(Usage)](#Random)|In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters. [Reference Paper](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)|
+|__Anneal__ [(Usage)](#Anneal)|This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.|
+|__Naive Evolution__ [(Usage)](#Evolution)|Naive Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naive Evolution requires many trials to works, but it's very simple and easy to expand new features. [Reference paper](https://arxiv.org/pdf/1703.01041.pdf)|
+|__SMAC__ [(Usage)](#SMAC)|SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by nni is a wrapper on the SMAC3 Github repo. Notice, SMAC need to be installed by `nnictl package` command. [Reference Paper,](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) [Github Repo](https://github.com/automl/SMAC3)|
+|__Batch tuner__ [(Usage)](#Batch)|Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.|
+|__Grid Search__ [(Usage)](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, qloguniform. The number q in quniform and qloguniform has special meaning (different from the spec in search space spec). It means the number of values that will be sampled evenly from the range low and high.|
+|__Hyperband__ [(Usage)](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of STEPs to find out promising one, then further training those promising ones to select several more promising one.[Reference Paper](https://arxiv.org/pdf/1603.06560.pdf)|
+|__Network Morphism__ [(Usage)](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train. [Reference Paper](https://arxiv.org/abs/1806.10282)|
+|__Metis Tuner__ [(Usage)](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. [Reference Paper](https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/)|
 
 <br>
 

docs/en_US/HowToDebug.md

 **How to Debug in NNI**
 ===
 
-*Coming soon*
+## Overview
+
+There are three parts that might have logs in NNI. They are nnimanager, dispatcher and trial. Here we will introduce them succinctly. More information please refer to [Overview](Overview.md).
+
+- **NNI controller**: NNI controller (nnictl) is the nni command-line tool that is used to manage experiments (e.g., start an experiment).
+- **nnimanager**: nnimanager is the core of NNI, whose log is important when the whole experiment fails (e.g., no webUI or training service fails)
+- **Dispatcher**: Dispatcher calls the methods of **Tuner** and **Assessor**. Logs of dispatcher are related to the tuner or assessor code.
+    - **Tuner**: Tuner is an AutoML algorithm, which generates a new configuration for the next try. A new trial will run with this configuration.
+    - **Assessor**: Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset) to tell whether this trial can be early stopped or not.
+- **Trial**: Trial code is the code you write to run your experiment, which is an individual attempt at applying a new configuration (e.g., a set of hyperparameter values, a specific nerual architecture).
+
+## Where is the log
+
+There are three kinds of log in NNI. When creating a new experiment, you can specify log level as debug by adding `--debug`. Besides, you can set more detailed log level in your configuration file by using
+`logLevel` keyword. Available logLevels are: `trace`, `debug`, `info`, `warning`, `error`, `fatal`.
+
+### NNI controller
+
+All possible errors that happen when launching an NNI experiment can be found here.
+
+You can use `nnictl log stderr` to find error information. For more options please refer to [NNICTL](NNICTLDOC.md)
+
+
+### Experiment Root Directory
+Every experiment has a root folder, which is shown on the right-top corner of webUI. Or you could assemble it by replacing the `experiment_id` with your actual experiment_id in path `~/nni/experiment/experiment_id/` in case of webUI failure. `experiment_id` could be seen when you run `nnictl create ...` to create a new experiment.
+
+> For flexibility, we also offer a `logDir` option in your configuration, which specifies the directory to store all experiments (defaults to `~/nni/experiment`). Please refer to [Configuration](ExperimentConfig.md) for more details.
+
+Under that directory, there is another directory named `log`, where `nnimanager.log` and `dispatcher.log` are placed.
+
+### Trial Root Directory
+
+Usually in webUI, you can click `+` in the left of every trial to expand it to see each trial's log path.
+
+Besides, there is another directory under experiment root directory, named `trials`, which stores all the trials.
+Every trial has a unique id as its directory name. In this directory, a file named `stderr` records trial error and another named `trial.log` records this trial's log. 
+
+## Different kinds of errors
+
+There are different kinds of errors. However, they can be divided into three categories based on their severity. So when nni fails, check each part sequentially.
+
+Generally, if webUI is started successfully, there is a `Status` in the `Overview` tab, serving as a possible indicator of what kind of error happens. Otherwise you should check manually.
+
+### **NNI** Fails
+
+This is the most serious error. When this happens, the whole experiment fails and no trial will be run. Usually this might be related to some installation problem.
+
+When this happens, you should check `nnictl`'s error output file `stderr` (i.e., nnictl log stderr) and then the `nnimanager`'s log to find if there is any error.
+
+
+### **Dispatcher** Fails
+
+Dispatcher fails. Usually, for some new users of NNI, it means that tuner fails. You could check dispatcher's log to see what happens to your dispatcher. For built-in tuner, some common errors might be invalid search space (unsupported type of search space or inconsistence between initializing args in configuration file and actual tuner's \_\_init\_\_ function args).
+
+Take the later situation as an example. If you write a customized tuner who's \_\_init\_\_ function has an argument called `optimize_mode`, which you do not provide in your configuration file, NNI will fail to run your tuner so the experiment fails. You can see errors in the webUI like:
+
+![](../img/dispatcher_error.jpg)
+
+Here we can see it is a dispatcher error. So we can check dispatcher's log, which might look like:
+
+```
+[2019-02-19 19:36:45] DEBUG (nni.main/MainThread) START
+[2019-02-19 19:36:47] ERROR (nni.main/MainThread) __init__() missing 1 required positional arguments: 'optimize_mode'
+Traceback (most recent call last):
+  File "/usr/lib/python3.7/site-packages/nni/__main__.py", line 202, in <module>
+    main()
+  File "/usr/lib/python3.7/site-packages/nni/__main__.py", line 164, in main
+    args.tuner_args)
+  File "/usr/lib/python3.7/site-packages/nni/__main__.py", line 81, in create_customized_class_instance
+    instance = class_constructor(**class_args)
+TypeError: __init__() missing 1 required positional arguments: 'optimize_mode'.
+```
+
+### **Trial** Fails
+
+In this situation, NNI can still run and create new trials. 
+
+It means your trial code (which is run by NNI) fails. This kind of error is strongly related to your trial code. Please check trial's log to fix any possible errors shown there.
+
+A common example of this would be run the mnist example without installing tensorflow. Surely there is an Import Error (that is, not installing tensorflow but trying to import it in your trial code) and thus every trial fails.
+
+![](../img/trial_error.jpg)
+
+As it shows, every trial has a log path, where you can find trial'log and stderr.
+

docs/en_US/Reference.rst

@@ -5,6 +5,7 @@ References
     :maxdepth: 3
     
     Command Line <NNICTLDOC>
+    Python API <sdk_reference>
     Annotation <AnnotationSpec>
     Configuration<ExperimentConfig>
     Search Space <SearchSpaceSpec>
\ No newline at end of file

src/sdk/pynni/nni/batch_tuner/README.md → docs/en_US/batchTuner.md

@@ -3,6 +3,6 @@ Batch Tuner on NNI
 
 ## Batch Tuner
 
-Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in [search space spec](../../../../../docs/en_US/SearchSpaceSpec.md).
+Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in [search space spec](SearchSpaceSpec.md).
 
 Suggested sceanrio: If the configurations you want to try have been decided, you can list them in searchspace file (using choice) and run them using batch tuner.
\ No newline at end of file

docs/en_US/conf.py

@@ -189,7 +189,6 @@ epub_exclude_files = ['search.html']
 github_doc_root = 'https://github.com/Microsoft/nni/tree/master/doc/'
 def setup(app):
     app.add_config_value('recommonmark_config', {
-        'url_resolver': lambda url: github_doc_root + url if url.startswith('..') else url,
-        'enable_auto_toc_tree': False,
+        'enable_auto_toc_tree': True,
     }, True)
     app.add_transform(AutoStructify)

src/sdk/pynni/nni/gridsearch_tuner/README.md → docs/en_US/gridsearchTuner.md

@@ -3,4 +3,4 @@ Grid Search on NNI
 
 ## Grid Search
 
-Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform`. **The number `q` in `quniform` and `qloguniform` has special meaning (different from the spec in [search space spec](../../../../../docs/en_US/SearchSpaceSpec.md)). It means the number of values that will be sampled evenly from the range `low` and `high`.**
+Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform`. **The number `q` in `quniform` and `qloguniform` has special meaning (different from the spec in [search space spec](SearchSpaceSpec.md)). It means the number of values that will be sampled evenly from the range `low` and `high`.**
\ No newline at end of file

src/sdk/pynni/nni/networkmorphism_tuner/README.md → docs/en_US/networkmorphismTuner.md

@@ -4,7 +4,7 @@
 
 [Autokeras](https://arxiv.org/abs/1806.10282) is a popular automl tools using Network Morphism. The basic idea of Autokeras is to use Bayesian Regression to estimate the metric of the Neural Network Architecture. Each time, it generates several child networks from father networks. Then it uses a naïve Bayesian regression estimate its metric value from history trained results of network and metric value pair. Next, it chooses the the child which has best estimated performance and adds it to the training queue. Inspired by its work and referring to its [code](https://github.com/jhfjhfj1/autokeras), we implement our Network Morphism method in our NNI platform.
 
-If you want to know about network morphism trial usage, please check [Readme.md](../../../../../examples/trials/network_morphism/README.md) of the trial to get more detail.
+If you want to know about network morphism trial usage, please check [Readme.md](https://github.com/Microsoft/nni/blob/master/examples/trials/network_morphism/README.md) of the trial to get more detail.
 
 ## 2. Usage
 

src/sdk/pynni/nni/smac_tuner/README.md → docs/en_US/smacTuner.md

@@ -5,4 +5,4 @@ SMAC Tuner on NNI
 
 [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by nni is a wrapper on [the SMAC3 github repo](https://github.com/automl/SMAC3).
 
-Note that SMAC on nni only supports a subset of the types in [search space spec](../../../../../docs/en_US/SearchSpaceSpec.md), including `choice`, `randint`, `uniform`, `loguniform`, `quniform(q=1)`.
+Note that SMAC on nni only supports a subset of the types in [search space spec](SearchSpaceSpec.md), including `choice`, `randint`, `uniform`, `loguniform`, `quniform(q=1)`.
\ No newline at end of file

docs/requirements.txt

@@ -9,5 +9,3 @@ json_tricks
 numpy
 scipy
 coverage
-git+https://github.com/QuanluZhang/ConfigSpace.git
-git+https://github.com/QuanluZhang/SMAC3.git

setup.py

@@ -63,7 +63,7 @@ setup(
         'requests',
         'scipy',
         'schema',
-        'pyhdfs'
+        'PythonWebHDFS'
     ],
 
     cmdclass={