Merge pull request #218 from microsoft/master

merge master

.gitignore

@@ -80,6 +80,7 @@ venv.bak/
 
 # VSCode
 .vscode
+.vs
 
 # In case you place source code in ~/nni/
 /experiments

LICENSE

-    MIT License
+Copyright (c) Microsoft Corporation.
 
-    Copyright (c) Microsoft Corporation. All rights reserved.
+MIT License
 
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -71,7 +71,7 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search
         <li><b>Examples</b></li>
          <ul>
            <li><a href="examples/trials/mnist-pytorch">MNIST-pytorch</li></a>
-           <li><a href="examples/trials/mnist">MNIST-tensorflow</li></a>
+           <li><a href="examples/trials/mnist-tfv1">MNIST-tensorflow</li></a>
            <li><a href="examples/trials/mnist-keras">MNIST-keras</li></a>
            <li><a href="docs/en_US/TrialExample/GbdtExample.md">Auto-gbdt</a></li>
            <li><a href="docs/en_US/TrialExample/Cifar10Examples.md">Cifar10-pytorch</li></a>
@@ -245,7 +245,7 @@ Linux and MacOS
 * Run the MNIST example.
 
 ```bash
-    nnictl create --config nni/examples/trials/mnist/config.yml
+    nnictl create --config nni/examples/trials/mnist-tfv1/config.yml
 ```
 
 Windows
@@ -253,7 +253,7 @@ Windows
 * Run the MNIST example.
 
 ```bash
-    nnictl create --config nni\examples\trials\mnist\config_windows.yml
+    nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
 ```
 
 * Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.

README_zh_CN.md

@@ -342,6 +342,7 @@ You can use these commands to get more information about the experiment
    * 在 NNI 中运行 [神经网络架构结构搜索](examples/trials/nas_cifar10/README_zh_CN.md) 
    * [NNI 中的自动特征工程](examples/trials/auto-feature-engineering/README_zh_CN.md) 
    * 使用 NNI 的 [矩阵分解超参调优](https://github.com/microsoft/recommenders/blob/master/notebooks/04_model_select_and_optimize/nni_surprise_svd.ipynb)
+   * [scikit-nni](https://github.com/ksachdeva/scikit-nni) 使用 NNI 为 scikit-learn 开发的超参搜索。
 * ### **相关文章**
    
    * [超参数优化的对比](docs/zh_CN/CommunitySharings/HpoComparision.md)
@@ -349,6 +350,7 @@ You can use these commands to get more information about the experiment
    * [并行化顺序算法：TPE](docs/zh_CN/CommunitySharings/ParallelizingTpeSearch.md)
    * [使用 NNI 为 SVD 自动调参](docs/zh_CN/CommunitySharings/RecommendersSvd.md)
    * [使用 NNI 为 SPTAG 自动调参](docs/zh_CN/CommunitySharings/SptagAutoTune.md)
+   * [使用 NNI 为 scikit-learn 查找超参](https://towardsdatascience.com/find-thy-hyper-parameters-for-scikit-learn-pipelines-using-microsoft-nni-f1015b1224c1)
    * **博客** - [AutoML 工具（Advisor，NNI 与 Google Vizier）的对比](http://gaocegege.com/Blog/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/katib-new#%E6%80%BB%E7%BB%93%E4%B8%8E%E5%88%86%E6%9E%90) 作者：[@gaocegege](https://github.com/gaocegege) - kubeflow/katib 的设计与实现的总结与分析章节
 
 ## **反馈**

deployment/deployment-pipelines.yml

-# Copyright (c) Microsoft Corporation
-# All rights reserved.
-#
-# MIT License
-#
-# Permission is hereby granted, free of charge,
-# to any person obtaining a copy of this software and associated
-# documentation files (the "Software"), to deal in the Software without restriction,
-# including without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and
-# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
-# The above copyright notice and this permission notice shall be included
-# in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
-# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
 
 jobs:
 - job: 'version_number_validation'

deployment/docker/Dockerfile

-# Copyright (c) Microsoft Corporation
-# All rights reserved.
-#
-# MIT License
-#
-# Permission is hereby granted, free of charge,
-# to any person obtaining a copy of this software and associated
-# documentation files (the "Software"), to deal in the Software without restriction,
-# including without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and
-# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
-# The above copyright notice and this permission notice shall be included
-# in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
-# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
 
 FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04
 

deployment/pypi/setup.py

-# Copyright (c) Microsoft Corporation. All rights reserved.
-#
-# MIT License
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
-# associated documentation files (the "Software"), to deal in the Software without restriction,
-# including without limitation the rights to use, copy, modify, merge, publish, distribute,
-# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all copies or
-# substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
-# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
-# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-# ==================================================================================================
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
 
 import setuptools
 import platform

docs/en_US/AdvancedFeature/MultiPhase.md

@@ -79,7 +79,7 @@ With this information, the tuner could know which trial is requesting a configur
 
 ### Tuners support multi-phase experiments:
 
-[TPE](../Tuner/HyperoptTuner.md), [Random](../Tuner/HyperoptTuner.md), [Anneal](../Tuner/HyperoptTuner.md), [Evolution](../Tuner/EvolutionTuner.md), [SMAC](../Tuner/SmacTuner.md), [NetworkMorphism](../Tuner/NetworkmorphismTuner.md), [MetisTuner](../Tuner/MetisTuner.md), [BOHB](../Tuner/BohbAdvisor.md), [Hyperband](../Tuner/HyperbandAdvisor.md), [ENAS tuner](https://github.com/countif/enas_nni/blob/master/nni/examples/tuners/enas/nni_controller_ptb.py).
+[TPE](../Tuner/HyperoptTuner.md), [Random](../Tuner/HyperoptTuner.md), [Anneal](../Tuner/HyperoptTuner.md), [Evolution](../Tuner/EvolutionTuner.md), [SMAC](../Tuner/SmacTuner.md), [NetworkMorphism](../Tuner/NetworkmorphismTuner.md), [MetisTuner](../Tuner/MetisTuner.md), [BOHB](../Tuner/BohbAdvisor.md), [Hyperband](../Tuner/HyperbandAdvisor.md).
 
 ### Training services support multi-phase experiment:
 [Local Machine](../TrainingService/LocalMode.md), [Remote Servers](../TrainingService/RemoteMachineMode.md), [OpenPAI](../TrainingService/PaiMode.md)

docs/en_US/Compressor/Quantizer.md

@@ -28,17 +28,23 @@ In [Quantization and Training of Neural Networks for Efficient Integer-Arithmeti
 ### Usage
 You can quantize your model to 8 bits with the code below before your training code.
 
-Tensorflow code
-```python
-from nni.compressors.tensorflow import QAT_Quantizer
-config_list = [{ 'q_bits': 8, 'op_types': ['default'] }]
-quantizer = QAT_Quantizer(tf.get_default_graph(), config_list)
-quantizer.compress()
-```
 PyTorch code
 ```python
 from nni.compressors.torch import QAT_Quantizer
-config_list = [{ 'q_bits': 8, 'op_types': ['default'] }]
+model = Mnist()
+
+config_list = [{
+    'quant_types': ['weight'],
+    'quant_bits': {
+        'weight': 8,
+    }, # you can just use `int` here because all `quan_types` share same bits length, see config for `ReLu6` below.
+    'op_types':['Conv2d', 'Linear']
+}, {
+    'quant_types': ['output'],
+    'quant_bits': 8,
+    'quant_start_step': 7000,
+    'op_types':['ReLU6']
+}]
 quantizer = QAT_Quantizer(model, config_list)
 quantizer.compress()
 ```
@@ -46,9 +52,17 @@ quantizer.compress()
 You can view example for more information
 
 #### User configuration for QAT Quantizer
-* **q_bits:** This is to specify the q_bits operations to be quantized to
-
-
+* **quant_types:** : list of string
+type of quantization you want to apply, currently support 'weight', 'input', 'output'
+* **quant_bits:** int or dict of {str : int}
+bits length of quantization, key is the quantization type, value is the length, eg. {'weight', 8},
+when the type is int, all quantization types share same bits length
+* **quant_start_step:** int
+disable quantization until model are run by certain number of steps, this allows the network to enter a more stable
+state where activation quantization ranges do not exclude a significant fraction of values, default value is 0
+
+### note
+batch normalization folding is currently not supported.
 ***
 
 ## DoReFa Quantizer

docs/en_US/Compressor/SlimPruner.md

@@ -34,6 +34,6 @@ We implemented one of the experiments in ['Learning Efficient Convolutional Netw
 | Model         | Error(paper/ours) | Parameters | Pruned    |
 | ------------- | ----------------- | ---------- | --------- |
 | VGGNet        | 6.34/6.40     | 20.04M   |           |
-| Pruned-VGGNet | 6.20/6.39     | 2.03M    | 88.5% |
+| Pruned-VGGNet | 6.20/6.26     | 2.03M    | 88.5% |
 
 The experiments code can be found at [examples/model_compress]( https://github.com/microsoft/nni/tree/master/examples/model_compress/)

docs/en_US/FeatureEngineering/GBDTSelector.md

+## GBDTSelector
+
+GBDTSelector is based on [LightGBM](https://github.com/microsoft/LightGBM), which is a gradient boosting framework that uses tree-based learning algorithms.
+
+When passing the data into the GBDT model, the model will construct the boosting tree. And the feature importance comes from the score in construction, which indicates how useful or valuable each feature was in the construction of the boosted decision trees within the model.
+
+We could use this method as a strong baseline in Feature Selector, especially when using the GBDT model as a classifier or regressor.
+
+For now, we support the `importance_type` is `split` and `gain`. But we will support customized `importance_type` in the future, which means the user could define how to calculate the `feature score` by themselves.
+
+### Usage
+
+First you need to install dependency:
+
+```
+pip install lightgbm
+```
+
+Then
+
+```python
+from nni.feature_engineering.gbdt_selector import GBDTSelector
+
+# load data
+...
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+# initlize a selector
+fgs = GBDTSelector()
+# fit data
+fgs.fit(X_train, y_train, ...)
+# get improtant features
+# will return the index with important feature here.
+print(fgs.get_selected_features(10))
+
+...
+```
+
+And you could reference the examples in `/examples/feature_engineering/gbdt_selector/`, too.
+
+
+**Requirement of `fit` FuncArgs**
+
+* **X** (array-like, require) - The training input samples which shape = [n_samples, n_features]
+
+* **y** (array-like, require) - The target values (class labels in classification, real numbers in regression) which shape = [n_samples].
+
+* **lgb_params** (dict, require) - The parameters for lightgbm model. The detail you could reference [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html)
+
+* **eval_ratio** (float, require) - The ratio of data size. It's used for split the eval data and train data from self.X.
+
+* **early_stopping_rounds** (int, require) - The early stopping setting in lightgbm. The detail you could reference [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html).
+
+* **importance_type** (str, require) - could be 'split' or 'gain'. The 'split' means ' result contains numbers of times the feature is used in a model' and the 'gain' means 'result contains total gains of splits which use the feature'. The detail you could reference in [here](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Booster.html#lightgbm.Booster.feature_importance).
+
+* **num_boost_round** (int, require) - number of boost round. The detail you could reference [here](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.train.html#lightgbm.train).
+
+**Requirement of `get_selected_features` FuncArgs**
+    
+* **topk** (int, require) - the topK impotance features you want to selected.
+

docs/en_US/FeatureEngineering/GradientFeatureSelector.md

+## GradientFeatureSelector
+
+The algorithm in GradinetFeatureSelector comes from ["Feature Gradients: Scalable Feature Selection via Discrete Relaxation"](https://arxiv.org/pdf/1908.10382.pdf).
+
+GradientFeatureSelector, a gradient-based search algorithm
+for feature selection. 
+
+1) This approach extends a recent result on the estimation of
+learnability in the sublinear data regime by showing that the calculation can be performed iteratively (i.e., in mini-batches) and in **linear time and space** with respect to both the number of features D and the sample size N. 
+
+2) This, along with a discrete-to-continuous relaxation of the search domain, allows for an **efficient, gradient-based** search algorithm among feature subsets for very **large datasets**.
+
+3) Crucially, this algorithm is capable of finding **higher-order correlations** between features and targets for both the N > D and N < D regimes, as opposed to approaches that do not consider such interactions and/or only consider one regime.
+
+
+### Usage
+
+```python
+from nni.feature_engineering.gradient_selector import FeatureGradientSelector
+
+# load data
+...
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+# initlize a selector
+fgs = FeatureGradientSelector(n_features=10)
+# fit data
+fgs.fit(X_train, y_train)
+# get improtant features
+# will return the index with important feature here.
+print(fgs.get_selected_features())
+
+...
+```
+
+And you could reference the examples in `/examples/feature_engineering/gradient_feature_selector/`, too.
+
+**Parameters of class FeatureGradientSelector constructor**
+
+* **order** (int, optional, default = 4) - What order of interactions to include. Higher orders may be more accurate but increase the run time. 12 is the maximum allowed order.
+
+* **penatly** (int, optional, default = 1) - Constant that multiplies the regularization term.
+
+* **n_features** (int, optional, default = None) - If None, will automatically choose number of features based on search. Otherwise, the number of top features to select.
+
+* **max_features** (int, optional, default = None) - If not None, will use the 'elbow method' to determine the number of features with max_features as the upper limit.
+
+* **learning_rate** (float, optional, default = 1e-1) - learning rate
+
+* **init** (*zero, on, off, onhigh, offhigh, or sklearn, optional, default = zero*) - How to initialize the vector of scores. 'zero' is the default.
+
+* **n_epochs** (int, optional, default = 1) - number of epochs to run
+
+* **shuffle** (bool, optional, default = True) - Shuffle "rows" prior to an epoch.
+
+* **batch_size** (int, optional, default = 1000) - Nnumber of "rows" to process at a time.
+
+* **target_batch_size** (int, optional, default = 1000) - Number of "rows" to accumulate gradients over. Useful when many rows will not fit into memory but are needed for accurate estimation.
+
+* **classification** (bool, optional, default = True) - If True, problem is classification, else regression.
+
+* **ordinal** (bool, optional, default = True) - If True, problem is ordinal classification. Requires classification to be True.
+
+* **balanced** (bool, optional, default = True) - If true, each class is weighted equally in optimization, otherwise weighted is done via support of each class. Requires classification to be True.
+
+* **prerocess** (str, optional, default = 'zscore') - 'zscore' which refers to centering and normalizing data to unit variance or 'center' which only centers the data to 0 mean.
+
+* **soft_grouping** (bool, optional, default = True) - If True, groups represent features that come from the same source. Used to encourage sparsity of groups and features within groups.
+
+* **verbose** (int, optional, default = 0) - Controls the verbosity when fitting. Set to 0 for no printing 1 or higher for printing every verbose number of gradient steps.
+
+* **device** (str, optional, default = 'cpu') - 'cpu' to run on CPU and 'cuda' to run on GPU. Runs much faster on GPU
+
+
+**Requirement of `fit` FuncArgs**
+
+* **X** (array-like, require) - The training input samples which shape = [n_samples, n_features]
+
+* **y** (array-like, require) - The target values (class labels in classification, real numbers in regression) which shape = [n_samples].
+
+* **groups** (array-like, optional, default = None) - Groups of columns that must be selected as a unit. e.g. [0, 0, 1, 2] specifies the first two columns are part of a group. Which shape is [n_features].
+
+**Requirement of `get_selected_features` FuncArgs**
+ 
+ For now, the `get_selected_features` function has no parameters.
+

docs/en_US/FeatureEngineering/Overview.md

+# FeatureEngineering
+
+We are glad to announce the alpha release for Feature Engineering toolkit on top of NNI, it's still in the experiment phase which might evolve based on user feedback. We'd like to invite you to use, feedback and even contribute.
+
+For now, we support the following feature selector:
+- [GradientFeatureSelector](./GradientFeatureSelector.md)
+- [GBDTSelector](./GBDTSelector.md)
+
+
+# How to use?
+
+```python
+from nni.feature_engineering.gradient_selector import GradientFeatureSelector
+# from nni.feature_engineering.gbdt_selector import GBDTSelector
+
+# load data
+...
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+# initlize a selector
+fgs = GradientFeatureSelector(...)
+# fit data
+fgs.fit(X_train, y_train)
+# get improtant features
+# will return the index with important feature here.
+print(fgs.get_selected_features(...))
+
+...
+```
+
+When using the built-in Selector, you first need to `import` a feature selector, and `initialize` it. You could call the function `fit` in the selector to pass the data to the selector. After that, you could use `get_seleteced_features` to get important features. The function parameters in different selectors might be different, so you need to check the docs before using it. 
+
+# How to customize?
+
+NNI provides _state-of-the-art_ feature selector algorithm in the builtin-selector. NNI also supports to build a feature selector by yourself.
+
+If you want to implement a customized feature selector, you need to:
+
+1. Inherit the base FeatureSelector class
+1. Implement _fit_ and _get_selected_features_ function
+1. Integrate with sklearn (Optional)
+
+Here is an example:
+
+**1. Inherit the base Featureselector Class**
+
+```python
+from nni.feature_engineering.feature_selector import FeatureSelector
+
+class CustomizedSelector(FeatureSelector):
+    def __init__(self, ...):
+    ...
+```
+
+**2. Implement _fit_ and _get_selected_features_ Function**
+
+```python
+from nni.tuner import Tuner
+
+from nni.feature_engineering.feature_selector import FeatureSelector
+
+class CustomizedSelector(FeatureSelector):
+    def __init__(self, ...):
+    ...
+
+    def fit(self, X, y, **kwargs):
+        """
+        Fit the training data to FeatureSelector
+
+        Parameters
+        ------------
+        X : array-like numpy matrix
+        The training input samples, which shape is [n_samples, n_features].
+        y: array-like numpy matrix
+        The target values (class labels in classification, real numbers in regression). Which shape is [n_samples].
+        """
+        self.X = X
+        self.y = y
+        ...
+    
+    def get_selected_features(self):
+        """
+        Get important feature
+
+        Returns
+        -------
+        list :
+        Return the index of the important feature.
+        """
+        ...
+        return self.selected_features_
+
+    ...
+```
+
+**3. Integrate with Sklearn**
+
+`sklearn.pipeline.Pipeline` can connect models in series, such as feature selector, normalization, and classification/regression to form a typical machine learning problem workflow. 
+The following step could help us to better integrate with sklearn, which means we could treat the customized feature selector as a mudule of the pipeline.
+
+1. Inherit the calss _sklearn.base.BaseEstimator_
+1. Implement _get_params_ and _set_params_ function in _BaseEstimator_
+1. Inherit the class _sklearn.feature_selection.base.SelectorMixin_
+1. Implement _get_support_, _transform_ and _inverse_transform_ Function in _SelectorMixin_
+
+Here is an example:
+
+**1. Inherit the BaseEstimator Class and its Function**
+
+```python
+from sklearn.base import BaseEstimator
+from nni.feature_engineering.feature_selector import FeatureSelector
+
+class CustomizedSelector(FeatureSelector, BaseEstimator):
+    def __init__(self, ...):
+    ...
+    
+    def get_params(self, ...):
+        """
+        Get parameters for this estimator.
+        """
+        params = self.__dict__
+        params = {key: val for (key, val) in params.items()
+        if not key.endswith('_')}
+        return params
+    
+    def set_params(self, **params):
+        """
+        Set the parameters of this estimator.
+        """
+        for param in params:
+        if hasattr(self, param):
+        setattr(self, param, params[param])
+        return self
+
+```
+
+**2. Inherit the SelectorMixin Class and its Function**
+```python
+from sklearn.base import BaseEstimator
+from sklearn.feature_selection.base import SelectorMixin
+
+from nni.feature_engineering.feature_selector import FeatureSelector
+
+class CustomizedSelector(FeatureSelector, BaseEstimator):
+    def __init__(self, ...):
+        ...
+    
+    def get_params(self, ...):
+        """
+        Get parameters for this estimator.
+        """
+        params = self.__dict__
+        params = {key: val for (key, val) in params.items()
+        if not key.endswith('_')}
+        return params
+        
+        def set_params(self, **params):
+        """
+        Set the parameters of this estimator.
+        """
+        for param in params:
+        if hasattr(self, param):
+        setattr(self, param, params[param])
+        return self
+
+    def get_support(self, indices=False):
+        """
+        Get a mask, or integer index, of the features selected.
+
+        Parameters
+        ----------
+        indices : bool
+        Default False. If True, the return value will be an array of integers, rather than a boolean mask.
+
+        Returns
+        -------
+        list :
+        returns support: An index that selects the retained features from a feature vector.
+        If indices are False, this is a boolean array of shape [# input features], in which an element is True iff its corresponding feature is selected for retention.
+        If indices are True, this is an integer array of shape [# output features] whose values
+        are indices into the input feature vector.
+        """
+        ...
+        return mask
+    
+
+    def transform(self, X):
+        """Reduce X to the selected features.
+
+        Parameters
+        ----------
+        X : array
+        which shape is [n_samples, n_features]
+
+        Returns
+        -------
+        X_r : array
+        which shape is [n_samples, n_selected_features]
+        The input samples with only the selected features.
+        """
+        ...
+        return X_r
+
+
+    def inverse_transform(self, X):
+        """
+        Reverse the transformation operation
+
+        Parameters
+        ----------
+        X : array
+        shape is [n_samples, n_selected_features]
+
+        Returns
+        -------
+        X_r : array
+        shape is [n_samples, n_original_features]
+        """
+        ...
+        return X_r
+```
+
+After integrating with Sklearn, we could use the feature selector as follows:
+```python
+from sklearn.linear_model import LogisticRegression
+
+# load data
+...
+X_train, y_train = ...
+
+# build a ppipeline
+pipeline = make_pipeline(XXXSelector(...), LogisticRegression())
+pipeline = make_pipeline(SelectFromModel(ExtraTreesClassifier(n_estimators=50)), LogisticRegression())
+pipeline.fit(X_train, y_train)
+
+# score
+print("Pipeline Score: ", pipeline.score(X_train, y_train))
+
+```
+
+# Benchmark
+
+`Baseline` means without any feature selection, we directly pass the data to LogisticRegression. For this benchmark, we only use 10% data from the train as test data.
+
+| Dataset | Baseline | GradientFeatureSelector | TreeBasedClassifier | #Train | #Feature | 
+| ----------- | ------ | ------ | ------- | ------- | -------- |
+| colon-cancer | 0.7547 | 0.7368 | 0.7223 | 62 | 2,000 |
+| gisette | 0.9725 | 0.89416 | 0.9792 | 6,000 | 5,000 |
+| avazu | 0.8834 | N/A | N/A | 40,428,967 | 1,000,000 |
+| rcv1 | 0.9644 | 0.7333 | 0.9615 | 20,242 | 47,236 |
+| news20.binary | 0.9208 | 0.6870 | 0.9070 | 19,996 | 1,355,191 |
+| real-sim | 0.9681 | 0.7969 | 0.9591 | 72,309 | 20,958 |
+
+The benchmark could be download in [here](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/
+)
+

docs/en_US/NAS/NasInterface.md

+# NNI NAS Programming Interface
+
+We are trying to support various NAS algorithms with unified programming interface, and it's still in experimental stage. It means the current programing interface might be updated in future.
+
+*previous [NAS annotation](../AdvancedFeature/GeneralNasInterfaces.md) interface will be deprecated soon.*
+
+## Programming interface for user model
+
+The programming interface of designing and searching a model is often demanded in two scenarios.
+
+1. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs  best. So, it needs an easy way to express the candidate layers or sub-models.
+2. When applying NAS on a neural network, it needs an unified way to express the search space of architectures, so that it doesn't need to update trial code for different searching algorithms.
+
+
+For expressing neural architecture search space in user code, we provide the following APIs (take PyTorch as example):
+
+```python
+# in PyTorch module class
+def __init__(self):
+    ...
+    # choose one ``op`` from ``ops``, for PyTorch this is a module.
+    # op_candidates: for PyTorch ``ops`` is a list of modules, for tensorflow it is a list of keras layers.
+    # key: the name of this ``LayerChoice`` instance
+    self.one_layer = nni.nas.pytorch.LayerChoice([
+        PoolBN('max', channels, 3, stride, 1, affine=False),
+        PoolBN('avg', channels, 3, stride, 1, affine=False),
+        FactorizedReduce(channels, channels, affine=False),
+        SepConv(channels, channels, 3, stride, 1, affine=False),
+        DilConv(channels, channels, 3, stride, 2, 2, affine=False)],
+        key="layer_name")
+    ...
+
+def forward(self, x):
+    ...
+    out = self.one_layer(x)
+    ...
+```
+This is for users to specify multiple candidate operations for a layer, one operation will be chosen at last. `key` is the identifier of the layer,it could be used to share choice between multiple `LayerChoice`. For example, there are two `LayerChoice` with the same candidate operations, and you want them to have the same choice (i.e., if first one chooses the `i`th op, the second one also chooses the `i`th op), give them the same key.
+
+```python
+def __init__(self):
+    ...
+    # choose ``n_selected`` from ``n_candidates`` inputs.
+    # n_candidates: the number of candidate inputs
+    # n_chosen: the number of chosen inputs
+    # key: the name of this ``InputChoice`` instance
+    self.input_switch = nni.nas.pytorch.InputChoice(
+        n_candidates=3,
+        n_chosen=1,
+        key="switch_name")
+    ...
+
+def forward(self, x):
+    ...
+    out = self.input_switch([in_tensor1, in_tensor2, in_tensor3])
+    ...
+```
+`InputChoice` is a PyTorch module, in init, it needs meta information, for example, from how many input candidates to choose how many inputs, the name of this initialized `InputChoice`. The real candidate input tensors can only be obtained in `forward` function. In `forward`, `InputChoice` instance is called with real candidate input tensors.
+
+Some [NAS trainers](#one-shot-training-mode) need to know the source layer the input tensors, thus, we add one input argument `choose_from` in `InputChoice` to indicate the source layer of each candidate input. `choose_from` is a list of string, each element is `key` of `LayerChoice` and `InputChoice` or the name of a module (refer to [the code](https://github.com/microsoft/nni/blob/master/src/sdk/pynni/nni/nas/pytorch/mutables.py) for more details).
+
+
+Besides `LayerChoice` and `InputChoice`, we also provide `MutableScope` which allows users to label a sub-network, thus, could provide more semantic information (e.g., the structure of the network) to NAS trainers. Here is an example:
+```python
+class Cell(mutables.MutableScope):
+    def __init__(self, scope_name):
+        super().__init__(scope_name)
+        self.layer1 = nni.nas.pytorch.LayerChoice(...)
+        self.layer2 = nni.nas.pytorch.LayerChoice(...)
+        self.layer3 = nni.nas.pytorch.LayerChoice(...)
+        ...
+```
+The three `LayerChoice` (`layer1`, `layer2`, `layer3`) are included in the `MutableScope` named `scope_name`. NAS trainer could get this hierarchical structure.
+
+
+## Two training modes
+
+After writing your model with search space embedded in the model using the above APIs, the next step is finding the best model from the search space. There are two training modes: [one-shot training mode](#one-shot-training-mode) and [classic distributed search](#classic-distributed-search).
+
+### One-shot training mode
+
+Similar to optimizers of deep learning models, the procedure of finding the best model from search space can be viewed as a type of optimizing process, we call it `NAS trainer`. There have been several NAS trainers, for example, `DartsTrainer` which uses SGD to train architecture weights and model weights iteratively, `ENASTrainer` which uses a controller to train the model. New and more efficient NAS trainers keep emerging in research community.
+
+NNI provides some popular NAS trainers, to use a NAS trainer, users could initialize a trainer after the model is defined:
+
+```python
+# create a DartsTrainer
+trainer = DartsTrainer(model,
+                       loss=criterion,
+                       metrics=lambda output, target: accuracy(output, target, topk=(1,)),
+                       optimizer=optim,
+                       num_epochs=args.epochs,
+                       dataset_train=dataset_train,
+                       dataset_valid=dataset_valid,)
+# finding the best model from search space
+trainer.train()
+# export the best found model
+trainer.export(file='./chosen_arch')
+```
+
+Different trainers could have different input arguments depending on their algorithms. Please refer to [each trainer's code](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch) for detailed arguments. After training, users could export the best one of the found models through `trainer.export()`. No need to start an NNI experiment through `nnictl`.
+
+The supported trainers can be found [here](./Overview.md#supported-one-shot-nas-algorithms). A very simple example using NNI NAS API can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/simple/train.py).
+
+The complete example code can be found [here]().
+
+### Classic distributed search
+
+Neural architecture search is originally executed by running each child model independently as a trial job. We also support this searching approach, and it naturally fits in NNI hyper-parameter tuning framework, where tuner generates child model for next trial and trials run in training service.
+
+For using this mode, no need to change the search space expressed with NNI NAS API (i.e., `LayerChoice`, `InputChoice`, `MutableScope`). After the model is initialized, apply the function `get_and_apply_next_architecture` on the model. One-shot NAS trainers are not used in this mode. Here is a simple example:
+```python
+class Net(nn.Module):
+    # defined model with LayerChoice and InputChoice
+    ...
+
+model = Net()
+# get the chosen architecture from tuner and apply it on model
+get_and_apply_next_architecture(model)
+# your code for training the model
+train(model)
+# test the trained model
+acc = test(model)
+# report the performance of the chosen architecture
+nni.report_final_result(acc)
+```
+
+The search space should be automatically generated and sent to tuner. As with NNI NAS API the search space is embedded in user code, users could use "[nnictl ss_gen](../Tutorial/Nnictl.md)" to generate search space file. Then, put the path of the generated search space in the field `searchSpacePath` of `config.yml`. The other fields in `config.yml` can be filled by referring [this tutorial](../Tutorial/QuickStart.md).
+
+You could use [NNI tuners](../Tuner/BuiltinTuner.md) to do the search.
+
+We support standalone mode for easy debugging, where you could directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for `LayerChoice` and `InputChoice` in this standalone mode.
+
+The complete example code can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/classic_nas/config_nas.yml).
+
+## Programming interface for NAS algorithm
+
+We also provide simple interface for users to easily implement a new NAS trainer on NNI.
+
+### Implement a new NAS trainer on NNI
+
+To implement a new NAS trainer, users basically only need to implement two classes by inheriting `BaseMutator` and `BaseTrainer` respectively.
+
+In `BaseMutator`, users need to overwrite `on_forward_layer_choice` and `on_forward_input_choice`, which are the implementation of `LayerChoice` and `InputChoice` respectively. Users could use property `mutables` to get all `LayerChoice` and `InputChoice` in the model code. Then users need to implement a new trainer, which instantiates the new mutator and implement the training logic. For details, please read [the code](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch) and the supported trainers, for example, [DartsTrainer](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch/darts).
+
+### Implement an NNI tuner for NAS
+
+NNI tuner for NAS takes the auto generated search space. The search space format of `LayerChoice` and `InputChoice` is shown below:
+```json
+{
+    "key_name": {
+        "_type": "layer_choice",
+        "_value": ["op1_repr", "op2_repr", "op3_repr"]
+    },
+    "key_name": {
+        "_type": "input_choice",
+        "_value": {
+            "candidates": ["in1_key", "in2_key", "in3_key"],
+            "n_chosen": 1
+        }
+    }
+}
+```
+
+Correspondingly, the generate architecture is in the following format:
+```json
+{
+    "key_name": {
+        "_value": "op1_repr",
+        "_idx": 0
+    },
+    "key_name": {
+        "_value": ["in2_key"],
+        "_idex": [1]
+    }
+}
+```
\ No newline at end of file

docs/en_US/NAS/Overview.md

+# Neural Architecture Search (NAS) on NNI
+
+Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging.
+
+However, it takes great efforts to implement NAS algorithms, and it is hard to reuse code base of existing algorithms in new one. To facilitate NAS innovations (e.g., design and implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexible programming interface is crucial.
+
+With this motivation, our ambition is to provide a unified architecture in NNI, to accelerate innovations on NAS, and apply state-of-art algorithms on real world problems faster.
+
+With [the unified interface](.NasInterface.md), there are two different modes for the architecture search. [The one](#supported-one-shot-nas-algorithms) is the so-called one-shot NAS, where a super-net is built based on search space, and using one shot training to generate good-performing child model. [The other](.ClassicNas.md) is the traditional searching approach, where each child model in search space runs as an independent trial, the performance result is sent to tuner and the tuner generates new child model.
+
+* [Supported One-shot NAS Algorithms](#supported-one-shot-nas-algorithms)
+* [Classic Distributed NAS with NNI experiment](.NasInterface.md#classic-distributed-search)
+* [NNI NAS Programming Interface](.NasInterface.md)
+
+## Supported One-shot NAS Algorithms
+
+NNI supports below NAS algorithms now and being adding more. User can reproduce an algorithm or use it on owned dataset. we also encourage user to implement other algorithms with [NNI API](#use-nni-api), to benefit more people.
+
+|Name|Brief Introduction of Algorithm|
+|---|---|
+| [ENAS](#enas) | Efficient Neural Architecture Search via Parameter Sharing [Reference Paper][1] |
+| [DARTS](#darts) | DARTS: Differentiable Architecture Search [Reference Paper][3] |
+| [P-DARTS](#p-darts) | Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation [Reference Paper](https://arxiv.org/abs/1904.12760)|
+
+Note, these algorithms run **standalone without nnictl**, and supports PyTorch only. Tensorflow 2.0 will be supported in future release.
+
+### Dependencies
+
+* NNI 1.2+
+* tensorboard
+* PyTorch 1.2+
+* git
+
+### ENAS
+
+[Efficient Neural Architecture Search via Parameter Sharing][1]. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. It uses parameter sharing between child models to achieve fast speed and excellent performance.
+
+#### Usage
+
+ENAS in NNI is still under development and we only support search phase for macro/micro search space on CIFAR10. Training from scratch and search space on PTB has not been finished yet.
+
+```bash
+# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+git clone https://github.com/Microsoft/nni.git
+
+# search the best architecture
+cd examples/nas/enas
+
+# search in macro search space
+python3 search.py --search-for macro
+
+# search in micro search space
+python3 search.py --search-for micro
+
+# view more options for search
+python3 search.py -h
+```
+
+### DARTS
+
+The main contribution of [DARTS: Differentiable Architecture Search][3] on algorithm is to introduce a novel algorithm for differentiable network architecture search on bilevel optimization.
+
+#### Usage
+
+```bash
+# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+git clone https://github.com/Microsoft/nni.git
+
+# search the best architecture
+cd examples/nas/darts
+python3 search.py
+
+# train the best architecture
+python3 retrain.py --arc-checkpoint ./checkpoints/epoch_49.json
+```
+
+### P-DARTS
+
+[Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://arxiv.org/abs/1904.12760) bases on [DARTS](#DARTS). It's contribution on algorithm is to introduce an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure.
+
+#### Usage
+
+```bash
+# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+git clone https://github.com/Microsoft/nni.git
+
+# search the best architecture
+cd examples/nas/pdarts
+python3 search.py
+
+# train the best architecture, it's the same progress as darts.
+cd ../darts
+python3 retrain.py --arc-checkpoint ../pdarts/checkpoints/epoch_2.json
+```
+
+## Use NNI API
+
+NOTE, we are trying to support various NAS algorithms with unified programming interface, and it's in very experimental stage. It means the current programing interface may be updated in future.
+
+*previous [NAS annotation](../AdvancedFeature/GeneralNasInterfaces.md) interface will be deprecated soon.*
+
+### Programming interface
+
+The programming interface of designing and searching a model is often demanded in two scenarios.
+
+1. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs  best. So, it needs an easy way to express the candidate layers or sub-models.
+2. When applying NAS on a neural network, it needs an unified way to express the search space of architectures, so that it doesn't need to update trial code for different searching algorithms.
+
+NNI proposed API is [here](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch). And [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts) is an example of NAS implementation, which bases on NNI proposed interface.
+
+[1]: https://arxiv.org/abs/1802.03268
+[2]: https://arxiv.org/abs/1707.07012
+[3]: https://arxiv.org/abs/1806.09055
+[4]: https://arxiv.org/abs/1806.10282
+[5]: https://arxiv.org/abs/1703.01041

docs/en_US/TrainingService/LocalMode.md

@@ -96,7 +96,7 @@ This command will be filled in the YAML configure file below. Please refer to [h
 
 **Prepare configure file**: Since you have already known which trial code you are going to run and which tuner you are going to use, it is time to prepare the YAML configure file. NNI provides a demo configure file for each trial example, `cat ~/nni/examples/trials/mnist-annotation/config.yml` to see it. Its content is basically shown below:
 
-```
+```yaml
 authorName: your_name
 experimentName: auto_mnist
 
@@ -112,6 +112,9 @@ maxTrialNum: 100
 # choice: local, remote
 trainingServicePlatform: local
 
+# search space file
+searchSpacePath: search_space.json
+
 # choice: true, false
 useAnnotation: true
 tuner:

docs/en_US/TrainingService/PaiMode.md

@@ -19,6 +19,8 @@ maxExecDuration: 3h
 maxTrialNum: 100
 # choice: local, remote, pai
 trainingServicePlatform: pai
+# search space file
+searchSpacePath: search_space.json
 # choice: true, false
 useAnnotation: true
 tuner:

docs/en_US/TrainingService/RemoteMachineMode.md

@@ -28,6 +28,8 @@ maxExecDuration: 1h
 maxTrialNum: 10
 #choice: local, remote, pai
 trainingServicePlatform: remote
+# search space file
+searchSpacePath: search_space.json
 #choice: true, false
 useAnnotation: true
 tuner:

docs/en_US/TrialExample/MnistExamples.md

@@ -2,7 +2,8 @@
 
 CNN MNIST classifier for deep learning is similar to `hello world` for programming languages. Thus, we use MNIST as example to introduce different features of NNI. The examples are listed below:
 
- - [MNIST with NNI API](#mnist)
+ - [MNIST with NNI API (TensorFlow v1.x)](#mnist-tfv1)
+ - [MNIST with NNI API (TensorFlow v2.x)](#mnist-tfv2)
  - [MNIST with NNI annotation](#mnist-annotation)
  - [MNIST in keras](#mnist-keras)
  - [MNIST -- tuning with batch tuner](#mnist-batch)
@@ -11,12 +12,19 @@ CNN MNIST classifier for deep learning is similar to `hello world` for programmi
  - [distributed MNIST (tensorflow) using kubeflow](#mnist-kubeflow-tf)
  - [distributed MNIST (pytorch) using kubeflow](#mnist-kubeflow-pytorch)
 
-<a name="mnist"></a>
-**MNIST with NNI API**
+<a name="mnist-tfv1"></a>
+**MNIST with NNI API (TensorFlow v1.x)**
 
 This is a simple network which has two convolutional layers, two pooling layers and a fully connected layer. We tune hyperparameters, such as dropout rate, convolution size, hidden size, etc. It can be tuned with most NNI built-in tuners, such as TPE, SMAC, Random. We also provide an exmaple YAML file which enables assessor.
 
-`code directory: examples/trials/mnist/`
+`code directory: examples/trials/mnist-tfv1/`
+
+<a name="mnist-tfv2"></a>
+**MNIST with NNI API (TensorFlow v2.x)**
+
+Same network to the example above, but written in TensorFlow v2.x Keras API.
+
+`code directory: examples/trials/mnist-tfv2/`
 
 <a name="mnist-annotation"></a>
 **MNIST with NNI annotation**

docs/en_US/Tutorial/Nnictl.md

@@ -22,6 +22,7 @@ nnictl support commands:
 * [nnictl webui](#webui)
 * [nnictl tensorboard](#tensorboard)
 * [nnictl package](#package)
+* [nnictl ss_gen](#ss_gen)
 * [nnictl --version](#version)
 
 ### Manage an experiment
@@ -733,6 +734,37 @@ Debug mode will disable version check function in Trialkeeper.
     nnictl package show
     ```
 
+<a name="ss_gen"></a>
+
+![](https://placehold.it/15/1589F0/000000?text=+) `Generate search space`
+
+* __nnictl ss_gen__
+  * Description
+
+    Generate search space from user trial code which uses NNI NAS APIs.
+
+  * Usage
+
+    ```bash
+    nnictl ss_gen [OPTIONS]
+    ```
+
+  * Options
+
+  |Name, shorthand|Required|Default|Description|
+  |------|------|------ |------|
+  |--trial_command|  True| |The command of the trial code|
+  |--trial_dir|  False| ./ |The directory of the trial code|
+  |--file|  False| nni_auto_gen_search_space.json |The file for storing generated search space|
+
+  * Example
+
+    > Generate a search space
+
+    ```bash
+    nnictl ss_gen --trial_command="python3 mnist.py" --trial_dir=./ --file=ss.json
+    ```
+
 <a name="version"></a>
 
 ![](https://placehold.it/15/1589F0/000000?text=+) `Check NNI version`