Merge pull request #2020 from microsoft/dev-refactor-doc

merge back to master: update doc index

docs/en_US/Compressor/Overview.md

 # Model Compression with NNI
 As larger neural networks with more layers and nodes are considered, reducing their storage and computational cost becomes critical, especially for some real-time applications. Model compression can be used to address this problem. 
 
-We are glad to announce the alpha release for model compression toolkit on top of NNI, it's still in the experiment phase which might evolve based on usage feedback. We'd like to invite you to use, feedback and even contribute.
+We are glad to introduce model compression toolkit on top of NNI, it's still in the experiment phase which might evolve based on usage feedback. We'd like to invite you to use, feedback and even contribute.
 
 NNI provides an easy-to-use toolkit to help user design and use compression algorithms. It currently supports PyTorch with unified interface. For users to compress their models, they only need to add several lines in their code. There are some popular model compression algorithms built-in in NNI. Users could further use NNI's auto tuning power to find the best compressed model, which is detailed in [Auto Model Compression](./AutoCompression.md). On the other hand, users could easily customize their new compression algorithms using NNI's interface, refer to the tutorial [here](#customize-new-compression-algorithms).
 
@@ -335,7 +335,7 @@ class YourQuantizer(Quantizer):
 If you do not customize `QuantGrad`, the default backward is Straight-Through Estimator. 
 _Coming Soon_ ...
 
-## **Reference and Feedback**
+## Reference and Feedback
 * To [report a bug](https://github.com/microsoft/nni/issues/new?template=bug-report.md) for this feature in GitHub;
 * To [file a feature or improvement request](https://github.com/microsoft/nni/issues/new?template=enhancement.md) for this feature in GitHub;
 * To know more about [Feature Engineering with NNI](https://github.com/microsoft/nni/blob/master/docs/en_US/FeatureEngineering/Overview.md);

docs/en_US/FeatureEngineering/Overview.md

@@ -7,7 +7,7 @@ For now, we support the following feature selector:
 - [GBDTSelector](./GBDTSelector.md)
 
 
-# How to use?
+## How to use?
 
 ```python
 from nni.feature_engineering.gradient_selector import GradientFeatureSelector
@@ -30,7 +30,7 @@ print(fgs.get_selected_features(...))
 
 When using the built-in Selector, you first need to `import` a feature selector, and `initialize` it. You could call the function `fit` in the selector to pass the data to the selector. After that, you could use `get_seleteced_features` to get important features. The function parameters in different selectors might be different, so you need to check the docs before using it. 
 
-# How to customize?
+## How to customize?
 
 NNI provides _state-of-the-art_ feature selector algorithm in the builtin-selector. NNI also supports to build a feature selector by yourself.
 
@@ -239,7 +239,7 @@ print("Pipeline Score: ", pipeline.score(X_train, y_train))
 
 ```
 
-# Benchmark
+## Benchmark
 
 `Baseline` means without any feature selection, we directly pass the data to LogisticRegression. For this benchmark, we only use 10% data from the train as test data. For the GradientFeatureSelector, we only take the top20 features. The metric is the mean accuracy on the given test data and labels.
 
@@ -257,7 +257,7 @@ The dataset of benchmark could be download in [here](https://www.csie.ntu.edu.tw
 
 The code could be refenrence `/examples/feature_engineering/gradient_feature_selector/benchmark_test.py`.
 
-## **Reference and Feedback**
+## Reference and Feedback
 * To [report a bug](https://github.com/microsoft/nni/issues/new?template=bug-report.md) for this feature in GitHub;
 * To [file a feature or improvement request](https://github.com/microsoft/nni/issues/new?template=enhancement.md) for this feature in GitHub;
 * To know more about [Neural Architecture Search with NNI](https://github.com/microsoft/nni/blob/master/docs/en_US/NAS/Overview.md);

docs/en_US/NAS/CDARTS.md

@@ -46,16 +46,12 @@ bash run_retrain_cifar.sh
 ..  autoclass:: nni.nas.pytorch.cdarts.CdartsTrainer
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.cdarts.RegularizedDartsMutator
     :members:
 
 ..  autoclass:: nni.nas.pytorch.cdarts.DartsDiscreteMutator
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.cdarts.RegularizedMutatorParallel
     :members:
 ```

docs/en_US/NAS/DARTS.md

@@ -43,8 +43,10 @@ python3 retrain.py --arc-checkpoint ./checkpoints/epoch_49.json
 ..  autoclass:: nni.nas.pytorch.darts.DartsTrainer
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.darts.DartsMutator
     :members:
 ```
+
+## Limitations
+
+* DARTS doesn't support DataParallel and needs to be customized in order to support DistributedDataParallel.

docs/en_US/NAS/ENAS.md

@@ -37,10 +37,6 @@ python3 search.py -h
 ..  autoclass:: nni.nas.pytorch.enas.EnasTrainer
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.enas.EnasMutator
     :members:
-
-    .. automethod:: __init__
 ```

docs/en_US/NAS/NasGuide.md

+# Guide: Using NAS on NNI
+
+```eval_rst
+.. contents::
+
+.. Note:: The APIs are in an experimental stage. The current programing interface is subject to change.
+```
+
+![](../../img/nas_abstract_illustration.png)
+
+Modern Neural Architecture Search (NAS) methods usually incorporate [three dimensions][1]: search space, search strategy, and performance estimation strategy. Search space often contains a limited neural network architectures to explore, while search strategy samples architectures from search space, gets estimations of their performance, and evolves itself. Ideally, search strategy should find the best architecture in the search space and report it to users. After users obtain such "best architecture", many methods use a "retrain step", which trains the network with the same pipeline as any traditional model.
+
+## Implement a Search Space
+
+Assuming now we've got a baseline model, what should we do to be empowered with NAS? Take [MNIST on PyTorch](https://github.com/pytorch/examples/blob/master/mnist/main.py) as an example, the code might look like this:
+
+```python
+from nni.nas.pytorch import mutables
+
+class Net(nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+        self.conv1 = mutables.LayerChoice([
+            nn.Conv2d(1, 32, 3, 1),
+            nn.Conv2d(1, 32, 5, 3)
+        ])  # try 3x3 kernel and 5x5 kernel
+        self.conv2 = nn.Conv2d(32, 64, 3, 1)
+        self.dropout1 = nn.Dropout2d(0.25)
+        self.dropout2 = nn.Dropout2d(0.5)
+        self.fc1 = nn.Linear(9216, 128)
+        self.fc2 = nn.Linear(128, 10)
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = F.relu(x)
+        # ... same as original ...
+        return output
+```
+
+The example above adds an option of choosing conv5x5 at conv1. The modification is as simple as declaring a `LayerChoice` with original conv3x3 and a new conv5x5 as its parameter. That's it! You don't have to modify the forward function in anyway. You can imagine conv1 as any another module without NAS.
+
+So how about the possibilities of connections? This can be done by `InputChoice`. To allow for a skipconnection on an MNIST example, we add another layer called conv3. In the following example, a possible connection from conv2 is added to the output of conv3.
+
+```python
+from nni.nas.pytorch import mutables
+
+class Net(nn.Module):
+    def __init__(self):
+        # ... same ...
+        self.conv2 = nn.Conv2d(32, 64, 3, 1)
+        self.conv3 = nn.Conv2d(64, 64, 1, 1)
+        # declaring that there is exactly one candidate to choose from
+        # search strategy will choose one or None
+        self.skipcon = mutables.InputChoice(n_candidates=1)
+        # ... same ...
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = F.relu(x)
+        x = self.conv2(x)
+        x0 = self.skipcon([x])  # choose one or none from [x]
+        x = self.conv3(x)
+        if x0 is not None:  # skipconnection is open
+            x += x0
+        x = F.max_pool2d(x, 2)
+        # ... same ...
+        return output
+```
+
+Input choice can be thought of as a callable module that receives a list of tensors and output the concatenation/sum/mean of some of them (sum by default), or `None` if none is selected. Like layer choices, input choices should be **initialized in `__init__` and called in `forward`**. We will see later that this is to allow search algorithms to identify these choices, and do necessary preparation.
+
+`LayerChoice` and `InputChoice` are both **mutables**. Mutable means "changeable". As opposed to traditional deep learning layers/modules which have fixed operation type once defined, models with mutables are essentially a series of possible models.
+
+Users can specify a **key** for each mutable. By default NNI will assign one for you that is globally unique, but in case users want to share choices (for example, there are two `LayerChoice` with the same candidate operations, and you want them to have the same choice, i.e., if first one chooses the i-th op, the second one also chooses the i-th op), they can give them the same key. The key marks the identity for this choice, and will be used in dumped checkpoint. So if you want to increase the readability of your exported architecture, manually assigning keys to each mutable would be a good idea. For advanced usage on mutables, see [Mutables](./NasReference.md#mutables).
+
+## Use a Search Algorithm
+
+Different in how the search space is explored and trials are spawned, there are at least two different ways users can do search. One runs NAS distributedly, which can be as naive as enumerating all the architectures and training each one from scratch, or leveraging more advanced technique, such as [SMASH][8], [ENAS][2], [DARTS][1], [FBNet][3], [ProxylessNAS][4], [SPOS][5], [Single-Path NAS][6],  [Understanding One-shot][7] and [GDAS][9]. Since training many different architectures are known to be expensive, another family of methods, called one-shot NAS, builds a supernet containing every candidate in the search space as its subnetwork, and in each step a subnetwork or combination of several subnetworks is trained.
+
+Currently, several one-shot NAS methods have been supported on NNI. For example, `DartsTrainer` which uses SGD to train architecture weights and model weights iteratively, `ENASTrainer` which [uses a controller to train the model][2]. New and more efficient NAS trainers keep emerging in research community.
+
+### One-Shot NAS
+
+Each one-shot NAS implements a trainer, which users can find detailed usages in the description of each algorithm. Here is a simple example, demonstrating how users can use `EnasTrainer`.
+
+```python
+# this is exactly same as traditional model training
+model = Net()
+dataset_train = CIFAR10(root="./data", train=True, download=True, transform=train_transform)
+dataset_valid = CIFAR10(root="./data", train=False, download=True, transform=valid_transform)
+criterion = nn.CrossEntropyLoss()
+optimizer = torch.optim.SGD(model.parameters(), 0.05, momentum=0.9, weight_decay=1.0E-4)
+
+# use NAS here
+def top1_accuracy(output, target):
+    # this is the function that computes the reward, as required by ENAS algorithm
+    batch_size = target.size(0)
+    _, predicted = torch.max(output.data, 1)
+    return (predicted == target).sum().item() / batch_size
+
+def metrics_fn(output, target):
+    # metrics function receives output and target and computes a dict of metrics
+    return {"acc1": reward_accuracy(output, target)}
+
+from nni.nas.pytorch import enas
+trainer = enas.EnasTrainer(model,
+                           loss=criterion,
+                           metrics=metrics_fn,
+                           reward_function=top1_accuracy,
+                           optimizer=optimizer,
+                           batch_size=128
+                           num_epochs=10,  # 10 epochs
+                           dataset_train=dataset_train,
+                           dataset_valid=dataset_valid,
+                           log_frequency=10)  # print log every 10 steps
+trainer.train()  # training
+trainer.export(file="model_dir/final_architecture.json")  # export the final architecture to file
+```
+
+Users can directly run their training file by `python3 train.py`, without `nnictl`. After training, users could export the best one of the found models through `trainer.export()`.
+
+Normally, the trainer exposes a few arguments that you can customize, for example, loss function, metrics function, optimizer, and datasets. These should satisfy the needs from most usages, and we do our best to make sure our built-in trainers work on as many models, tasks and datasets as possible. But there is no guarantee. For example, some trainers have assumption that the task has to be a classification task; some trainers might have a different definition of "epoch" (e.g., an ENAS epoch = some child steps + some controller steps); most trainers do not have support for distributed training: they won't wrap your model with `DataParallel` or `DistributedDataParallel` to do that. So after a few tryouts, if you want to actually use the trainers on your very customized applications, you might very soon need to [customize your trainer](#extend-the-ability-of-one-shot-trainers).
+
+### Distributed NAS
+
+Neural architecture search is originally executed by running each child model independently as a trial job. We also support this searching approach, and it naturally fits in NNI hyper-parameter tuning framework, where tuner generates child model for next trial and trials run in training service.
+
+To use this mode, there is no need to change the search space expressed with NNI NAS API (i.e., `LayerChoice`, `InputChoice`, `MutableScope`). After the model is initialized, apply the function `get_and_apply_next_architecture` on the model. One-shot NAS trainers are not used in this mode. Here is a simple example:
+
+```python
+model = Net()
+
+# get the chosen architecture from tuner and apply it on model
+get_and_apply_next_architecture(model)
+train(model)  # your code for training the model
+acc = test(model)  # test the trained model
+nni.report_final_result(acc)  # report the performance of the chosen architecture
+```
+
+The search space should be generated and sent to tuner. As with NNI NAS API the search space is embedded in user code, users could use "[nnictl ss_gen](../Tutorial/Nnictl.md)" to generate search space file. Then, put the path of the generated search space in the field `searchSpacePath` of `config.yml`. The other fields in `config.yml` can be filled by referring [this tutorial](../Tutorial/QuickStart.md).
+
+You could use [NNI tuners](../Tuner/BuiltinTuner.md) to do the search. Currently, only PPO Tuner supports NAS search space.
+
+We support standalone mode for easy debugging, where you could directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for `LayerChoice` and `InputChoice` in this standalone mode.
+
+A complete example can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/classic_nas/config_nas.yml).
+
+### Retrain with Exported Architecture
+
+After the searching phase, it's time to train the architecture found. Unlike many open-source NAS algorithms who write a whole new model specifically for retraining. We found that searching model and retraining model are usual very similar, and therefore you can construct your final model with the exact model code. For example
+
+```python
+model = Net()
+apply_fixed_architecture(model, "model_dir/final_architecture.json")
+```
+
+The JSON is simply a mapping from mutable keys to one-hot or multi-hot representation of choices. For example
+
+```json
+{
+    "LayerChoice1": [false, true, false, false],
+    "InputChoice2": [true, true, false]
+}
+```
+
+After applying, the model is then fixed and ready for a final training. The model works as a single model, although it might contain more parameters than expected. This comes with pros and cons. The good side is, you can directly load the checkpoint dumped from supernet during search phase and start retrain from there. However, this is also a model with redundant parameters, which may cause problems when trying to count the number of parameters in model. For deeper reasons and possible workaround, see [Trainers](./NasReference.md#retrain).
+
+Also refer to [DARTS](./DARTS.md) for example code of retraining.
+
+## Customize a Search Algorithm
+
+### Extend the Ability of One-Shot Trainers
+
+Users might want to do multiple things if they are using the trainers on real tasks, for example, distributed training, half-precision training, logging periodically, writing tensorboard, dumping checkpoints and so on. As mentioned previously, some trainers do have support for some of the items listed above; others might not. Generally, there are two recommended ways to add anything you want to an existing trainer: inherit an existing trainer and override, or copy an existing trainer and modify.
+
+Either way, you are walking into the scope of implementing a new trainer. Basically, implementing a one-shot trainer is no different from any traditional deep learning trainer, except that a new concept called mutator will reveal itself. So that the implementation will be different in at least two places:
+
+* Initialization
+
+```python
+model = Model()
+mutator = MyMutator(model)
+```
+
+* Training
+
+```python
+for _ in range(epochs):
+    for x, y in data_loader:
+        mutator.reset()  # reset all the choices in model
+        out = model(x)  # like traditional model
+        loss = criterion(out, y)
+        loss.backward()
+        # no difference below
+```
+
+To demonstrate what mutators are for, we need to know how one-shot NAS normally works. Usually, one-shot NAS "co-optimize model weights and architecture weights". It repeatedly: sample an architecture or combination of several architectures from the supernet, train the chosen architectures like traditional deep learning model, update the trained parameters to the supernet, and use the metrics or loss as some signal to guide the architecture sampler. The mutator, is the architecture sampler here, often defined to be another deep-learning model. Therefore, you can treat it as any model, by defining parameters in it and optimizing it with optimizers. One mutator is initialized with exactly one model. Once a mutator is binded to a model, it cannot be rebinded to another model.
+
+`mutator.reset()` is the core step. That's where all the choices in the model are finalized. The reset result will be always effective, until the next reset flushes the data. After the reset, the model can be seen as a traditional model to do forward-pass and backward-pass.
+
+Finally, mutators provide a method called `mutator.export()` that export a dict with architectures to the model. Note that currently this dict this a mapping from keys of mutables to tensors of selection. So in order to dump to json, users need to convert the tensors explicitly into python list.
+
+Meanwhile, NNI provides some useful tools so that users can implement trainers more easily. See [Trainers](./NasReference.md#trainers) for details.
+
+### Implement New Mutators
+
+To start with, here is the pseudo-code that demonstrates what happens on `mutator.reset()` and `mutator.export()`.
+
+```python
+def reset(self):
+    self.apply_on_model(self.sample_search())
+```
+
+```python
+def export(self):
+    return self.sample_final()
+```
+
+On reset, a new architecture is sampled with `sample_search()` and applied on the model. Then the model is trained for one or more steps in search phase. On export, a new architecture is sampled with `sample_final()` and **do nothing to the model**. This is either for checkpoint or exporting the final architecture.
+
+The requirements of return values of `sample_search()` and `sample_final()` are the same: a mapping from mutable keys to tensors. The tensor can be either a BoolTensor (true for selected, false for negative), or a FloatTensor which applies weight on each candidate. The selected branches will then be computed (in `LayerChoice`, modules will be called; in `InputChoice`, it's just tensors themselves), and reduce with the reduction operation specified in the choices. For most algorithms only worrying about the former part, here is an example of your mutator implementation.
+
+```python
+class RandomMutator(Mutator):
+    def __init__(self, model):
+        super().__init__(model)  # don't forget to call super
+        # do something else
+
+    def sample_search(self):
+        result = dict()
+        for mutable in self.mutables:  # this is all the mutable modules in user model
+            # mutables share the same key will be de-duplicated
+            if isinstance(mutable, LayerChoice):
+                # decided that this mutable should choose `gen_index`
+                gen_index = np.random.randint(mutable.length)
+                result[mutable.key] = torch.tensor([i == gen_index for i in range(mutable.length)], 
+                                                   dtype=torch.bool)
+            elif isinstance(mutable, InputChoice):
+                if mutable.n_chosen is None:  # n_chosen is None, then choose any number
+                    result[mutable.key] = torch.randint(high=2, size=(mutable.n_candidates,)).view(-1).bool()
+                # else do something else
+        return result
+
+    def sample_final(self):
+        return self.sample_search()  # use the same logic here. you can do something different
+```
+
+The complete example of random mutator can be found [here](https://github.com/microsoft/nni/blob/master/src/sdk/pynni/nni/nas/pytorch/random/mutator.py).
+
+For advanced usages, e.g., users want to manipulate the way modules in `LayerChoice` are executed, they can inherit `BaseMutator`, and overwrite `on_forward_layer_choice` and `on_forward_input_choice`, which are the callback implementation of `LayerChoice` and `InputChoice` respectively. Users can still use property `mutables` to get all `LayerChoice` and `InputChoice` in the model code. For details, please refer to [reference](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch) here to learn more.
+
+```eval_rst
+.. tip::
+    A useful application of random mutator is for debugging. Use
+   
+    .. code-block:: python
+
+        mutator = RandomMutator(model)
+        mutator.reset()
+
+    will immediately set one possible candidate in the search space as the active one.
+```
+
+### Implemented a Distributed NAS Tuner
+
+Before learning how to write a one-shot NAS tuner, users should first learn how to write a general tuner. read [Customize Tuner](../Tuner/CustomizeTuner.md) for tutorials.
+
+When users call "[nnictl ss_gen](../Tutorial/Nnictl.md)" to generate search space file, a search space file like this will be generated:
+
+```json
+{
+    "key_name": {
+        "_type": "layer_choice",
+        "_value": ["op1_repr", "op2_repr", "op3_repr"]
+    },
+    "key_name": {
+        "_type": "input_choice",
+        "_value": {
+            "candidates": ["in1_key", "in2_key", "in3_key"],
+            "n_chosen": 1
+        }
+    }
+}
+```
+
+This is the exact search space tuners will receive in `update_search_space`. It's then tuners' responsibility to interpret the search space and generate new candidates in `generate_parameters`. A valid "parameters" will be in the following format:
+
+```json
+{
+    "key_name": {
+        "_value": "op1_repr",
+        "_idx": 0
+    },
+    "key_name": {
+        "_value": ["in2_key"],
+        "_idex": [1]
+    }
+}
+```
+
+Send it through `generate_parameters`, and the tuner would look like any HPO tuner. Refer to [SPOS](./SPOS.md) example code for an example.
+
+[1]: https://arxiv.org/abs/1808.05377
+[2]: https://arxiv.org/abs/1802.03268
+[3]: https://arxiv.org/abs/1812.03443
+[4]: https://arxiv.org/abs/1812.00332
+[5]: https://arxiv.org/abs/1904.00420
+[6]: https://arxiv.org/abs/1904.02877
+[7]: http://proceedings.mlr.press/v80/bender18a
+[8]: https://arxiv.org/abs/1708.05344
+[9]: https://arxiv.org/abs/1910.04465
\ No newline at end of file

docs/en_US/NAS/NasInterface.md

-# NNI NAS Programming Interface
-
-We are trying to support various NAS algorithms with unified programming interface, and it's still in experimental stage. It means the current programing interface might be updated in future.
-
-## Programming interface for user model
-
-The programming interface of designing and searching a model is often demanded in two scenarios.
-
-1. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs  best. So, it needs an easy way to express the candidate layers or sub-models.
-2. When applying NAS on a neural network, it needs an unified way to express the search space of architectures, so that it doesn't need to update trial code for different searching algorithms.
-
-
-For expressing neural architecture search space in user code, we provide the following APIs (take PyTorch as example):
-
-```python
-# in PyTorch module class
-def __init__(self):
-    ...
-    # choose one ``op`` from ``ops``, for PyTorch this is a module.
-    # op_candidates: for PyTorch ``ops`` is a list of modules, for tensorflow it is a list of keras layers.
-    # key: the name of this ``LayerChoice`` instance
-    self.one_layer = nni.nas.pytorch.LayerChoice([
-        PoolBN('max', channels, 3, stride, 1, affine=False),
-        PoolBN('avg', channels, 3, stride, 1, affine=False),
-        FactorizedReduce(channels, channels, affine=False),
-        SepConv(channels, channels, 3, stride, 1, affine=False),
-        DilConv(channels, channels, 3, stride, 2, 2, affine=False)],
-        key="layer_name")
-    ...
-
-def forward(self, x):
-    ...
-    out = self.one_layer(x)
-    ...
-```
-This is for users to specify multiple candidate operations for a layer, one operation will be chosen at last. `key` is the identifier of the layer,it could be used to share choice between multiple `LayerChoice`. For example, there are two `LayerChoice` with the same candidate operations, and you want them to have the same choice (i.e., if first one chooses the `i`th op, the second one also chooses the `i`th op), give them the same key.
-
-```python
-def __init__(self):
-    ...
-    # choose ``n_selected`` from ``n_candidates`` inputs.
-    # n_candidates: the number of candidate inputs
-    # n_chosen: the number of chosen inputs
-    # key: the name of this ``InputChoice`` instance
-    self.input_switch = nni.nas.pytorch.InputChoice(
-        n_candidates=3,
-        n_chosen=1,
-        key="switch_name")
-    ...
-
-def forward(self, x):
-    ...
-    out = self.input_switch([in_tensor1, in_tensor2, in_tensor3])
-    ...
-```
-`InputChoice` is a PyTorch module, in init, it needs meta information, for example, from how many input candidates to choose how many inputs, and the name of this initialized `InputChoice`. The real candidate input tensors can only be obtained in `forward` function. In the `forward` function, the `InputChoice` module you create in `__init__` (e.g., `self.input_switch`) is called with real candidate input tensors.
-
-Some [NAS trainers](#one-shot-training-mode) need to know the source layer the input tensors, thus, we add one input argument `choose_from` in `InputChoice` to indicate the source layer of each candidate input. `choose_from` is a list of string, each element is `key` of `LayerChoice` and `InputChoice` or the name of a module (refer to [the code](https://github.com/microsoft/nni/blob/master/src/sdk/pynni/nni/nas/pytorch/mutables.py) for more details).
-
-
-Besides `LayerChoice` and `InputChoice`, we also provide `MutableScope` which allows users to label a sub-network, thus, could provide more semantic information (e.g., the structure of the network) to NAS trainers. Here is an example:
-```python
-class Cell(mutables.MutableScope):
-    def __init__(self, scope_name):
-        super().__init__(scope_name)
-        self.layer1 = nni.nas.pytorch.LayerChoice(...)
-        self.layer2 = nni.nas.pytorch.LayerChoice(...)
-        self.layer3 = nni.nas.pytorch.LayerChoice(...)
-        ...
-```
-The three `LayerChoice` (`layer1`, `layer2`, `layer3`) are included in the `MutableScope` named `scope_name`. NAS trainer could get this hierarchical structure.
-
-
-## Two training modes
-
-After writing your model with search space embedded in the model using the above APIs, the next step is finding the best model from the search space. There are two training modes: [one-shot training mode](#one-shot-training-mode) and [classic distributed search](#classic-distributed-search).
-
-### One-shot training mode
-
-Similar to optimizers of deep learning models, the procedure of finding the best model from search space can be viewed as a type of optimizing process, we call it `NAS trainer`. There have been several NAS trainers, for example, `DartsTrainer` which uses SGD to train architecture weights and model weights iteratively, `ENASTrainer` which uses a controller to train the model. New and more efficient NAS trainers keep emerging in research community.
-
-NNI provides some popular NAS trainers, to use a NAS trainer, users could initialize a trainer after the model is defined:
-
-```python
-# create a DartsTrainer
-trainer = DartsTrainer(model,
-                       loss=criterion,
-                       metrics=lambda output, target: accuracy(output, target, topk=(1,)),
-                       optimizer=optim,
-                       num_epochs=args.epochs,
-                       dataset_train=dataset_train,
-                       dataset_valid=dataset_valid,)
-# finding the best model from search space
-trainer.train()
-# export the best found model
-trainer.export(file='./chosen_arch')
-```
-
-Different trainers could have different input arguments depending on their algorithms. Please refer to [each trainer's code](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch) for detailed arguments. After training, users could export the best one of the found models through `trainer.export()`. No need to start an NNI experiment through `nnictl`.
-
-The supported trainers can be found [here](Overview.md#supported-one-shot-nas-algorithms). A very simple example using NNI NAS API can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/simple/train.py).
-
-### Classic distributed search
-
-Neural architecture search is originally executed by running each child model independently as a trial job. We also support this searching approach, and it naturally fits in NNI hyper-parameter tuning framework, where tuner generates child model for next trial and trials run in training service.
-
-For using this mode, no need to change the search space expressed with NNI NAS API (i.e., `LayerChoice`, `InputChoice`, `MutableScope`). After the model is initialized, apply the function `get_and_apply_next_architecture` on the model. One-shot NAS trainers are not used in this mode. Here is a simple example:
-```python
-class Net(nn.Module):
-    # defined model with LayerChoice and InputChoice
-    ...
-
-model = Net()
-# get the chosen architecture from tuner and apply it on model
-get_and_apply_next_architecture(model)
-# your code for training the model
-train(model)
-# test the trained model
-acc = test(model)
-# report the performance of the chosen architecture
-nni.report_final_result(acc)
-```
-
-The search space should be automatically generated and sent to tuner. As with NNI NAS API the search space is embedded in user code, users could use "[nnictl ss_gen](../Tutorial/Nnictl.md)" to generate search space file. Then, put the path of the generated search space in the field `searchSpacePath` of `config.yml`. The other fields in `config.yml` can be filled by referring [this tutorial](../Tutorial/QuickStart.md).
-
-You could use [NNI tuners](../Tuner/BuiltinTuner.md) to do the search.
-
-We support standalone mode for easy debugging, where you could directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for `LayerChoice` and `InputChoice` in this standalone mode.
-
-The complete example code can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/classic_nas/config_nas.yml).
-
-## Programming interface for NAS algorithm
-
-We also provide simple interface for users to easily implement a new NAS trainer on NNI.
-
-### Implement a new NAS trainer on NNI
-
-To implement a new NAS trainer, users basically only need to implement two classes by inheriting `BaseMutator` and `BaseTrainer` respectively.
-
-In `BaseMutator`, users need to overwrite `on_forward_layer_choice` and `on_forward_input_choice`, which are the implementation of `LayerChoice` and `InputChoice` respectively. Users could use property `mutables` to get all `LayerChoice` and `InputChoice` in the model code. Then users need to implement a new trainer, which instantiates the new mutator and implement the training logic. For details, please read [the code](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch) and the supported trainers, for example, [DartsTrainer](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch/darts).
-
-### Implement an NNI tuner for NAS
-
-NNI tuner for NAS takes the auto generated search space. The search space format of `LayerChoice` and `InputChoice` is shown below:
-```json
-{
-    "key_name": {
-        "_type": "layer_choice",
-        "_value": ["op1_repr", "op2_repr", "op3_repr"]
-    },
-    "key_name": {
-        "_type": "input_choice",
-        "_value": {
-            "candidates": ["in1_key", "in2_key", "in3_key"],
-            "n_chosen": 1
-        }
-    }
-}
-```
-
-Correspondingly, the generate architecture is in the following format:
-```json
-{
-    "key_name": {
-        "_value": "op1_repr",
-        "_idx": 0
-    },
-    "key_name": {
-        "_value": ["in2_key"],
-        "_idex": [1]
-    }
-}
-```

docs/en_US/NAS/NasReference.md

+# NAS Reference
+
+```eval_rst
+.. contents::
+```
+
+## Mutables
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.mutables.Mutable
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.LayerChoice
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.InputChoice
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.MutableScope
+    :members:
+```
+
+### Utilities
+
+```eval_rst
+..  autofunction:: nni.nas.pytorch.utils.global_mutable_counting
+```
+
+## Mutators
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.base_mutator.BaseMutator
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutator.Mutator
+    :members:
+```
+
+### Random Mutator
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.random.RandomMutator
+    :members:
+```
+
+### Utilities
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.utils.StructuredMutableTreeNode
+    :members:
+```
+
+## Trainers
+
+### Trainer
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.base_trainer.BaseTrainer
+    :members:
+
+..  autoclass:: nni.nas.pytorch.trainer.Trainer
+    :members:
+```
+
+### Retrain
+
+```eval_rst
+..  autofunction:: nni.nas.pytorch.fixed.apply_fixed_architecture
+
+..  autoclass:: nni.nas.pytorch.fixed.FixedArchitecture
+    :members:
+```
+
+### Distributed NAS
+
+```eval_rst
+..  autofunction:: nni.nas.pytorch.classic_nas.get_and_apply_next_architecture
+
+..  autoclass:: nni.nas.pytorch.classic_nas.mutator.ClassicMutator
+    :members:
+```
+
+### Callbacks
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.callbacks.Callback
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.LRSchedulerCallback
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.ArchitectureCheckpoint
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.ModelCheckpoint
+    :members:
+```
+
+### Utilities
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.utils.AverageMeterGroup
+    :members:
+
+..  autoclass:: nni.nas.pytorch.utils.AverageMeter
+    :members:
+
+..  autofunction:: nni.nas.pytorch.utils.to_device
+```

docs/en_US/NAS/Overview.md

@@ -6,11 +6,7 @@ However, it takes great efforts to implement NAS algorithms, and it is hard to r
 
 With this motivation, our ambition is to provide a unified architecture in NNI, to accelerate innovations on NAS, and apply state-of-art algorithms on real world problems faster.
 
-With [the unified interface](./NasInterface.md), there are two different modes for the architecture search. [One](#supported-one-shot-nas-algorithms) is the so-called one-shot NAS, where a super-net is built based on search space, and using one shot training to generate good-performing child model. [The other](./NasInterface.md#classic-distributed-search) is the traditional searching approach, where each child model in search space runs as an independent trial, the performance result is sent to tuner and the tuner generates new child model.
-
-* [Supported One-shot NAS Algorithms](#supported-one-shot-nas-algorithms)
-* [Classic Distributed NAS with NNI experiment](./NasInterface.md#classic-distributed-search)
-* [NNI NAS Programming Interface](./NasInterface.md)
+With the unified interface, there are two different modes for the architecture search. [One](#supported-one-shot-nas-algorithms) is the so-called one-shot NAS, where a super-net is built based on search space, and using one shot training to generate good-performing child model. [The other](#supported-distributed-nas-algorithms) is the traditional searching approach, where each child model in search space runs as an independent trial, the performance result is sent to tuner and the tuner generates new child model.
 
 ## Supported One-shot NAS Algorithms
 
@@ -33,18 +29,26 @@ Here are some common dependencies to run the examples. PyTorch needs to be above
 * PyTorch 1.2+
 * git
 
-## Use NNI API
+## Supported Distributed NAS Algorithms
+
+|Name|Brief Introduction of Algorithm|
+|---|---|
+| [SPOS](SPOS.md) | [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://arxiv.org/abs/1904.00420) constructs a simplified supernet trained with an uniform path sampling method, and applies an evolutionary algorithm to efficiently search for the best-performing architectures. |
 
-NOTE, we are trying to support various NAS algorithms with unified programming interface, and it's in very experimental stage. It means the current programing interface may be updated in future.
+```eval_rst
+.. Note:: SPOS is a two-stage algorithm, whose first stage is one-shot and second stage is distributed, leveraging result of first stage as a checkpoint.
+```
 
-### Programming interface
+## Use NNI API
 
 The programming interface of designing and searching a model is often demanded in two scenarios.
 
 1. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs  best. So, it needs an easy way to express the candidate layers or sub-models.
 2. When applying NAS on a neural network, it needs an unified way to express the search space of architectures, so that it doesn't need to update trial code for different searching algorithms.
 
-NNI proposed API is [here](https://github.com/microsoft/nni/tree/master/src/sdk/pynni/nni/nas/pytorch). And [here](https://github.com/microsoft/nni/tree/master/examples/nas/naive) is an example of NAS implementation, which bases on NNI proposed interface.
+[Here](./NasGuide.md) is a user guide to get started with using NAS on NNI.
+
+## Reference and Feedback
 
 [1]: https://arxiv.org/abs/1802.03268
 [2]: https://arxiv.org/abs/1707.07012
@@ -52,9 +56,5 @@ NNI proposed API is [here](https://github.com/microsoft/nni/tree/master/src/sdk/
 [4]: https://arxiv.org/abs/1806.10282
 [5]: https://arxiv.org/abs/1703.01041
 
-## **Reference and Feedback**
 * To [report a bug](https://github.com/microsoft/nni/issues/new?template=bug-report.md) for this feature in GitHub;
-* To [file a feature or improvement request](https://github.com/microsoft/nni/issues/new?template=enhancement.md) for this feature in GitHub;
-* To know more about [Feature Engineering with NNI](https://github.com/microsoft/nni/blob/master/docs/en_US/FeatureEngineering/Overview.md);
-* To know more about [Model Compression with NNI](https://github.com/microsoft/nni/blob/master/docs/en_US/Compressor/Overview.md);
-* To know more about [Hyperparameter Tuning with NNI](https://github.com/microsoft/nni/blob/master/docs/en_US/Tuner/BuiltinTuner.md);
+* To [file a feature or improvement request](https://github.com/microsoft/nni/issues/new?template=enhancement.md) for this feature in GitHub.
\ No newline at end of file

docs/en_US/NAS/SPOS.md

@@ -93,17 +93,11 @@ By default, it will use `architecture_final.json`. This architecture is provided
 ..  autoclass:: nni.nas.pytorch.spos.SPOSEvolution
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.spos.SPOSSupernetTrainer
     :members:
 
-    .. automethod:: __init__
-
 ..  autoclass:: nni.nas.pytorch.spos.SPOSSupernetTrainingMutator
     :members:
-
-    .. automethod:: __init__
 ```
 
 ## Known Limitations

docs/en_US/Release.md

@@ -213,7 +213,7 @@
 
 ### Major Features
 
-* [Support NNI on Windows](Tutorial/NniOnWindows.md)
+* [Support NNI on Windows](Tutorial/InstallationWin.md)
   * NNI running on windows for local mode
 * [New advisor: BOHB](Tuner/BohbAdvisor.md)
   * Support a new advisor BOHB, which is a robust and efficient hyperparameter tuning algorithm, combines the advantages of Bayesian optimization and Hyperband

docs/en_US/TrainingService/RemoteMachineMode.md

@@ -4,9 +4,9 @@ NNI can run one experiment on multiple remote machines through SSH, called `remo
 
 ## Remote machine requirements
 
-* It only supports Linux as remote machines, and [linux part in system specification](../Tutorial/Installation.md) is same as NNI local mode.
+* It only supports Linux as remote machines, and [linux part in system specification](../Tutorial/InstallationLinux.md) is same as NNI local mode.
 
-* Follow [installation](../Tutorial/Installation.md) to install NNI on each machine.
+* Follow [installation](../Tutorial/InstallationLinux.md) to install NNI on each machine.
 
 * Make sure remote machines meet environment requirements of your trial code. If the default environment does not meet the requirements, the setup script can be added into `command` field of NNI config.
 

docs/en_US/Tutorial/FAQ.md

@@ -45,7 +45,7 @@ Probably it's a problem with your network config. Here is a checklist.
 
 
 ### NNI on Windows problems
-Please refer to [NNI on Windows](NniOnWindows.md)
+Please refer to [NNI on Windows](InstallationWin.md#FAQ)
 
 
 ### More FAQ issues

docs/en_US/Tutorial/HowToUseDocker.md

@@ -35,7 +35,7 @@ Note:
 
 If you start a docker image using NNI's offical image `msranni/nni`, you could directly start NNI experiments by using `nnictl` command. Our offical image has NNI's running environment and basic python and deep learning frameworks environment.
 
-If you start your own docker image, you may need to install NNI package first, please [refer](Installation.md).
+If you start your own docker image, you may need to install NNI package first, please refer to [NNI installation](InstallationLinux.md).
 
 If you want to run NNI's offical examples, you may need to clone NNI repo in github using
 ```

docs/en_US/Tutorial/Installation.md → docs/en_US/Tutorial/InstallationLinux.md

-# Installation of NNI
+# Install on Linux & Mac
 
-Currently we support installation on Linux, macOS and Windows.
+## Installation
 
-## Install on Linux or macOS
+Installation on Linux and macOS follow the same instruction below.
 
-* Install NNI through pip
+### Install NNI through pip
 
   Prerequisite: `python 64-bit >= 3.5`
 
@@ -12,46 +12,22 @@ Currently we support installation on Linux, macOS and Windows.
   python3 -m pip install --upgrade nni
   ```
 
-* Install NNI through source code
+### Install NNI through source code
 
   If you are interested on special or latest code version, you can install NNI through source code.
 
   Prerequisites: `python 64-bit >=3.5`, `git`, `wget`
 
   ```bash
-  git clone -b v0.8 https://github.com/Microsoft/nni.git
+  git clone -b v1.3 https://github.com/Microsoft/nni.git
   cd nni
   ./install.sh
   ```
 
-* Use NNI in a docker image
+### Use NNI in a docker image
 
   You can also install NNI in a docker image. Please follow the instructions [here](https://github.com/Microsoft/nni/tree/master/deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`.
 
-## Install on Windows
-
-  Anaconda or Miniconda is highly recommended to manage multiple Python environments.
-
-* Install NNI through pip
-
-  Prerequisites: `python 64-bit >= 3.5`
-
-  ```bash
-  python -m pip install --upgrade nni
-  ```
-
-* Install NNI through source code
-
-  If you are interested on special or latest code version, you can install NNI through source code.
-
-  Prerequisites: `python 64-bit >=3.5`, `git`, `PowerShell`.
-
-  ```bash
-  git clone -b v0.8 https://github.com/Microsoft/nni.git
-  cd nni
-  powershell -ExecutionPolicy Bypass -file install.ps1
-  ```
-
 ## Verify installation
 
 The following example is built on TensorFlow 1.x. Make sure **TensorFlow 1.x is used** when running it.
@@ -64,18 +40,10 @@ The following example is built on TensorFlow 1.x. Make sure **TensorFlow 1.x is
 
 * Run the MNIST example.
 
-  Linux or macOS
-
   ```bash
   nnictl create --config nni/examples/trials/mnist-tfv1/config.yml
   ```
 
-  Windows
-
-  ```bash
-  nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
-  ```
-
 * Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.
 
 ```text
@@ -138,18 +106,6 @@ Due to potential programming changes, the minimum system requirements of NNI may
 | **Internet**         | Boardband internet connection         |
 | **Resolution**       | 1024 x 768 minimum display resolution |
 
-### Windows
-
-|                      | Recommended                                    | Minimum                                |
-| -------------------- | ---------------------------------------------- | -------------------------------------- |
-| **Operating System** | Windows 10 1809 or above                       |
-| **CPU**              | Intel® Core™ i5 or AMD Phenom™ II X3 or better | Intel® Core™ i3 or AMD Phenom™ X3 8650 |
-| **GPU**              | NVIDIA® GeForce® GTX 660 or better             | NVIDIA® GeForce® GTX 460               |
-| **Memory**           | 6 GB RAM                                       | 4 GB RAM                               |
-| **Storage**          | 30 GB available hare drive space               |
-| **Internet**         | Boardband internet connection                  |
-| **Resolution**       | 1024 x 768 minimum display resolution          |
-
 ## Further reading
 
 * [Overview](../Overview.md)

docs/en_US/Tutorial/InstallationWin.md

+# Install on Windows
+
+## Installation
+
+Anaconda or Miniconda is highly recommended to manage multiple Python environments.
+
+### Install NNI through pip
+
+  Prerequisites: `python 64-bit >= 3.5`
+
+  ```bash
+  python -m pip install --upgrade nni
+  ```
+
+### Install NNI through source code
+
+  If you are interested on special or latest code version, you can install NNI through source code.
+
+  Prerequisites: `python 64-bit >=3.5`, `git`, `PowerShell`.
+
+  ```bash
+  git clone -b v1.3 https://github.com/Microsoft/nni.git
+  cd nni
+  powershell -ExecutionPolicy Bypass -file install.ps1
+  ```
+
+## Verify installation
+
+The following example is built on TensorFlow 1.x. Make sure **TensorFlow 1.x is used** when running it.
+
+* Download the examples via clone the source code.
+
+  ```bash
+  git clone -b v1.3 https://github.com/Microsoft/nni.git
+  ```
+
+* Run the MNIST example.
+
+  ```bash
+  nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
+  ```
+
+  Note: for other examples you need to change trial command `python3` to `python` in each example YAML, if python3 is called through `python` on your machine.
+
+* Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.
+
+```text
+INFO: Starting restful server...
+INFO: Successfully started Restful server!
+INFO: Setting local config...
+INFO: Successfully set local config!
+INFO: Starting experiment...
+INFO: Successfully started experiment!
+-----------------------------------------------------------------------
+The experiment id is egchD4qy
+The Web UI urls are: http://223.255.255.1:8080   http://127.0.0.1:8080
+-----------------------------------------------------------------------
+
+You can use these commands to get more information about the experiment
+-----------------------------------------------------------------------
+         commands                       description
+1. nnictl experiment show        show the information of experiments
+2. nnictl trial ls               list all of trial jobs
+3. nnictl top                    monitor the status of running experiments
+4. nnictl log stderr             show stderr log content
+5. nnictl log stdout             show stdout log content
+6. nnictl stop                   stop an experiment
+7. nnictl trial kill             kill a trial job by id
+8. nnictl --help                 get help information about nnictl
+-----------------------------------------------------------------------
+```
+
+* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](../Tutorial/WebUI.md) are more Web UI pages.
+
+![overview](../../img/webui_overview_page.png)
+
+![detail](../../img/webui_trialdetail_page.png)
+
+## System requirements
+
+Below are the minimum system requirements for NNI on Windows, Windows 10.1809 is well tested and recommend. Due to potential programming changes, the minimum system requirements for NNI may change over time.
+
+|                      | Recommended                                    | Minimum                                |
+| -------------------- | ---------------------------------------------- | -------------------------------------- |
+| **Operating System** | Windows 10 1809 or above                       |
+| **CPU**              | Intel® Core™ i5 or AMD Phenom™ II X3 or better | Intel® Core™ i3 or AMD Phenom™ X3 8650 |
+| **GPU**              | NVIDIA® GeForce® GTX 660 or better             | NVIDIA® GeForce® GTX 460               |
+| **Memory**           | 6 GB RAM                                       | 4 GB RAM                               |
+| **Storage**          | 30 GB available hare drive space               |
+| **Internet**         | Boardband internet connection                  |
+| **Resolution**       | 1024 x 768 minimum display resolution          |
+
+## FAQ
+
+### simplejson failed when installing NNI
+
+Make sure C++ 14.0 compiler installed.
+>building 'simplejson._speedups' extension error: [WinError 3] The system cannot find the path specified
+
+### Trial failed with missing DLL in command line or PowerShell
+
+This error caused by missing LIBIFCOREMD.DLL and LIBMMD.DLL and fail to install SciPy. Using Anaconda or Miniconda with Python(64-bit) can solve it.
+>ImportError: DLL load failed
+
+### Trial failed on webUI
+
+Please check the trial log file stderr for more details.
+
+If there is a stderr file, please check out. Two possible cases are as follows:
+
+* forget to change the trial command `python3` into `python` in each experiment YAML.
+* forget to install experiment dependencies such as TensorFlow, Keras and so on.
+
+### Fail to use BOHB on Windows
+Make sure C++ 14.0 compiler installed then try to run `nnictl package install --name=BOHB` to install the dependencies.
+
+### Not supported tuner on Windows
+SMAC is not supported currently, the specific reason can be referred to this [GitHub issue](https://github.com/automl/SMAC3/issues/483).
+
+### Use a Windows server as a remote worker
+Currently you can't.
+
+Note:
+
+* If there is any error like `Segmentation fault`, please refer to [FAQ](FAQ.md)
+
+
+## Further reading
+
+* [Overview](../Overview.md)
+* [Use command line tool nnictl](Nnictl.md)
+* [Use NNIBoard](WebUI.md)
+* [Define search space](SearchSpaceSpec.md)
+* [Config an experiment](ExperimentConfig.md)
+* [How to run an experiment on local (with multiple GPUs)?](../TrainingService/LocalMode.md)
+* [How to run an experiment on multiple machines?](../TrainingService/RemoteMachineMode.md)
+* [How to run an experiment on OpenPAI?](../TrainingService/PaiMode.md)
+* [How to run an experiment on Kubernetes through Kubeflow?](../TrainingService/KubeflowMode.md)
+* [How to run an experiment on Kubernetes through FrameworkController?](../TrainingService/FrameworkControllerMode.md)
\ No newline at end of file

docs/en_US/Tutorial/NniOnWindows.md

-# NNI on Windows (experimental feature)
-
-Running NNI on Windows is an experimental feature. Windows 10.1809 is well tested and recommended.
-
-## **Installation on Windows**
-
-  please refer to [Installation](Installation.md) for more details.
-
-When these things are done, use the **config_windows.yml** configuration to start an experiment for validation.
-
-```bash
-nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
-```
-
-For other examples you need to change trial command `python3` into `python` in each example YAML.
-
-## **FAQ**
-
-### simplejson failed when installing NNI
-
-Make sure C++ 14.0 compiler installed.
->building 'simplejson._speedups' extension error: [WinError 3] The system cannot find the path specified
-
-### Trial failed with missing DLL in command line or PowerShell
-
-This error caused by missing LIBIFCOREMD.DLL and LIBMMD.DLL and fail to install SciPy. Using Anaconda or Miniconda with Python(64-bit) can solve it.
->ImportError: DLL load failed
-
-### Trial failed on webUI
-
-Please check the trial log file stderr for more details.
-
-If there is a stderr file, please check out. Two possible cases are as follows:
-
-* forget to change the trial command `python3` into `python` in each experiment YAML.
-* forget to install experiment dependencies such as TensorFlow, Keras and so on.
-
-### Fail to use BOHB on Windows
-Make sure C++ 14.0 compiler installed then try to run `nnictl package install --name=BOHB` to install the dependencies.
-
-### Not supported tuner on Windows
-SMAC is not supported currently, the specific reason can be referred to this [GitHub issue](https://github.com/automl/SMAC3/issues/483).
-
-### Use a Windows server as a remote worker
-Currently you can't.
-
-Note:
-
-* If there is any error like `Segmentation fault`, please refer to [FAQ](FAQ.md)

docs/en_US/Tutorial/QuickStart.md

@@ -20,7 +20,7 @@ Note:
 
 * For Linux and macOS `--user` can be added if you want to install NNI in your home directory, which does not require any special privileges.
 * If there is any error like `Segmentation fault`, please refer to [FAQ](FAQ.md)
-* For the `system requirements` of NNI, please refer to [Install NNI](Installation.md)
+* For the `system requirements` of NNI, please refer to [Install NNI on Linux&Mac](InstallationLinux.md) or [Windows](InstallationWin.md)
 
 ## "Hello World" example on MNIST
 

docs/en_US/advanced.rst

-Advanced Features
-=====================
-
-..  toctree::
-    MultiPhase<./AdvancedFeature/MultiPhase>

docs/en_US/assessors.rst

-Assessors
-==============
-In order to save our computing resources, NNI supports an early stop policy and creates **Assessor** to finish this job.
-
-Assessor receives the intermediate result from Trial and decides whether the Trial should be killed by specific algorithm. Once the Trial experiment meets the early stop conditions(which means assessor is pessimistic about the final results), the assessor will kill the trial and the status of trial will be `"EARLY_STOPPED"`.
-
-Here is an experimental result of MNIST after using 'Curvefitting' Assessor in 'maximize' mode, you can see that assessor successfully **early stopped** many trials with bad hyperparameters in advance. If you use assessor, we may get better hyperparameters under the same computing resources.
-
-*Implemented code directory: config_assessor.yml <https://github.com/Microsoft/nni/blob/master/examples/trials/mnist-tfv1/config_assessor.yml>*
-
-..  image:: ../img/Assessor.png
-
-Like Tuners, users can either use built-in Assessors, or customize an Assessor on their own. Please refer to the following tutorials for detail:
-
-..  toctree::
-    :maxdepth: 2
-
-    Builtin Assessors <builtin_assessor>
-    Customized Assessors <Assessor/CustomizeAssessor>