Bootstrapping tutorials in documentation (#4522)

docs/en_US/Tutorial/QuickStart.rst → docs/source/Tutorial/QuickStart.rst

@@ -228,7 +228,6 @@ Wait for the message ``INFO: Successfully started experiment!`` in the command l
 
 If you prepared ``trial``\ , ``search space``\ , and ``config`` according to the above steps and successfully created an NNI job, NNI will automatically tune the optimal hyper-parameters and run different hyper-parameter sets for each trial according to the defined search space. You can see its progress through the WebUI clearly.
 
-
 Step 5: View the Experiment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -246,7 +245,6 @@ View Overview Page
 
 Information about this experiment will be shown in the WebUI, including the experiment profile and search space message. NNI also supports downloading this information and the parameters through the **Experiment summary** button.
 
-
 .. image:: ../../img/webui-img/full-oview.png
    :target: ../../img/webui-img/full-oview.png
    :alt: overview
@@ -257,7 +255,6 @@ View Trials Detail Page
 
 You could see the best trial metrics and hyper-parameter graph in this page. And the table content includes more columns when you click the button ``Add/Remove columns``.
 
-
 .. image:: ../../img/webui-img/full-detail.png
    :target: ../../img/webui-img/full-detail.png
    :alt: detail
@@ -286,4 +283,3 @@ Related Topic
 * `How to use Annotation? <AnnotationSpec.rst>`__
 * `How to use the command line tool nnictl? <Nnictl.rst>`__
 * `How to launch Tensorboard on WebUI? <Tensorboard.rst>`__
-

docs/zh_CN/Tutorial/QuickStart.rst → docs/source/Tutorial/QuickStart_zh.rst

-.. df9198c942071002424be17beef23cf5
+.. de9c05c0da9751f920113d0b169494a2
 
 快速入门
 ==========
@@ -89,7 +89,7 @@ NNI 用来帮助超参调优。它的流程如下：
 
 
 第一步：修改 ``Trial`` 代码
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 修改 ``Trial`` 代码来从 NNI 获取超参，并向 NNI 报告训练结果。
 
@@ -122,7 +122,7 @@ NNI 用来帮助超参调优。它的流程如下：
 
 
 第二步：定义搜索空间
-^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 编写 YAML 格式的 **搜索空间** 文件，包括所有需要搜索的超参的 **名称** 和 **分布** （离散和连续值均可）。
 
@@ -148,7 +148,7 @@ NNI 用来帮助超参调优。它的流程如下：
 
 
 第三步：配置 experiment
-^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 除了在第二步中定义的搜索空间，还需要定义 YAML 格式的 **配置** 文件，声明 experiment 的关键信息，例如 Trail 文件，调优算法，最大 Trial 运行次数和最大持续时间等。
 
@@ -178,7 +178,7 @@ Experiment 的配置文件可以参考 `文档 <../reference/experiment_config.r
 
 
 第四步：运行 experiment
-^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Linux 和 macOS
 **************
@@ -231,7 +231,7 @@ Windows
 如果根据上述步骤准备好了相应 ``Trial`` ， **搜索空间** 和 **配置** ，并成功创建的 NNI 任务。NNI 会自动开始通过配置的搜索空间来运行不同的超参集合，搜索最好的超参。 通过 Web 界面可看到 NNI 的进度。
 
 第五步：查看 experiment
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 启动 experiment 后，可以在命令行界面找到如下的 **Web 界面地址** ：
 
@@ -253,7 +253,7 @@ Experiment 相关信息会显示在界面上，包括配置和搜索空间等。
 
 
 查看 Trial 详情页面
-***********************
+**********************************
 
 可以在此页面中看到最佳的 ``Trial`` 指标和超参数图。 您可以点击 ``Add/Remove columns`` 按钮向表格中添加更多列。
 
@@ -263,7 +263,7 @@ Experiment 相关信息会显示在界面上，包括配置和搜索空间等。
 
 
 查看 experiment 管理页面
-***********************
+**********************************
 
 ``All experiments`` 页面可以查看计算机上的所有实验。 
 

docs/zh_CN/Tutorial/WebUI.rst → docs/source/Tutorial/WebUI_zh.rst

@@ -272,7 +272,6 @@ Trial 在训练过程中可能有大量中间结果。 为了更清楚的理解
 * ``Tensorboard`` 请参考 `此文档 <Tensorboard.rst>`__。
 
 
-
 * 可使用 ``Copy as python`` 按钮来拷贝 Trial 的参数。
 
 

docs/en_US/Tuner/BuiltinTuner.rst → docs/source/builtin_tuner.rst

-HyperParameter Tuning with NNI Built-in Tuners
-==============================================
+Builtin-Tuners
+==============
 
-To fit a machine/deep learning model into different tasks/problems, hyperparameters always need to be tuned. Automating the process of hyperparaeter tuning always requires a good tuning algorithm. NNI has provided state-of-the-art tuning algorithms as part of our built-in tuners and makes them easy to use. Below is the brief summary of NNI's current built-in tuners:
+NNI provides an easy way to adopt an approach to set up parameter tuning algorithms, we call them **Tuner**.
 
-Note: Click the **Tuner's name** to get the Tuner's installation requirements, suggested scenario, and an example configuration. A link for a detailed description of each algorithm is located at the end of the suggested scenario for each tuner. Here is an `article <../CommunitySharings/HpoComparison.rst>`__ comparing different Tuners on several problems.
+Tuner receives metrics from `Trial` to evaluate the performance of a specific parameters/architecture configuration. Tuner sends the next hyper-parameter or architecture configuration to Trial.
 
-Currently, we support the following algorithms:
+The following table briefly describes the built-in tuners provided by NNI. Click the **Tuner's name** to get the Tuner's installation requirements, suggested scenario, and an example configuration. A link for a detailed description of each algorithm is located at the end of the suggested scenario for each tuner. Here is an `article <../CommunitySharings/HpoComparison.rst>`__ comparing different Tuners on several problems.
 
 .. list-table::
    :header-rows: 1
@@ -17,84 +17,58 @@ Currently, we support the following algorithms:
    * - `TPE <./TpeTuner.rst>`__
      - The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
 
-       TPE, as a black-box optimization, can be used in various scenarios and shows good performance in general. Especially when you have limited computation resources and can only try a small number of trials. From a large amount of experiments, we found that TPE is far better than Random Search.
-
    * - `Random Search <./RandomTuner.rst>`__
      - In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters. `Reference Paper <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__
 
-       Random search is suggested when each trial does not take very long (e.g., each trial can be completed very quickly, or early stopped by the assessor), and you have enough computational resources. It's also useful if you want to uniformly explore the search space. Random Search can be considered a baseline search algorithm.
-
    * - `Anneal <./AnnealTuner.rst>`__
      - This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
 
-       Anneal is suggested when each trial does not take very long and you have enough computation resources (very similar to Random Search). It's also useful when the variables in the search space can be sample from some prior distribution.
-
    * - `Naïve Evolution <./EvolutionTuner.rst>`__
      - Naïve Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naïve Evolution requires many trials to work, but it's very simple and easy to expand new features. `Reference paper <https://arxiv.org/pdf/1703.01041.pdf>`__
 
-       Its computational resource requirements are relatively high. Specifically, it requires a large initial population to avoid falling into a local optimum. If your trial is short or leverages assessor, this tuner is a good choice. It is also suggested when your trial code supports weight transfer; that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training process.
-
    * - `SMAC <./SmacTuner.rst>`__
-     - SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo. Notice, SMAC needs to be installed by ``pip install nni[SMAC]`` command. `Reference Paper, <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ `GitHub Repo <https://github.com/automl/SMAC3>`__
-
-       **Please note that SMAC doesn't support running on Windows currently**. For the specific reason, please refer to this `GitHub issue <https://github.com/automl/SMAC3/issues/483>`__.
+     - SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo.
 
-       Similar to TPE, SMAC is also a black-box tuner that can be tried in various scenarios and is suggested when computational resources are limited. It is optimized for discrete hyperparameters, thus, it's suggested when most of your hyperparameters are discrete.
+       Notice, SMAC needs to be installed by ``pip install nni[SMAC]`` command. `Reference Paper, <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ `GitHub Repo <https://github.com/automl/SMAC3>`__
 
    * - `Batch tuner <./BatchTuner.rst>`__
      - Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.
 
-       If the configurations you want to try have been decided beforehand, you can list them in search space file (using ``choice``) and run them using batch tuner.
-       `Detailed Description <./BatchTuner.rst>`__
-
    * - `Grid Search <./GridsearchTuner.rst>`__
      - Grid Search performs an exhaustive searching through the search space.
 
-       This is suggested when the search space is small. It's suggested when it is feasible to exhaustively sweep the whole search space.
-
    * - `Hyperband <./HyperbandAdvisor.rst>`__
      - Hyperband tries to use limited resources to explore as many configurations as possible and returns the most promising ones as a final result. The basic idea is to generate many configurations and run them for a small number of trials. The half least-promising configurations are thrown out, the remaining are further trained along with a selection of new configurations. The size of these populations is sensitive to resource constraints (e.g. allotted search time). `Reference Paper <https://arxiv.org/pdf/1603.06560.pdf>`__
 
-       This is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. For example, when models that are more accurate early on in training are also more accurate later on.
-
-   * - `Network Morphism <./NetworkmorphismTuner.rst>`__
-     - Network Morphism provides functions to automatically search for deep learning architectures. It generates child networks that inherit the knowledge from their parent network which it is a morph from. This includes changes in depth, width, and skip-connections. Next, it estimates the value of a child network using historic architecture and metric pairs. Then it selects the most promising one to train. `Reference Paper <https://arxiv.org/abs/1806.10282>`__
-
-       This is suggested when you want to apply deep learning methods to your task but you have no idea how to choose or design a network. You may modify this :githublink:`example <examples/trials/network_morphism/cifar10/cifar10_keras.py>` to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate, or optimizer. Currently, this tuner only supports the computer vision domain.
-
    * - `Metis Tuner <./MetisTuner.rst>`__
      - Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. `Reference Paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__
 
-       Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on subsequent trials. Here is an :githublink:`example <examples/trials/auto-gbdt/search_space_metis.json>` on the use of Metis. Users only need to send the final result, such as ``accuracy``, to the tuner by calling the NNI SDK.
-
-       Note that the only acceptable types of search space types are ``quniform``, ``uniform``, ``randint``, and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
-
    * - `BOHB <./BohbAdvisor.rst>`__
      - BOHB is a follow-up work to Hyperband. It targets the weakness of Hyperband that new configurations are generated randomly without leveraging finished trials. For the name BOHB, HB means Hyperband, BO means Bayesian Optimization. BOHB leverages finished trials by building multiple TPE models, a proportion of new configurations are generated through these models. `Reference Paper <https://arxiv.org/abs/1807.01774>`__
 
-       Similar to Hyperband, BOHB is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. In this case, it may converge to a better configuration than Hyperband due to its usage of Bayesian optimization.
-
    * - `GP Tuner <./GPTuner.rst>`__
      - Gaussian Process Tuner is a sequential model-based optimization (SMBO) approach with Gaussian Process as the surrogate. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__, `Github Repo <https://github.com/fmfn/BayesianOptimization>`__
 
-       Note that the only acceptable types within the search space are ``randint``, ``uniform``, ``quniform``, ``loguniform``, ``qloguniform``, and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
-
-       As a strategy in a Sequential Model-based Global Optimization (SMBO) algorithm, GP Tuner uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve and common tools can be employed to solve it. Therefore, GP Tuner is most adequate for situations where the function to be optimized is very expensive to evaluate. GP can be used when computational resources are limited. However, GP Tuner has a computational cost that grows at *O(N^3)* due to the requirement of inverting the Gram matrix, so it's not suitable when lots of trials are needed.
-
    * - `PBT Tuner <./PBTTuner.rst>`__
      - PBT Tuner is a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. `Reference Paper <https://arxiv.org/abs/1711.09846v1>`__
 
-       Population Based Training (PBT) bridges and extends parallel search methods and sequential optimization methods. It requires relatively small computation resource, by inheriting weights from currently good-performing ones to explore better ones periodically. With PBTTuner, users finally get a trained model, rather than a configuration that could reproduce the trained model by training the model from scratch. This is because model weights are inherited periodically through the whole search process. PBT can also be seen as a training approach. If you don't need to get a specific configuration, but just expect a good model, PBTTuner is a good choice.
-
    * - `DNGO Tuner <./DngoTuner.rst>`__
      - Use of neural networks as an alternative to GPs to model distributions over functions in bayesian optimization.
 
-       Applicable to large scale hyperparameter optimization. Bayesian optimization that rapidly finds competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.
-
-Usage of Built-in Tuners
-------------------------
-
-Using a built-in tuner provided by the NNI SDK requires one to declare the  **name** and **classArgs** in the ``config.yml`` file.
-Click tuners' name in above table to see their specification.
-
-Note: Some built-in tuners have dependencies that need to be installed using ``pip install nni[<tuner>]``, like SMAC's dependencies can be installed using ``pip install nni[SMAC]``.
+..  toctree::
+    :maxdepth: 1
+
+    TPE <Tuner/TpeTuner>
+    Random Search <Tuner/RandomTuner>
+    Anneal <Tuner/AnnealTuner>
+    Naive Evolution <Tuner/EvolutionTuner>
+    SMAC <Tuner/SmacTuner>
+    Metis Tuner <Tuner/MetisTuner>
+    Batch Tuner <Tuner/BatchTuner>
+    Grid Search <Tuner/GridsearchTuner>
+    GP Tuner <Tuner/GPTuner>
+    Network Morphism <Tuner/NetworkmorphismTuner>
+    Hyperband <Tuner/HyperbandAdvisor>
+    BOHB <Tuner/BohbAdvisor>
+    PBT Tuner <Tuner/PBTTuner>
+    DNGO Tuner <Tuner/DngoTuner>

docs/zh_CN/builtin_tuner.rst → docs/source/builtin_tuner_zh.rst

-.. 1ff18ebada0efec66cd793f1a000f3fe
+.. 10b9097fcfec13f98bb6914b40bd0337
 
 内置 Tuner
 ==========

docs/en_US/conf.py → docs/source/conf.py

@@ -16,6 +16,7 @@ import os
 import subprocess
 import sys
 sys.path.insert(0, os.path.abspath('../..'))
+sys.path.insert(0, os.path.abspath('../extension'))
 
 
 # -- Project information ---------------------------------------------------
@@ -41,20 +42,46 @@ release = 'v2.6'
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'sphinx_gallery.gen_gallery',
     'sphinx.ext.autodoc',
     'sphinx.ext.mathjax',
     'sphinxarg.ext',
     'sphinx.ext.napoleon',
     'sphinx.ext.viewcode',
     'sphinx.ext.intersphinx',
-    'nbsphinx',
+    # 'nbsphinx',  # nbsphinx has conflicts with sphinx-gallery.
     'sphinx.ext.extlinks',
     'IPython.sphinxext.ipython_console_highlighting',
+
+    # Custom extensions in extension/ folder.
+    'inplace_translation',
+    'cardlinkitem',
+    'patch_docutils',
 ]
 
 # Add mock modules
 autodoc_mock_imports = ['apex', 'nni_node', 'tensorrt', 'pycuda', 'nn_meter']
 
+# Sphinx gallery examples
+sphinx_gallery_conf = {
+    'examples_dirs': '../../examples/tutorials',   # path to your example scripts
+    'gallery_dirs': 'tutorials',                   # path to where to save gallery generated output
+
+    # Control ignored python files.
+    'ignore_pattern': r'__init__\.py|/scripts/',
+
+    # This is `/plot` by default. Only files starting with `/plot` will be executed.
+    # All files should be executed in our case.
+    'filename_pattern': r'.*',
+
+    # Disabling download button of all scripts
+    'download_all_examples': False,
+
+    # Change default thumbnail
+    # Working directory is strange, needs full path.
+    'default_thumb_file': os.path.join(os.path.dirname(__file__), '../img/thumbnails/nni_icon_blue.png'),
+}
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['../templates']
 
@@ -75,7 +102,17 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md', '**.ipynb_checkpoints']
+exclude_patterns = [
+    '_build',
+    'Thumbs.db',
+    '.DS_Store',
+    'Release_v1.0.md',
+    '**.ipynb_checkpoints',
+    # Exclude translations. They will be added back via replacement later if language is set.
+    '**_zh.rst',
+    # Exclude generated tutorials index
+    'tutorials/index.rst',
+]
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
@@ -161,13 +198,22 @@ html_title = 'Neural Network Intelligence'
 html_css_files = [
     'css/material_custom.css',
     'css/material_dropdown.css',
+    'css/sphinx_gallery.css',
     'css/index_page.css',
 ]
 html_js_files = [
     'js/version.js',
     'js/github.js',
+    'js/sphinx_gallery.js',
 ]
 
+# HTML context that can be used in jinja templates
+git_commit_id = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
+
+html_context = {
+    'git_commit_id': git_commit_id
+}
+
 # -- Options for HTMLHelp output ---------------------------------------------
 
 # Output file base name for HTML help builder.
@@ -244,8 +290,6 @@ epub_exclude_files = ['search.html']
 
 # external links (for github code)
 # Reference the code via :githublink:`path/to/your/example/code.py`
-git_commit_id = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
-
 extlinks = {
     'githublink': ('https://github.com/microsoft/nni/blob/' + git_commit_id + '/%s', 'Github link: ')
 }