Merge master into dev-retiarii (#3178)

docs/tools/patches/1.txt

+   * - GP Tuner
+     - :raw-html:`✓`
+     - 
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - 
+     - 
+     - 
+%%%%%%
+   * - GP Tuner
+     - :raw-html:`✓`
+     - 
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - :raw-html:`✓`
+     - 
+     - 
+     - 
+     - 

docs/tools/patches/10.txt

+An SSH server needs a port; you need to expose Docker's SSH port to NNI as the connection port. For example, if you set your container's SSH port as  **``A``** \ , you should map the container's port ** ``A``**  to your remote host machine's other port ** ``B``** \ , NNI will connect port ** ``B``**  as an SSH port, and your host machine will map the connection from port ** ``B``**  to port ** ``A``**  then NNI could connect to your Docker container.
+%%%%%%
+An SSH server needs a port; you need to expose Docker's SSH port to NNI as the connection port. For example, if you set your container's SSH port as ``A``, you should map the container's port ``A`` to your remote host machine's other port ``B``, NNI will connect port ``B`` as an SSH port, and your host machine will map the connection from port ``B`` to port ``A`` then NNI could connect to your Docker container.

docs/tools/patches/11.txt

+If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
+%%%%%%
+If the id ends with \*, nnictl will stop all experiments whose ids matchs the regular.

docs/tools/patches/12.txt

+..
+
+   make: *** [install-XXX] Segmentation fault (core dumped)
+%%%%%%
+.. code-block:: text
+
+   make: *** [install-XXX] Segmentation fault (core dumped)

docs/tools/patches/13.txt

+  Click ``Submit job`` button in web portal.
+%%%%%%
+Click ``Submit job`` button in web portal.

docs/tools/patches/14.txt

+:raw-html:`<div >
+    <img src="https://github.com/microsoft/Cream/blob/main/demo/intro.jpg" width="800"/>
+</div>`
+%%%%%%
+:raw-html:`<div ><img src="https://github.com/microsoft/Cream/blob/main/demo/intro.jpg" width="800"/></div>`
\ No newline at end of file

docs/tools/patches/15.txt

+.. list-table::
+   :header-rows: 1
+
+%%%%%%
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+

docs/tools/patches/16.txt

+.. code-block:: bash
+
+   1.1 Declare NNI API
+       Include `import nni` in your trial code to use NNI APIs.
+%%%%%%
+..
+
+   1.1 Declare NNI API
+       Include `import nni` in your trial code to use NNI APIs.

docs/tools/patches/17.txt

+.. code-block:: bash
+
+   from nni.compression.pytorch.utils.counter import count_flops_params
+%%%%%%
+.. code-block:: python
+
+   from nni.compression.pytorch.utils.counter import count_flops_params

docs/tools/patches/18.txt

+.. code-block:: bash
+
+   NNI's official image msranni/nni does not support SSH servers for the time being; you should build your own Docker image with an SSH configuration or use other images as a remote server.
+%%%%%%
+.. code-block:: text
+
+   NNI's official image msranni/nni does not support SSH servers for the time being; you should build your own Docker image with an SSH configuration or use other images as a remote server.

docs/tools/patches/19.txt

+Code Styles & Naming Conventions
+--------------------------------
+
+
+* We follow `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as ``flake8`` or ``pylint``
+* We also follow `NumPy Docstring Style <https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy>`__ for Python Docstring Conventions. During the `documentation building <Contributing.rst#documentation>`__\ , we use `sphinx.ext.napoleon <https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html>`__ to generate Python API documentation from Docstring.
+* For docstrings, please refer to `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`__ and `pandas docstring guide <https://python-sprints.github.io/pandas/guide/pandas_docstring.html>`__
+
+  * For function docstring, **description** , **Parameters**\ , and** Returns**\ /** Yields** are mandatory.
+  * For class docstring, **description**\ ,** Attributes** are mandatory.
+  * For docstring to describe ``dict``\ , which is commonly used in our hyper-param format description, please refer to [RiboKit : Doc Standards
+
+    * Internal Guideline on Writing Standards](https://ribokit.github.io/docs/text/)
+
+Documentation
+-------------
+
+Our documentation is built with :githublink:`sphinx <docs>`.
+
+
+* 
+  Before submitting the documentation change, please **build homepage locally**\ : ``cd docs/en_US && make html``\ , then you can see all the built documentation webpage under the folder ``docs/en_US/_build/html``. It's also highly recommended taking care of** every WARNING** during the build, which is very likely the signal of a** deadlink** and other annoying issues.
+
+* 
+  For links, please consider using **relative paths** first. However, if the documentation is written in Markdown format, and:
+
+
+  * It's an image link which needs to be formatted with embedded html grammar, please use global URL like ``https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png``\ , which can be automatically generated by dragging picture onto `Github Issue <https://github.com/Microsoft/nni/issues/new>`__ Box.
+  * It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at ``https://github.com/Microsoft/nni/tree/v1.9/`` (\ :githublink:`mnist.py <examples/trials/mnist-tfv1/mnist.py>` for example).
+%%%%%%
+Code Styles & Naming Conventions
+--------------------------------
+
+* We follow `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as ``flake8`` or ``pylint``
+* We also follow `NumPy Docstring Style <https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy>`__ for Python Docstring Conventions. During the `documentation building <Contributing.rst#documentation>`__\ , we use `sphinx.ext.napoleon <https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html>`__ to generate Python API documentation from Docstring.
+* For docstrings, please refer to `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`__ and `pandas docstring guide <https://python-sprints.github.io/pandas/guide/pandas_docstring.html>`__
+
+  * For function docstring, **description**, **Parameters**, and **Returns** **Yields** are mandatory.
+  * For class docstring, **description**, **Attributes** are mandatory.
+  * For docstring to describe ``dict``, which is commonly used in our hyper-param format description, please refer to RiboKit Doc Standards
+
+    * `Internal Guideline on Writing Standards <https://ribokit.github.io/docs/text/>`__
+
+Documentation
+-------------
+
+Our documentation is built with :githublink:`sphinx <docs>`.
+
+* Before submitting the documentation change, please **build homepage locally**: ``cd docs/en_US && make html``, then you can see all the built documentation webpage under the folder ``docs/en_US/_build/html``. It's also highly recommended taking care of **every WARNING** during the build, which is very likely the signal of a **deadlink** and other annoying issues.
+
+* 
+  For links, please consider using **relative paths** first. However, if the documentation is written in Markdown format, and:
+
+
+  * It's an image link which needs to be formatted with embedded html grammar, please use global URL like ``https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png``, which can be automatically generated by dragging picture onto `Github Issue <https://github.com/Microsoft/nni/issues/new>`__ Box.
+  * It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at ``https://github.com/Microsoft/nni/tree/v1.9/`` (:githublink:`mnist.py <examples/trials/mnist-tfv1/mnist.py>` for example).

docs/tools/patches/2.txt

+   * - 
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - Ubuntu 16.04 or above
+   * - **CPU**
+     - Intel® Core™ i5 or AMD Phenom™ II X3 or better
+     - Intel® Core™ i3 or AMD Phenom™ X3 8650
+   * - **GPU**
+     - NVIDIA® GeForce® GTX 660 or better
+     - NVIDIA® GeForce® GTX 460
+   * - **Memory**
+     - 6 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 30 GB available hare drive space
+   * - **Internet**
+     - Boardband internet connection
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+%%%%%%
+   * - 
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - Ubuntu 16.04 or above
+     -
+   * - **CPU**
+     - Intel® Core™ i5 or AMD Phenom™ II X3 or better
+     - Intel® Core™ i3 or AMD Phenom™ X3 8650
+   * - **GPU**
+     - NVIDIA® GeForce® GTX 660 or better
+     - NVIDIA® GeForce® GTX 460
+   * - **Memory**
+     - 6 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 30 GB available hare drive space
+     -
+   * - **Internet**
+     - Boardband internet connection
+     -
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+     -

docs/tools/patches/20.txt

+..
+
+   1.1 Declare NNI API
+       Include `import nni` in your trial code to use NNI APIs.
+
+   1.2 Get predefined parameters
+       Use the following code snippet:
+
+           RECEIVED_PARAMS = nni.get_next_parameter()
+
+       to get hyper-parameters' values assigned by tuner. `RECEIVED_PARAMS` is an object, for example:
+
+           {"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}
+
+   1.3 Report NNI results
+       Use the API:
+
+           `nni.report_intermediate_result(accuracy)`
+
+       to send `accuracy` to assessor.
+
+       Use the API:
+
+           `nni.report_final_result(accuracy)`
+
+       to send `accuracy` to tuner.
+%%%%%%
+* Declare NNI API: include ``import nni`` in your trial code to use NNI APIs.
+* Get predefined parameters
+
+Use the following code snippet:
+
+.. code-block:: python
+
+   RECEIVED_PARAMS = nni.get_next_parameter()
+
+to get hyper-parameters' values assigned by tuner. ``RECEIVED_PARAMS`` is an object, for example:
+
+.. code-block:: json
+
+   {"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}
+
+* Report NNI results: Use the API: ``nni.report_intermediate_result(accuracy)`` to send ``accuracy`` to assessor.
+  Use the API: ``nni.report_final_result(accuracy)`` to send `accuracy` to tuner.

docs/tools/patches/3.txt

+   * - 
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - macOS 10.14.1 or above
+   * - **CPU**
+     - Intel® Core™ i7-4770 or better
+     - Intel® Core™ i5-760 or better
+   * - **GPU**
+     - AMD Radeon™ R9 M395X or better
+     - NVIDIA® GeForce® GT 750M or AMD Radeon™ R9 M290 or better
+   * - **Memory**
+     - 8 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 70GB available space SSD
+     - 70GB available space 7200 RPM HDD
+   * - **Internet**
+     - Boardband internet connection
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+%%%%%%
+   * -
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - macOS 10.14.1 or above
+     - 
+   * - **CPU**
+     - Intel® Core™ i7-4770 or better
+     - Intel® Core™ i5-760 or better
+   * - **GPU**
+     - AMD Radeon™ R9 M395X or better
+     - NVIDIA® GeForce® GT 750M or AMD Radeon™ R9 M290 or better
+   * - **Memory**
+     - 8 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 70GB available space SSD
+     - 70GB available space 7200 RPM HDD
+   * - **Internet**
+     - Boardband internet connection
+     - 
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+     - 

docs/tools/patches/4.txt

+   * - 
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - Windows 10 1809 or above
+   * - **CPU**
+     - Intel® Core™ i5 or AMD Phenom™ II X3 or better
+     - Intel® Core™ i3 or AMD Phenom™ X3 8650
+   * - **GPU**
+     - NVIDIA® GeForce® GTX 660 or better
+     - NVIDIA® GeForce® GTX 460
+   * - **Memory**
+     - 6 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 30 GB available hare drive space
+   * - **Internet**
+     - Boardband internet connection
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+%%%%%%
+   * -
+     - Recommended
+     - Minimum
+   * - **Operating System**
+     - Windows 10 1809 or above
+     - 
+   * - **CPU**
+     - Intel® Core™ i5 or AMD Phenom™ II X3 or better
+     - Intel® Core™ i3 or AMD Phenom™ X3 8650
+   * - **GPU**
+     - NVIDIA® GeForce® GTX 660 or better
+     - NVIDIA® GeForce® GTX 460
+   * - **Memory**
+     - 6 GB RAM
+     - 4 GB RAM
+   * - **Storage**
+     - 30 GB available hare drive space
+     - 
+   * - **Internet**
+     - Boardband internet connection
+     - 
+   * - **Resolution**
+     - 1024 x 768 minimum display resolution
+     - 

docs/tools/patches/5.txt

+   * - 
+     - s=4
+     - s=3
+     - s=2
+     - s=1
+     - s=0
+   * - i
+     - n r
+     - n r
+     - n r
+     - n r
+     - n r
+   * - 0
+     - 81 1
+     - 27 3
+     - 9 9
+     - 6 27
+     - 5 81
+   * - 1
+     - 27 3
+     - 9 9
+     - 3 27
+     - 2 81
+     - 
+   * - 2
+     - 9 9
+     - 3 27
+     - 1 81
+     - 
+     - 
+   * - 3
+     - 3 27
+     - 1 81
+     - 
+     - 
+     - 
+   * - 4
+     - 1 81
+     - 
+     - 
+     - 
+%%%%%%
+   * -
+     - s=4
+     - s=3
+     - s=2
+     - s=1
+     - s=0
+   * - i
+     - n r
+     - n r
+     - n r
+     - n r
+     - n r
+   * - 0
+     - 81 1
+     - 27 3
+     - 9 9
+     - 6 27
+     - 5 81
+   * - 1
+     - 27 3
+     - 9 9
+     - 3 27
+     - 2 81
+     -
+   * - 2
+     - 9 9
+     - 3 27
+     - 1 81
+     -
+     -
+   * - 3
+     - 3 27
+     - 1 81
+     -
+     -
+     -
+   * - 4
+     - 1 81
+     -
+     -
+     -
+     -

docs/tools/patches/6.txt

+*Please refer to `here <https://nni.readthedocs.io/en/latest/sdk_reference.html>`__ for more APIs (e.g., ``nni.get_sequence_id()``\ ) provided by NNI.
+%%%%%%
+*Please refer to `here <https://nni.readthedocs.io/en/latest/sdk_reference.html>`__ for more APIs (e.g., ``nni.get_sequence_id()``\ ) provided by NNI.*

docs/tools/patches/7.txt

+   #. For each filter 
+      .. image:: http://latex.codecogs.com/gif.latex?F_{i,j}
+         :target: http://latex.codecogs.com/gif.latex?F_{i,j}
+         :alt: 
+      , calculate the sum of its absolute kernel weights
+      .. image:: http://latex.codecogs.com/gif.latex?s_j=\sum_{l=1}^{n_i}\sum|K_l|
+         :target: http://latex.codecogs.com/gif.latex?s_j=\sum_{l=1}^{n_i}\sum|K_l|
+         :alt: 
+
+   #. Sort the filters by 
+      .. image:: http://latex.codecogs.com/gif.latex?s_j
+         :target: http://latex.codecogs.com/gif.latex?s_j
+         :alt: 
+      .
+   #. Prune 
+      .. image:: http://latex.codecogs.com/gif.latex?m
+         :target: http://latex.codecogs.com/gif.latex?m
+         :alt: 
+       filters with the smallest sum values and their corresponding feature maps. The
+        kernels in the next convolutional layer corresponding to the pruned feature maps are also
+      .. code-block:: bash
+
+         removed.
+
+   #. A new kernel matrix is created for both the 
+      .. image:: http://latex.codecogs.com/gif.latex?i
+         :target: http://latex.codecogs.com/gif.latex?i
+         :alt: 
+      th and 
+      .. image:: http://latex.codecogs.com/gif.latex?i+1
+         :target: http://latex.codecogs.com/gif.latex?i+1
+         :alt: 
+      th layers, and the remaining kernel
+        weights are copied to the new model.
+%%%%%%
+   #. For each filter :math:`F_{i,j}`, calculate the sum of its absolute kernel weights :math:`s_j=\sum_{l=1}^{n_i}\sum|K_l|`.
+
+   #. Sort the filters by :math:`s_j`.
+
+   #. Prune :math:`m` filters with the smallest sum values and their corresponding feature maps. The
+      kernels in the next convolutional layer corresponding to the pruned feature maps are also removed.
+
+   #. A new kernel matrix is created for both the :math:`i`-th and :math:`i+1`-th layers, and the remaining kernel
+      weights are copied to the new model.

docs/tools/patches/8.txt

+#. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this `guideline <https://kubernetes.io/docs/setup/>`__ to set up Kubernetes
+#. Prepare a **kubeconfig** file, which will be used by NNI to interact with your Kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the** KUBECONFIG** environment variable. Refer this `guideline <https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig>`__ to learn more about kubeconfig.
+#. If your NNI trial job needs GPU resource, you should follow this `guideline <https://github.com/NVIDIA/k8s-device-plugin>`__ to configure **Nvidia device plugin for Kubernetes**.
+#. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in ``root_squash option``\ , otherwise permission issue may raise when NNI copies files to NFS. Refer this `page <https://linux.die.net/man/5/exports>`__ to learn what root_squash option is), or** Azure File Storage**.
+#. 
+   Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
+
+   .. code-block:: bash
+
+       apt-get install nfs-common
+
+#. 
+   Install **NNI**\ , follow the install guide `here <../Tutorial/QuickStart.rst>`__.
+%%%%%%
+#. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this `guideline <https://kubernetes.io/docs/setup/>`__ to set up Kubernetes
+#. Prepare a **kubeconfig** file, which will be used by NNI to interact with your Kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the**KUBECONFIG** environment variable. Refer this `guideline <https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig>`__ to learn more about kubeconfig.
+#. If your NNI trial job needs GPU resource, you should follow this `guideline <https://github.com/NVIDIA/k8s-device-plugin>`__ to configure **Nvidia device plugin for Kubernetes**.
+#. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in ``root_squash option``\ , otherwise permission issue may raise when NNI copies files to NFS. Refer this `page <https://linux.die.net/man/5/exports>`__ to learn what root_squash option is), or **Azure File Storage**.
+#. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
+
+.. code-block:: bash
+
+    apt-get install nfs-common
+
+#. Install **NNI**\ , follow the install guide `here <../Tutorial/QuickStart>`__.

docs/tools/patches/9.txt

+#. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this `guideline <https://kubernetes.io/docs/setup/>`__ to set up Kubernetes
+#. Download, set up, and deploy **Kubeflow** to your Kubernetes cluster. Follow this `guideline <https://www.kubeflow.org/docs/started/getting-started/>`__ to setup Kubeflow.
+#. Prepare a **kubeconfig** file, which will be used by NNI to interact with your Kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the** KUBECONFIG** environment variable. Refer this `guideline <https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig>`__ to learn more about kubeconfig.
+#. If your NNI trial job needs GPU resource, you should follow this `guideline <https://github.com/NVIDIA/k8s-device-plugin>`__ to configure **Nvidia device plugin for Kubernetes**.
+#. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in ``root_squash option``\ , otherwise permission issue may raise when NNI copy files to NFS. Refer this `page <https://linux.die.net/man/5/exports>`__ to learn what root_squash option is), or** Azure File Storage**.
+#. 
+   Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
+
+   .. code-block:: bash
+
+       apt-get install nfs-common
+
+#. 
+   Install **NNI**\ , follow the install guide `here <../Tutorial/QuickStart.rst>`__.
+%%%%%%
+#. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this `guideline <https://kubernetes.io/docs/setup/>`__ to set up Kubernetes
+#. Download, set up, and deploy **Kubeflow** to your Kubernetes cluster. Follow this `guideline <https://www.kubeflow.org/docs/started/getting-started/>`__ to setup Kubeflow.
+#. Prepare a **kubeconfig** file, which will be used by NNI to interact with your Kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the**KUBECONFIG** environment variable. Refer this `guideline <https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig>`__ to learn more about kubeconfig.
+#. If your NNI trial job needs GPU resource, you should follow this `guideline <https://github.com/NVIDIA/k8s-device-plugin>`__ to configure **Nvidia device plugin for Kubernetes**.
+#. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in ``root_squash option``\ , otherwise permission issue may raise when NNI copy files to NFS. Refer this `page <https://linux.die.net/man/5/exports>`__ to learn what root_squash option is), or**Azure File Storage**.
+#. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
+
+.. code-block:: bash
+
+    apt-get install nfs-common
+
+#. Install **NNI**\ , follow the install guide `here <../Tutorial/QuickStart>`__.