Unverified Commit 5bf5e46c authored by Yuge Zhang's avatar Yuge Zhang Committed by GitHub
Browse files

Port markdown docs to rst and introduce "githublink" (#3107)

parent dbb2434f
Python API Reference of Auto Tune
=================================
.. contents::
Trial
-----
.. autofunction:: nni.get_next_parameter
.. autofunction:: nni.get_current_parameter
.. autofunction:: nni.report_intermediate_result
.. autofunction:: nni.report_final_result
.. autofunction:: nni.get_experiment_id
.. autofunction:: nni.get_trial_id
.. autofunction:: nni.get_sequence_id
Tuner
-----
.. autoclass:: nni.tuner.Tuner
:members:
.. autoclass:: nni.algorithms.hpo.hyperopt_tuner.hyperopt_tuner.HyperoptTuner
:members:
.. autoclass:: nni.algorithms.hpo.evolution_tuner.evolution_tuner.EvolutionTuner
:members:
.. autoclass:: nni.algorithms.hpo.smac_tuner.SMACTuner
:members:
.. autoclass:: nni.algorithms.hpo.gridsearch_tuner.GridSearchTuner
:members:
.. autoclass:: nni.algorithms.hpo.networkmorphism_tuner.networkmorphism_tuner.NetworkMorphismTuner
:members:
.. autoclass:: nni.algorithms.hpo.metis_tuner.metis_tuner.MetisTuner
:members:
.. autoclass:: nni.algorithms.hpo.ppo_tuner.PPOTuner
:members:
.. autoclass:: nni.algorithms.hpo.batch_tuner.batch_tuner.BatchTuner
:members:
.. autoclass:: nni.algorithms.hpo.gp_tuner.gp_tuner.GPTuner
:members:
Assessor
--------
.. autoclass:: nni.assessor.Assessor
:members:
.. autoclass:: nni.assessor.AssessResult
:members:
.. autoclass:: nni.algorithms.hpo.curvefitting_assessor.CurvefittingAssessor
:members:
.. autoclass:: nni.algorithms.hpo.medianstop_assessor.MedianstopAssessor
:members:
Advisor
-------
.. autoclass:: nni.runtime.msg_dispatcher_base.MsgDispatcherBase
:members:
.. autoclass:: nni.algorithms.hpo.hyperband_advisor.hyperband_advisor.Hyperband
:members:
.. autoclass:: nni.algorithms.hpo.bohb_advisor.bohb_advisor.BOHB
:members:
Utilities
---------
.. autofunction:: nni.utils.merge_parameter
...@@ -12,12 +12,10 @@ ...@@ -12,12 +12,10 @@
# add these directories to sys.path here. If the directory is relative to the # add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here. # documentation root, use os.path.abspath to make it absolute, like shown here.
# #
from recommonmark.transform import AutoStructify
from recommonmark.parser import CommonMarkParser
import os import os
import subprocess
import sys import sys
sys.path.insert(0, os.path.abspath('../../src/sdk/pynni')) sys.path.insert(0, os.path.abspath('../..'))
sys.path.insert(1, os.path.abspath('../../src/sdk/pycli'))
# -- Project information --------------------------------------------------- # -- Project information ---------------------------------------------------
...@@ -43,12 +41,12 @@ release = 'v1.9' ...@@ -43,12 +41,12 @@ release = 'v1.9'
extensions = [ extensions = [
'sphinx.ext.autodoc', 'sphinx.ext.autodoc',
'sphinx.ext.mathjax', 'sphinx.ext.mathjax',
'sphinx_markdown_tables',
'sphinxarg.ext', 'sphinxarg.ext',
'sphinx.ext.napoleon', 'sphinx.ext.napoleon',
'sphinx.ext.viewcode', 'sphinx.ext.viewcode',
'sphinx.ext.intersphinx', 'sphinx.ext.intersphinx',
'nbsphinx', 'nbsphinx',
'sphinx.ext.extlinks',
] ]
# Add mock modules # Add mock modules
...@@ -59,12 +57,7 @@ templates_path = ['_templates'] ...@@ -59,12 +57,7 @@ templates_path = ['_templates']
# The suffix(es) of source filenames. # The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string: # You can specify multiple suffix as a list of string:
# source_suffix = ['.rst']
source_parsers = {
'.md': CommonMarkParser
}
source_suffix = ['.rst', '.md']
# The master toctree document. # The master toctree document.
master_doc = 'contents' master_doc = 'contents'
...@@ -197,12 +190,14 @@ epub_title = project ...@@ -197,12 +190,14 @@ epub_title = project
# A list of files that should not be packed into the epub file. # A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html'] epub_exclude_files = ['search.html']
# external links (for github code)
# Reference the code via :githublink:`path/to/your/example/code.py`
git_commit_id = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
extlinks = {
'githublink': ('https://github.com/microsoft/nni/blob/' + git_commit_id + '/%s', 'Github link: ')
}
# -- Extension configuration ------------------------------------------------- # -- Extension configuration -------------------------------------------------
def setup(app): def setup(app):
app.add_config_value('recommonmark_config', {
'enable_eval_rst': True,
'enable_auto_toc_tree': False,
}, True)
app.add_transform(AutoStructify)
app.add_stylesheet('css/custom.css') app.add_stylesheet('css/custom.css')
...@@ -17,7 +17,7 @@ Neural Network Intelligence ...@@ -17,7 +17,7 @@ Neural Network Intelligence
Feature Engineering <feature_engineering> Feature Engineering <feature_engineering>
References <reference> References <reference>
Use Cases and Solutions <CommunitySharings/community_sharings> Use Cases and Solutions <CommunitySharings/community_sharings>
Research and Publications <ResearchPublications.md> Research and Publications <ResearchPublications>
FAQ <Tutorial/FAQ> FAQ <Tutorial/FAQ>
How to Contribute <contribution> How to Contribute <contribution>
Changelog <Release> Changelog <Release>
...@@ -9,4 +9,4 @@ Advanced Features ...@@ -9,4 +9,4 @@ Advanced Features
Write a New Advisor <Tuner/CustomizeAdvisor> Write a New Advisor <Tuner/CustomizeAdvisor>
Write a New Training Service <TrainingService/HowToImplementTrainingService> Write a New Training Service <TrainingService/HowToImplementTrainingService>
Install Customized Algorithms as Builtin Tuners/Assessors/Advisors <Tutorial/InstallCustomizedAlgos> Install Customized Algorithms as Builtin Tuners/Assessors/Advisors <Tutorial/InstallCustomizedAlgos>
How to install customized tuner as a builtin tuner <Tuner/InstallCustomizedTuner.md> How to install customized tuner as a builtin tuner <Tuner/InstallCustomizedTuner>
NNI Client
==========
NNI client is a python API of ``nnictl``\ , which implements the most commonly used commands. Users can use this API to control their experiments, collect experiment results and conduct advanced analyses based on experiment results in python code directly instead of using command line. Here is an example:
.. code-block:: bash
from nni.experiment import Experiment
# create an experiment instance
exp = Experiment()
# start an experiment, then connect the instance to this experiment
# you can also use `resume_experiment`, `view_experiment` or `connect_experiment`
# only one of them should be called in one instance
exp.start_experiment('nni/examples/trials/mnist-pytorch/config.yml', port=9090)
# update the experiment's concurrency
exp.update_concurrency(3)
# get some information about the experiment
print(exp.get_experiment_status())
print(exp.get_job_statistics())
print(exp.list_trial_jobs())
# stop the experiment, then disconnect the instance from the experiment.
exp.stop_experiment()
References
----------
.. autoclass:: nni.experiment.Experiment
:members:
.. autoclass:: nni.experiment.TrialJob
:members:
.. autoclass:: nni.experiment.TrialHyperParameters
:members:
.. autoclass:: nni.experiment.TrialMetricData
:members:
.. autoclass:: nni.experiment.TrialResult
:members:
sphinx==1.8.3 sphinx==3.3.1
sphinx-argparse==0.2.5 sphinx-argparse==0.2.5
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2 sphinx-rtd-theme==0.4.2
sphinxcontrib-websupport==1.1.0 sphinxcontrib-websupport==1.1.0
recommonmark==0.5.0
pygments==2.7.1 pygments==2.7.1
hyperopt hyperopt
json_tricks json_tricks
......
import argparse
import m2r
import os
import re
import shutil
from pathlib import Path
def single_line_process(line):
if line == ' .. contents::':
return '.. contents::'
# https://github.com/sphinx-doc/sphinx/issues/3921
line = re.sub(r'(`.*? <.*?>`)_', r'\1__', line)
# inline emphasis
line = re.sub(r'\*\*\\ (.*?)\\ \*\*', r' **\1** ', line)
line = re.sub(r'\*(.*?)\\ \*', r'*\1*', line)
line = re.sub(r'\*\*(.*?) \*\*', r'**\1** ', line)
line = re.sub(r'\\\*\\\*(.*?)\*\*', r'**\1**', line)
line = re.sub(r'\\\*\\\*(.*?)\*\*\\ ', r'**\1**', line)
line = line.replace(r'\* - `\**', r'* - `**')
line = re.sub(r'\\\* \*\*(.*?)\*\* \(\\\*\s*(.*?)\s*\*\\ \)', r'* \1 (\2)', line)
line = re.sub(r'\<(.*)\.md(\>|#)', r'<\1.rst\2', line)
line = re.sub(r'`\*\*(.*?)\*\* <#(.*?)>`__', r'`\1 <#\2>`__', line)
line = re.sub(r'\*\* (classArgs|stop|FLOPS.*?|pruned.*?|large.*?|path|preCommand|2D.*?|codeDirectory|ps|worker|Tuner|Assessor)\*\*',
r' **\1**', line)
line = line.replace('.. code-block:::: bash', '.. code-block:: bash')
line = line.replace('raw-html-m2r', 'raw-html')
line = line.replace('[toc]', '.. toctree::')
# image
line = re.sub(r'\:raw\-html\:`\<img src\=\"(.*?)\" style\=\"zoom\: ?(\d+)\%\;\" \/\>`', r'\n.. image:: \1\n :scale: \2%', line)
# special case (per line handling)
line = line.replace('Nb = |Db|', r'Nb = \|Db\|')
line = line.replace(' Here is just a small list of libraries ', '\nHere is just a small list of libraries ')
line = line.replace(' Find the data management region in job submission page.', 'Find the data management region in job submission page.')
line = line.replace('Tuner/InstallCustomizedTuner.md', 'Tuner/InstallCustomizedTuner')
line = line.replace('&#10003;', ':raw-html:`&#10003;`')
line = line.replace(' **builtinTunerName** and** classArgs**', '**builtinTunerName** and **classArgs**')
line = line.replace('`\ ``nnictl ss_gen`` <../Tutorial/Nnictl.rst>`__', '`nnictl ss_gen <../Tutorial/Nnictl.rst>`__')
line = line.replace('**Step 1. Install NNI, follow the install guide `here <../Tutorial/QuickStart.rst>`__.**',
'**Step 1. Install NNI, follow the install guide** `here <../Tutorial/QuickStart.rst>`__.')
line = line.replace('*Please refer to `here ', 'Please refer to `here ')
# line = line.replace('\* **optimize_mode** ', '* **optimize_mode** ')
if line == '~' * len(line):
line = '^' * len(line)
return line
def special_case_replace(full_text):
replace_pairs = {}
replace_pairs['PyTorch\n"""""""'] = '**PyTorch**'
replace_pairs['Search Space\n============'] = '.. role:: raw-html(raw)\n :format: html\n\nSearch Space\n============'
for file in os.listdir(Path(__file__).parent / 'patches'):
with open(Path(__file__).parent / 'patches' / file) as f:
r, s = f.read().split('%%%%%%\n')
replace_pairs[r] = s
for r, s in replace_pairs.items():
full_text = full_text.replace(r, s)
return full_text
def process_table(content):
content = content.replace('------ |', '------|')
lines = []
for line in content.split('\n'):
if line.startswith(' |'):
line = line[2:]
lines.append(line)
return '\n'.join(lines)
def process_github_link(line):
line = re.sub(r'`(\\ ``)?([^`]*?)(``)? \<(.*?)(blob|tree)/v1.9/(.*?)\>`__', r':githublink:`\2 <\6>`', line)
if 'githublink' in line:
line = re.sub(r'\*Example: (.*)\*', r'*Example:* \1', line)
line = line.replace('https://nni.readthedocs.io/en/latest', '')
return line
for root, dirs, files in os.walk('en_US'):
root = Path(root)
for file in files:
if not file.endswith('.md') or file == 'Release_v1.0.md':
continue
with open(root / file) as f:
md_content = f.read()
if file == 'Nnictl.md':
md_content = process_table(md_content)
out = m2r.convert(md_content)
lines = out.split('\n')
if lines[0] == '':
lines = lines[1:]
# remove code-block eval_rst
i = 0
while i < len(lines):
line = lines[i]
if line.strip() == '.. code-block:: eval_rst':
space_count = line.index('.')
lines[i] = lines[i + 1] = None
if i > 0 and lines[i - 1]:
lines[i] = '' # blank line
i += 2
while i < len(lines) and (lines[i].startswith(' ' * (space_count + 3)) or lines[i] == ''):
lines[i] = lines[i][space_count + 3:]
i += 1
elif line.strip() == '.. code-block' or line.strip() == '.. code-block::':
lines[i] += ':: bash'
i += 1
else:
i += 1
lines = [l for l in lines if l is not None]
lines = list(map(single_line_process, lines))
if file != 'Release.md':
# githublink
lines = list(map(process_github_link, lines))
out = '\n'.join(lines)
out = special_case_replace(out)
with open(root / (Path(file).stem + '.rst'), 'w') as f:
f.write(out)
# back it up and remove
moved_root = Path('archive_en_US') / root.relative_to('en_US')
moved_root.mkdir(exist_ok=True)
shutil.move(root / file, moved_root / file)
* - GP Tuner
- :raw-html:`&#10003;`
-
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
-
-
-
%%%%%%
* - GP Tuner
- :raw-html:`&#10003;`
-
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
- :raw-html:`&#10003;`
-
-
-
-
An SSH server needs a port; you need to expose Docker's SSH port to NNI as the connection port. For example, if you set your container's SSH port as **``A``** \ , you should map the container's port ** ``A``** to your remote host machine's other port ** ``B``** \ , NNI will connect port ** ``B``** as an SSH port, and your host machine will map the connection from port ** ``B``** to port ** ``A``** then NNI could connect to your Docker container.
%%%%%%
An SSH server needs a port; you need to expose Docker's SSH port to NNI as the connection port. For example, if you set your container's SSH port as ``A``, you should map the container's port ``A`` to your remote host machine's other port ``B``, NNI will connect port ``B`` as an SSH port, and your host machine will map the connection from port ``B`` to port ``A`` then NNI could connect to your Docker container.
If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
%%%%%%
If the id ends with \*, nnictl will stop all experiments whose ids matchs the regular.
..
make: *** [install-XXX] Segmentation fault (core dumped)
%%%%%%
.. code-block:: text
make: *** [install-XXX] Segmentation fault (core dumped)
Click ``Submit job`` button in web portal.
%%%%%%
Click ``Submit job`` button in web portal.
:raw-html:`<div >
<img src="https://github.com/microsoft/Cream/blob/main/demo/intro.jpg" width="800"/>
</div>`
%%%%%%
:raw-html:`<div ><img src="https://github.com/microsoft/Cream/blob/main/demo/intro.jpg" width="800"/></div>`
\ No newline at end of file
.. list-table::
:header-rows: 1
%%%%%%
.. list-table::
:header-rows: 1
:widths: auto
.. code-block:: bash
1.1 Declare NNI API
Include `import nni` in your trial code to use NNI APIs.
%%%%%%
..
1.1 Declare NNI API
Include `import nni` in your trial code to use NNI APIs.
.. code-block:: bash
from nni.compression.pytorch.utils.counter import count_flops_params
%%%%%%
.. code-block:: python
from nni.compression.pytorch.utils.counter import count_flops_params
.. code-block:: bash
NNI's official image msranni/nni does not support SSH servers for the time being; you should build your own Docker image with an SSH configuration or use other images as a remote server.
%%%%%%
.. code-block:: text
NNI's official image msranni/nni does not support SSH servers for the time being; you should build your own Docker image with an SSH configuration or use other images as a remote server.
Code Styles & Naming Conventions
--------------------------------
* We follow `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as ``flake8`` or ``pylint``
* We also follow `NumPy Docstring Style <https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy>`__ for Python Docstring Conventions. During the `documentation building <Contributing.rst#documentation>`__\ , we use `sphinx.ext.napoleon <https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html>`__ to generate Python API documentation from Docstring.
* For docstrings, please refer to `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`__ and `pandas docstring guide <https://python-sprints.github.io/pandas/guide/pandas_docstring.html>`__
* For function docstring, **description** , **Parameters**\ , and** Returns**\ /** Yields** are mandatory.
* For class docstring, **description**\ ,** Attributes** are mandatory.
* For docstring to describe ``dict``\ , which is commonly used in our hyper-param format description, please refer to [RiboKit : Doc Standards
* Internal Guideline on Writing Standards](https://ribokit.github.io/docs/text/)
Documentation
-------------
Our documentation is built with :githublink:`sphinx <docs>`.
*
Before submitting the documentation change, please **build homepage locally**\ : ``cd docs/en_US && make html``\ , then you can see all the built documentation webpage under the folder ``docs/en_US/_build/html``. It's also highly recommended taking care of** every WARNING** during the build, which is very likely the signal of a** deadlink** and other annoying issues.
*
For links, please consider using **relative paths** first. However, if the documentation is written in Markdown format, and:
* It's an image link which needs to be formatted with embedded html grammar, please use global URL like ``https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png``\ , which can be automatically generated by dragging picture onto `Github Issue <https://github.com/Microsoft/nni/issues/new>`__ Box.
* It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at ``https://github.com/Microsoft/nni/tree/v1.9/`` (\ :githublink:`mnist.py <examples/trials/mnist-tfv1/mnist.py>` for example).
%%%%%%
Code Styles & Naming Conventions
--------------------------------
* We follow `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as ``flake8`` or ``pylint``
* We also follow `NumPy Docstring Style <https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy>`__ for Python Docstring Conventions. During the `documentation building <Contributing.rst#documentation>`__\ , we use `sphinx.ext.napoleon <https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html>`__ to generate Python API documentation from Docstring.
* For docstrings, please refer to `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`__ and `pandas docstring guide <https://python-sprints.github.io/pandas/guide/pandas_docstring.html>`__
* For function docstring, **description**, **Parameters**, and **Returns** **Yields** are mandatory.
* For class docstring, **description**, **Attributes** are mandatory.
* For docstring to describe ``dict``, which is commonly used in our hyper-param format description, please refer to RiboKit Doc Standards
* `Internal Guideline on Writing Standards <https://ribokit.github.io/docs/text/>`__
Documentation
-------------
Our documentation is built with :githublink:`sphinx <docs>`.
* Before submitting the documentation change, please **build homepage locally**: ``cd docs/en_US && make html``, then you can see all the built documentation webpage under the folder ``docs/en_US/_build/html``. It's also highly recommended taking care of **every WARNING** during the build, which is very likely the signal of a **deadlink** and other annoying issues.
*
For links, please consider using **relative paths** first. However, if the documentation is written in Markdown format, and:
* It's an image link which needs to be formatted with embedded html grammar, please use global URL like ``https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png``, which can be automatically generated by dragging picture onto `Github Issue <https://github.com/Microsoft/nni/issues/new>`__ Box.
* It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at ``https://github.com/Microsoft/nni/tree/v1.9/`` (:githublink:`mnist.py <examples/trials/mnist-tfv1/mnist.py>` for example).
* -
- Recommended
- Minimum
* - **Operating System**
- Ubuntu 16.04 or above
* - **CPU**
- Intel® Core™ i5 or AMD Phenom™ II X3 or better
- Intel® Core™ i3 or AMD Phenom™ X3 8650
* - **GPU**
- NVIDIA® GeForce® GTX 660 or better
- NVIDIA® GeForce® GTX 460
* - **Memory**
- 6 GB RAM
- 4 GB RAM
* - **Storage**
- 30 GB available hare drive space
* - **Internet**
- Boardband internet connection
* - **Resolution**
- 1024 x 768 minimum display resolution
%%%%%%
* -
- Recommended
- Minimum
* - **Operating System**
- Ubuntu 16.04 or above
-
* - **CPU**
- Intel® Core™ i5 or AMD Phenom™ II X3 or better
- Intel® Core™ i3 or AMD Phenom™ X3 8650
* - **GPU**
- NVIDIA® GeForce® GTX 660 or better
- NVIDIA® GeForce® GTX 460
* - **Memory**
- 6 GB RAM
- 4 GB RAM
* - **Storage**
- 30 GB available hare drive space
-
* - **Internet**
- Boardband internet connection
-
* - **Resolution**
- 1024 x 768 minimum display resolution
-
..
1.1 Declare NNI API
Include `import nni` in your trial code to use NNI APIs.
1.2 Get predefined parameters
Use the following code snippet:
RECEIVED_PARAMS = nni.get_next_parameter()
to get hyper-parameters' values assigned by tuner. `RECEIVED_PARAMS` is an object, for example:
{"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}
1.3 Report NNI results
Use the API:
`nni.report_intermediate_result(accuracy)`
to send `accuracy` to assessor.
Use the API:
`nni.report_final_result(accuracy)`
to send `accuracy` to tuner.
%%%%%%
* Declare NNI API: include ``import nni`` in your trial code to use NNI APIs.
* Get predefined parameters
Use the following code snippet:
.. code-block:: python
RECEIVED_PARAMS = nni.get_next_parameter()
to get hyper-parameters' values assigned by tuner. ``RECEIVED_PARAMS`` is an object, for example:
.. code-block:: json
{"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}
* Report NNI results: Use the API: ``nni.report_intermediate_result(accuracy)`` to send ``accuracy`` to assessor.
Use the API: ``nni.report_final_result(accuracy)`` to send `accuracy` to tuner.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment