Commit 1011377c authored by qianyj's avatar qianyj
Browse files

the source code of NNI for DCU

parent abc22158
../../../en_US/_templates/nnSpider/cut.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/errorEmotion.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/holiday.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/nobug.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/sign.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/sweat.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/weaving.html
\ No newline at end of file
../../../en_US/_templates/nnSpider/working.html
\ No newline at end of file
../en_US/autotune_ref.rst
\ No newline at end of file
.. d5351e951811dcaeeda7f270427187fd
内置 Assessor
=================
为了节省计算资源,NNI 支持提前终止策略,并且通过叫做 **Assessor** 的接口来执行此操作。
Assessor 从 Trial 中接收中间结果,并通过指定的算法决定此 Trial 是否应该终止。 一旦 Trial 满足了提前终止策略(这表示 Assessor 认为最终结果不会太好),Assessor 会终止此 Trial,并将其状态标志为 `EARLY_STOPPED`。
这是 MNIST 在 "最大化" 模式下使用 "曲线拟合" Assessor 的实验结果。 可以看到 Assessor 成功的 **提前终止** 了许多结果不好超参组合的 Trial。 使用 Assessor,能在相同的计算资源下,得到更好的结果。
实验代码: :githublink:`config_assessor.yml <examples/trials/mnist-pytorch/config_assessor.yml>`
.. image:: ../img/Assessor.png
.. toctree::
:maxdepth: 1
概述<./Assessor/BuiltinAssessor>
Medianstop<./Assessor/MedianstopAssessor>
Curvefitting(曲线拟合)<./Assessor/CurvefittingAssessor>
.. 1ff18ebada0efec66cd793f1a000f3fe
内置 Tuner
==========
为了让机器学习和深度学习模型适应不同的任务和问题,我们需要进行超参数调优,而自动化调优依赖于优秀的调优算法。NNI 内置了先进的调优算法,并且提供了易于使用的 API。
在 NNI 中,调优算法被称为“tuner”。Tuner 向 trial 发送超参数,接收运行结果从而评估这组超参的性能,然后将下一组超参发送给新的 trial。
下表简要介绍了 NNI 内置的调优算法。点击 tuner 的名称可以查看其安装需求、推荐使用场景、示例配置文件等详细信息。`这篇文章 <../CommunitySharings/HpoComparison.rst>`__ 对比了各个 tuner 在不同场景下的性能。
.. list-table::
:header-rows: 1
:widths: auto
* - Tuner
- 算法简介
* - `TPE <./TpeTuner.rst>`__
- Tree-structured Parzen Estimator (TPE) 是一种基于序列模型的优化方法 (sequential model-based optimization, SMBO)。SMBO方法根据历史数据来顺序地构造模型,从而预估超参性能,并基于此模型来选择新的超参。`参考论文 <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
* - `Random Search (随机搜索) <./RandomTuner.rst>`__
- 随机搜索在超算优化中表现出了令人意外的性能。如果没有对超参分布的先验知识,我们推荐使用随机搜索作为基线方法。`参考论文 <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__
* - `Anneal (退火) <./AnnealTuner.rst>`__
- 朴素退火算法首先基于先验进行采样,然后逐渐逼近实际性能较好的采样点。该算法是随即搜索的变体,利用了反应曲面的平滑性。该实现中退火率不是自适应的。
* - `Naive Evolution(朴素进化) <./EvolutionTuner.rst>`__
- 朴素进化算法来自于 Large-Scale Evolution of Image Classifiers。它基于搜索空间随机生成一个种群,在每一代中选择较好的结果,并对其下一代进行变异。朴素进化算法需要很多 Trial 才能取得最优效果,但它也非常简单,易于扩展。`参考论文 <https://arxiv.org/pdf/1703.01041.pdf>`__
* - `SMAC <./SmacTuner.rst>`__
- SMAC 是基于序列模型的优化方法 (SMBO)。它利用使用过的最突出的模型(高斯随机过程模型),并将随机森林引入到SMBO中,来处理分类参数。NNI 的 SMAC tuner 封装了 GitHub 上的 `SMAC3 <https://github.com/automl/SMAC3>`__。`参考论文 <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__
注意:SMAC 算法需要使用 ``pip install nni[SMAC]`` 安装依赖,暂不支持 Windows 操作系统。
* - `Batch(批处理) <./BatchTuner.rst>`__
- 批处理允许用户直接提供若干组配置,为每种配置运行一个 trial。
* - `Grid Search(网格遍历) <./GridsearchTuner.rst>`__
- 网格遍历会穷举搜索空间中的所有超参组合。
* - `Hyperband <./HyperbandAdvisor.rst>`__
- Hyperband 试图用有限的资源探索尽可能多的超参组合。该算法的思路是,首先生成大量超参配置,将每组超参运行较短的一段时间,随后抛弃其中效果较差的一半,让较好的超参继续运行,如此重复多轮。`参考论文 <https://arxiv.org/pdf/1603.06560.pdf>`__
* - `Metis <./MetisTuner.rst>`__
- 大多数调参工具仅仅预测最优配置,而 Metis 的优势在于它有两个输出:(a) 最优配置的当前预测结果, 以及 (b) 下一次 trial 的建议。大多数工具假设训练集没有噪声数据,但 Metis 会知道是否需要对某个超参重新采样。`参考论文 <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__
* - `BOHB <./BohbAdvisor.rst>`__
- BOHB 是 Hyperband 算法的后续工作。 Hyperband 在生成新的配置时,没有利用已有的 trial 结果,而本算法利用了 trial 结果。BOHB 中,HB 表示 Hyperband,BO 表示贝叶斯优化(Byesian Optimization)。 BOHB 会建立多个 TPE 模型,从而利用已完成的 Trial 生成新的配置。`参考论文 <https://arxiv.org/abs/1807.01774>`__
* - `GP (高斯过程) <./GPTuner.rst>`__
- GP Tuner 是基于序列模型的优化方法 (SMBO),使用高斯过程进行 surrogate。`参考论文 <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
* - `PBT <./PBTTuner.rst>`__
- PBT Tuner 是一种简单的异步优化算法,在固定的计算资源下,它能有效的联合优化一组模型及其超参来最优化性能。`参考论文 <https://arxiv.org/abs/1711.09846v1>`__
* - `DNGO <./DngoTuner.rst>`__
- DNGO 是基于序列模型的优化方法 (SMBO),该算法使用神经网络(而不是高斯过程)去建模贝叶斯优化中所需要的函数分布。
.. toctree::
:maxdepth: 1
TPE <Tuner/TpeTuner>
Random Search(随机搜索) <Tuner/RandomTuner>
Anneal(退火) <Tuner/AnnealTuner>
Naïve Evolution(朴素进化) <Tuner/EvolutionTuner>
SMAC <Tuner/SmacTuner>
Metis Tuner <Tuner/MetisTuner>
Batch Tuner(批处理) <Tuner/BatchTuner>
Grid Search(网格遍历) <Tuner/GridsearchTuner>
GP Tuner <Tuner/GPTuner>
Network Morphism <Tuner/NetworkmorphismTuner>
Hyperband <Tuner/HyperbandAdvisor>
BOHB <Tuner/BohbAdvisor>
PBT Tuner <Tuner/PBTTuner>
DNGO Tuner <Tuner/DngoTuner>
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import subprocess
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- Project information ---------------------------------------------------
project = 'NNI'
copyright = '2021, Microsoft'
author = 'Microsoft'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = 'v2.3'
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.mathjax',
'sphinxarg.ext',
'sphinx.ext.napoleon',
'sphinx.ext.viewcode',
'sphinx.ext.intersphinx',
'nbsphinx',
'sphinx.ext.extlinks',
'IPython.sphinxext.ipython_console_highlighting',
]
# 添加示例模块
autodoc_mock_imports = ['apex', 'nni_node', 'tensorrt', 'pycuda']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
source_suffix = ['.rst']
# The master toctree document.
master_doc = 'contents'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md', '**.ipynb_checkpoints']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
html_additional_pages = {
'index': 'index.html',
}
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
'logo_only': True,
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['../static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
html_logo = '../img/nni_logo_dark.png'
html_title = '支持神经网络结构搜索、模型压缩、超参调优的开源自动机器学习工具 (%s %s)' % \
(project, release)
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'NeuralNetworkIntelligencedoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'NeuralNetworkIntelligence.tex', 'Neural Network Intelligence Documentation',
'Microsoft', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'neuralnetworkintelligence', 'Neural Network Intelligence Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'NeuralNetworkIntelligence', 'Neural Network Intelligence Documentation',
author, 'NeuralNetworkIntelligence', 'One line description of project.',
'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# external links (for github code)
# Reference the code via :githublink:`path/to/your/example/code.py`
git_commit_id = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()
extlinks = {
'githublink': ('https://github.com/microsoft/nni/blob/' + git_commit_id + '/%s', 'Github link: ')
}
# -- Extension configuration -------------------------------------------------
def setup(app):
app.add_css_file('css/custom.css')
.. 823dad4710f371e923033eebd9dba255
###########################
Neural Network Intelligence
###########################
.. toctree::
:caption: 目录
:maxdepth: 2
:titlesonly:
概述<Overview>
安装 <installation>
入门<Tutorial/QuickStart>
自动(超参数)调优 <hyperparameter_tune>
神经网络架构搜索<nas>
模型压缩<model_compression>
特征工程<feature_engineering>
参考<reference>
示例与解决方案<CommunitySharings/community_sharings>
研究和出版物 <ResearchPublications>
常见问题 <Tutorial/FAQ>
如何贡献 <contribution>
更改日志 <Release>
.. 24da49b25d3d36c476a69aceb825cb94
###############################
贡献代码
###############################
.. toctree::
设置开发环境<./Tutorial/SetupNniDeveloperEnvironment>
贡献指南<./Tutorial/Contributing>
\ No newline at end of file
.. d19a00598b8eca71c825d80c0a7106f2
######################
示例
######################
.. toctree::
:maxdepth: 2
MNIST<./TrialExample/MnistExamples>
Cifar10<./TrialExample/Cifar10Examples>
Scikit-learn<./TrialExample/SklearnExamples>
GBDT<./TrialExample/GbdtExample>
Pix2pix<./TrialExample/Pix2pixExample>
\ No newline at end of file
.. 0958703dcd6f8078a1ad1bcaef9c7199
###################
特征工程
###################
很高兴在 NNI 上引入了特征工程工具包,
其仍处于试验阶段,会根据使用反馈来演化。
诚挚邀请您使用、反馈,或更多贡献。
详细信息,参考以下教程:
.. toctree::
:maxdepth: 2
概述 <FeatureEngineering/Overview>
GradientFeatureSelector <FeatureEngineering/GradientFeatureSelector>
GBDTSelector <FeatureEngineering/GBDTSelector>
.. 43bb394b1e25458a948c134058ec68ac
高级功能
=================
.. toctree::
:maxdepth: 2
编写新的 Tuner <Tuner/CustomizeTuner>
编写新的 Assessor <Assessor/CustomizeAssessor>
编写新的 Advisor <Tuner/CustomizeAdvisor>
编写新的训练平台 <TrainingService/HowToImplementTrainingService>
安装自定义的 Tuners/Assessors/Advisors <Tutorial/InstallCustomizedAlgos>
../en_US/hpo_benchmark.rst
\ No newline at end of file
../en_US/hpo_benchmark_stats.rst
\ No newline at end of file
.. 6ed30d3a87dbc4c1c4650cf56f074045
##############
自动超参数调优
##############
自动调优是 NNI 的主要功能之一。它的工作模式是反复运行 trial 代码,每次向其提供不同的超参组合,从而对 trial 的运行结果进行调优。NNI 提供了很多流行的自动调优算法(称为 Tuner)和一些提前终止算法(称为 Assessor)。NNI 支持在多种训练平台上运行 trial,包括本机、远程服务器、Azure Machine Learning、基于 Kubernetes 的集群(如 OpenPAI、Kubeflow)等等。
NNI 具有高扩展性,用户可以根据需求实现自己的 Tuner 算法和训练平台。
.. toctree::
:maxdepth: 2
实现 Trial <./TrialExample/Trials>
Tuners <builtin_tuner>
Assessors <builtin_assessor>
训练平台 <training_services>
示例 <examples>
Web 界面 <Tutorial/WebUI>
如何调试 <Tutorial/HowToDebug>
高级功能 <hpo_advanced>
Tuner 基准测试 <hpo_benchmark>
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment