Skip to content

GitLab

Unverified Commit 070df4a0 authored by liuzhe-lz's avatar liuzhe-lz Committed by GitHub
Browse files

Merge pull request #4291 from microsoft/v2.5 

merge v2.5 back to master
parents 821706b8 6a082fe9
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 241 additions and 92 deletions
docs/en_US/Tutorial/NNSpider.md 0 → 100644
View file @ 070df4a0
<!-- <style>
table, tr, td{
border: none;
}
div{
width: 300px;
height: 200px;
border: 1px solid grey;
background: #ccc;
box-sizing: border-box;
}
img{
width: 260px;
height: 160px;
margin: 20px;
}
</style> -->
<table>
<tr>
<td>
<div>
<img style="
width: 300px;
"
src="../../img/emoicons/NoBug.png"/>
</div>
</td>
<td>
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Holiday.png"/>
</div>
</td>
<td>
<div>
<img style="
width: 300px;
height: 180px;
" src="../../img/emoicons/Error.png"/>
</div>
</td>
</tr>
<tr>
<td align="center">No bug</td>
<td align="center">Holiday</td>
<td align="center">Error</td>
</tr>
<tr>
<td>
<div>
<img style="
width: 300px;
height: 210px;
" src="../../img/emoicons/Working.png"/>
</div>
</td>
<td >
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Sign.png"/>
</div>
</td>
<td>
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Crying.png"/>
</div>
</td>
</tr>
<tr>
<td align="center" >Working</td>
<td align="center" >Sign</td>
<td align="center" >Crying</td>
</tr>
<tr>
<td>
<div>
<img style="
width: 300px;
height: 190px;
" src="../../img/emoicons/Cut.png"/>
</div>
</td>
<td>
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Weaving.png"/>
</div>
</td>
<td>
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Comfort.png"/>
</div>
</td>
</tr>
<tr>
<td align="center">Cut</td>
<td align="center">Weaving</td>
<td align="center">Comfort</td>
</tr>
<tr>
<td>
<div>
<img style="
width: 300px;
" src="../../img/emoicons/Sweat.png"/>
</div>
</td>
<td></td>
<td></td>
</tr>
<tr>
<td align="center">Sweat</td>
<td align="center"></td>
<td align="center"></td>
</tr>
</table>
docs/en_US/conf.py
View file @ 070df4a0
......@@ -27,7 +27,7 @@ author = 'Microsoft'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = 'v2.4'
release = 'v2.5'
# -- General configuration ---------------------------------------------------
......
docs/en_US/model_compression.rst
View file @ 070df4a0
......@@ -26,6 +26,7 @@ For details, please refer to the following tutorials:
Overview <Compression/Overview>
Quick Start <Compression/QuickStart>
Pruning <Compression/pruning>
Pruning V2 <Compression/v2_pruning>
Quantization <Compression/quantization>
Utilities <Compression/CompressionUtils>
Advanced Usage <Compression/advanced>
......
docs/en_US/reference/experiment_config.rst
View file @ 070df4a0
......@@ -114,17 +114,17 @@ ExperimentConfig
- Description
* - experimentName
- ``Optional[str]``
- ``str``, optional
- Mnemonic name of the experiment, which will be shown in WebUI and nnictl.
* - searchSpaceFile
- ``Optional[str]``
- ``str``, optional
- Path_ to the JSON file containing the search space.
Search space format is determined by tuner. The common format for built-in tuners is documented `here <../Tutorial/SearchSpaceSpec.rst>`__.
Mutually exclusive to ``searchSpace``.
* - searchSpace
- ``Optional[JSON]``
- ``JSON``, optional
- Search space object.
The format is determined by tuner. Common format for built-in tuners is documented `here <../Tutorial/SearchSpaceSpec.rst>`__.
Note that ``None`` means "no such field" so empty search space should be written as ``{}``.
......@@ -137,9 +137,8 @@ ExperimentConfig
Note that using ``python3`` on Linux and macOS, and using ``python`` on Windows.
* - trialCodeDirectory
- ``str``
- `Path`_ to the directory containing trial source files.
default: ``"."``.
- ``str``, optional
- Default: ``"."``. `Path`_ to the directory containing trial source files.
All files in this directory will be sent to the training machine, unless in the ``.nniignore`` file.
(See :ref:`nniignore <nniignore>` for details.)
......@@ -149,8 +148,8 @@ ExperimentConfig
The real concurrency also depends on hardware resources and may be less than this value.
* - trialGpuNumber
- ``Optional[int]``
- This field might have slightly different meanings for various training services,
- ``int`` or ``None``, optional
- Default: None. This field might have slightly different meanings for various training services,
especially when set to ``0`` or ``None``.
See `training service's document <../training_services.rst>`__ for details.
......@@ -159,75 +158,72 @@ ExperimentConfig
but they can still use all GPU resources if they want.
* - maxExperimentDuration
- ``Optional[str]``
- Limit the duration of this experiment if specified.
format: ``number + s|m|h|d``
examples: ``"10m"``, ``"0.5h"``
- ``str``, optional
- Limit the duration of this experiment if specified. The duration is unlimited if not set.
Format: ``number + s|m|h|d``.
Examples: ``"10m"``, ``"0.5h"``.
When time runs out, the experiment will stop creating trials but continue to serve WebUI.
* - maxTrialNumber
- ``Optional[int]``
- Limit the number of trials to create if specified.
- ``int``, optional
- Limit the number of trials to create if specified. The trial number is unlimited if not set.
When the budget runs out, the experiment will stop creating trials but continue to serve WebUI.
* - maxTrialDuration
- ``Optional[str]``
- Limit the duration of trial job if specified.
format: ``number + s|m|h|d``
examples: ``"10m"``, ``"0.5h"``
- ``str``, optional
- Limit the duration of trial job if specified. The duration is unlimited if not set.
Format: ``number + s|m|h|d``.
Examples: ``"10m"``, ``"0.5h"``.
When time runs out, the current trial job will stop.
* - nniManagerIp
- ``Optional[str]``
- IP of the current machine, used by training machines to access NNI manager. Not used in local mode.
If not specified, IPv4 address of ``eth0`` will be used.
- ``str``, optional
- Default: default connection chosen by system. IP of the current machine, used by training machines to access NNI manager. Not used in local mode.
Except for the local mode, it is highly recommended to set this field manually.
* - useAnnotation
- ``bool``
- Enable `annotation <../Tutorial/AnnotationSpec.rst>`__.
default: ``False``.
- ``bool``, optional
- Default: ``False``. Enable `annotation <../Tutorial/AnnotationSpec.rst>`__.
When using annotation, ``searchSpace`` and ``searchSpaceFile`` should not be specified manually.
* - debug
- ``bool``
- Enable debug mode.
default: ``False``
- ``bool``, optional
- Default: ``False``. Enable debug mode.
When enabled, logging will be more verbose and some internal validation will be loosened.
* - logLevel
- ``Optional[str]``
- Set log level of the whole system.
- ``str``, optional
- Default: ``info`` or ``debug``, depending on ``debug`` option. Set log level of the whole system.
values: ``"trace"``, ``"debug"``, ``"info"``, ``"warning"``, ``"error"``, ``"fatal"``
Defaults to "info" or "debug", depending on ``debug`` option. When debug mode is enabled, Loglevel is set to "debug", otherwise, Loglevel is set to "info".
When debug mode is enabled, Loglevel is set to "debug", otherwise, Loglevel is set to "info".
Most modules of NNI will be affected by this value, including NNI manager, tuner, training service, etc.
The exception is trial, whose logging level is directly managed by trial code.
For Python modules, "trace" acts as logging level 0 and "fatal" acts as ``logging.CRITICAL``.
* - experimentWorkingDirectory
- ``Optional[str]``
- Specify the :ref:`directory <path>` to place log, checkpoint, metadata, and other run-time stuff.
By default uses ``~/nni-experiments``.
- ``str``, optional
- Default: ``~/nni-experiments``.
Specify the :ref:`directory <path>` to place log, checkpoint, metadata, and other run-time stuff.
NNI will create a subdirectory named by experiment ID, so it is safe to use the same directory for multiple experiments.
* - tunerGpuIndices
- ``Optional[list[int] | str | int]``
- ``list[int]`` or ``str`` or ``int``, optional
- Limit the GPUs visible to tuner, assessor, and advisor.
This will be the ``CUDA_VISIBLE_DEVICES`` environment variable of tuner process.
Because tuner, assessor, and advisor run in the same process, this option will affect them all.
* - tuner
- ``Optional[AlgorithmConfig]``
- ``AlgorithmConfig``, optional
- Specify the tuner.
The built-in tuners can be found `here <../builtin_tuner.rst>`__ and you can follow `this tutorial <../Tuner/CustomizeTuner.rst>`__ to customize a new tuner.
* - assessor
- ``Optional[AlgorithmConfig]``
- ``AlgorithmConfig``, optional
- Specify the assessor.
The built-in assessors can be found `here <../builtin_assessor.rst>`__ and you can follow `this tutorial <../Assessor/CustomizeAssessor.rst>`__ to customize a new assessor.
* - advisor
- ``Optional[AlgorithmConfig]``
- ``AlgorithmConfig``, optional
- Specify the advisor.
NNI provides two built-in advisors: `BOHB <../Tuner/BohbAdvisor.rst>`__ and `Hyperband <../Tuner/HyperbandAdvisor.rst>`__, and you can follow `this tutorial <../Tuner/CustomizeAdvisor.rst>`__ to customize a new advisor.
......@@ -236,7 +232,7 @@ ExperimentConfig
- Specify the `training service <../TrainingService/Overview.rst>`__.
* - sharedStorage
- ``Optional[SharedStorageConfig]``
- ``SharedStorageConfig``, optional
- Configure the shared storage, detailed usage can be found `here <../Tutorial/HowToUseSharedStorage.rst>`__.
AlgorithmConfig
......@@ -259,23 +255,23 @@ For customized algorithms, there are two ways to describe them:
- Description
* - name
- ``Optional[str]``
- Name of the built-in or registered algorithm.
- ``str`` or ``None``, optional
- Default: None. Name of the built-in or registered algorithm.
``str`` for the built-in and registered algorithm, ``None`` for other customized algorithms.
* - className
- ``Optional[str]``
- Qualified class name of not registered customized algorithm.
- ``str`` or ``None``, optional
- Default: None. Qualified class name of not registered customized algorithm.
``None`` for the built-in and registered algorithm, ``str`` for other customized algorithms.
example: ``"my_tuner.MyTuner"``
* - codeDirectory
- ``Optional[str]``
- `Path`_ to the directory containing the customized algorithm class.
- ``str`` or ``None``, optional
- Default: None. Path_ to the directory containing the customized algorithm class.
``None`` for the built-in and registered algorithm, ``str`` for other customized algorithms.
* - classArgs
- ``Optional[dict[str, Any]]``
- ``dict[str, Any]``, optional
- Keyword arguments passed to algorithm class' constructor.
See algorithm's document for supported value.
......@@ -311,8 +307,8 @@ Detailed usage can be found `here <../TrainingService/LocalMode.rst>`__.
-
* - useActiveGpu
- ``Optional[bool]``
- Specify whether NNI should submit trials to GPUs occupied by other tasks.
- ``bool``, optional
- Default: ``False``. Specify whether NNI should submit trials to GPUs occupied by other tasks.
Must be set when ``trialGpuNumber`` greater than zero.
Following processes can make GPU "active":
......@@ -325,12 +321,11 @@ Detailed usage can be found `here <../TrainingService/LocalMode.rst>`__.
When you create multiple NNI experiments and ``useActiveGpu`` is set to ``True``, they will submit multiple trials to the same GPU(s) simultaneously.
* - maxTrialNumberPerGpu
- ``int``
- Specify how many trials can share one GPU.
default: ``1``
- ``int``, optional
- Default: ``1``. Specify how many trials can share one GPU.
* - gpuIndices
- ``Optional[list[int] | str | int]``
- ``list[int]`` or ``str`` or ``int``, optional
- Limit the GPUs visible to trial processes.
If ``trialGpuNumber`` is less than the length of this value, only a subset will be visible to each trial.
This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.
......@@ -357,8 +352,8 @@ Detailed usage can be found `here <../TrainingService/RemoteMachineMode.rst>`__.
- List of training machines.
* - reuseMode
- ``bool``
- Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
- ``bool``, optional
- Default: ``True``. Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
RemoteMachineConfig
"""""""""""""""""""
......@@ -376,31 +371,29 @@ RemoteMachineConfig
- IP or hostname (domain name) of the machine.
* - port
- ``int``
- SSH service port.
default: ``22``
- ``int``, optional
- Default: ``22``. SSH service port.
* - user
- ``str``
- Login user name.
* - password
- ``Optional[str]``
- ``str``, optional
- If not specified, ``sshKeyFile`` will be used instead.
* - sshKeyFile
- ``Optional[str]``
- ``str``, optional
- `Path`_ to ``sshKeyFile`` (identity file).
Only used when ``password`` is not specified.
* - sshPassphrase
- ``Optional[str]``
- ``str``, optional
- Passphrase of SSH identity file.
* - useActiveGpu
- ``bool``
- Specify whether NNI should submit trials to GPUs occupied by other tasks.
default: ``False``
- ``bool``, optional
- Default: ``False``. Specify whether NNI should submit trials to GPUs occupied by other tasks.
Must be set when ``trialGpuNumber`` greater than zero.
Following processes can make GPU "active":
......@@ -413,18 +406,17 @@ RemoteMachineConfig
When you create multiple NNI experiments and ``useActiveGpu`` is set to ``True``, they will submit multiple trials to the same GPU(s) simultaneously.
* - maxTrialNumberPerGpu
- ``int``
- Specify how many trials can share one GPU.
default: ``1``
- ``int``, optional
- Default: ``1``. Specify how many trials can share one GPU.
* - gpuIndices
- ``Optional[list[int] | str | int]``
- ``list[int]`` or ``str`` or ``int``, optional
- Limit the GPUs visible to trial processes.
If ``trialGpuNumber`` is less than the length of this value, only a subset will be visible to each trial.
This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.
* - pythonPath
- ``Optional[str]``
- ``str``, optional
- Specify a Python environment.
This path will be inserted at the front of PATH. Here are some examples:
......@@ -434,7 +426,7 @@ RemoteMachineConfig
If you are working on Anaconda, there is some difference. On Windows, you also have to add ``../script`` and ``../Library/bin`` separated by ``;``. Examples are as below:
- (linux anaconda) pythonPath: ``/home/yourname/anaconda3/envs/myenv/bin/``
- (windows anaconda) pythonPath: ``C:/Users/yourname/.conda/envs/myenv;C:/Users/yourname/.conda/envs/myenv/Scripts;C:/Users/yourname/.conda/envs/myenv/Library/bin``
- (windows anaconda) pythonPath: ``C:/Users/yourname/.conda/envs/myenv``; ``C:/Users/yourname/.conda/envs/myenv/Scripts``; ``C:/Users/yourname/.conda/envs/myenv/Library/bin``
This is useful if preparing steps vary for different machines.
......@@ -485,9 +477,8 @@ Detailed usage can be found `here <../TrainingService/PaiMode.rst>`__.
- Specify the storage name used in OpenPAI.
* - dockerImage
- ``str``
- Name and tag of docker image to run the trials.
default: ``"msranni/nni:latest"``.
- ``str``, optional
- Default: ``"msranni/nni:latest"``. Name and tag of docker image to run the trials.
* - localStorageMountPoint
- ``str``
......@@ -499,16 +490,15 @@ Detailed usage can be found `here <../TrainingService/PaiMode.rst>`__.
This must be an absolute path.
* - reuseMode
- ``bool``
- Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
default: ``False``.
- ``bool``, optional
- Default: ``True``. Enable `reuse mode <../TrainingService/Overview.rst#training-service-under-reuse-mode>`__.
* - openpaiConfig
- ``Optional[JSON]``
- ``JSON``, optional
- Embedded OpenPAI config file.
* - openpaiConfigFile
- ``Optional[str]``
- ``str``, optional
- `Path`_ to OpenPAI config file.
An example can be found `here <https://github.com/microsoft/pai/blob/master/docs/manual/cluster-user/examples/hello-world-job.yaml>`__.
......@@ -528,11 +518,10 @@ Detailed usage can be found `here <../TrainingService/AMLMode.rst>`__.
* - platform
- ``"aml"``
-
* - dockerImage
- ``str``
- Name and tag of docker image to run the trials.
default: ``"msranni/nni:latest"``
- ``str``, optional
- Default: ``"msranni/nni:latest"``. Name and tag of docker image to run the trials.
* - subscriptionId
- ``str``
......@@ -568,17 +557,16 @@ Detailed usage can be found `here <../TrainingService/DlcMode.rst>`__.
-
* - type
- ``str``
- Job spec type.
default: ``"worker"``.
- ``str``, optional
- Default: ``"Worker"``. Job spec type.
* - image
- ``str``
- Name and tag of docker image to run the trials.
* - jobType
- ``str``
- PAI-DLC training job type, ``"TFJob"`` or ``"PyTorchJob"``.
- ``str``, optional
- Default: ``"TFJob"``. PAI-DLC training job type, ``"TFJob"`` or ``"PyTorchJob"``.
* - podCount
- ``str``
......@@ -698,7 +686,7 @@ azureBlobConfig
- Azure storage account name.
* - storageAccountKey
- ``Optional[str]``
- ``str``
- Azure storage account key.
* - containerName
......
docs/img/emoicons/home.svg 0 → 100644
View file @ 070df4a0
This diff is collapsed.
docs/requirements.txt
View file @ 070df4a0
......@@ -26,3 +26,5 @@ gym
tianshou
https://download.pytorch.org/whl/cpu/torch-1.7.1%2Bcpu-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.8.2%2Bcpu-cp37-cp37m-linux_x86_64.whl
pytorch-lightning
onnx
examples/model_compress/pruning/v2/iterative_pruning_torch.py
View file @ 070df4a0
import sys
from tqdm import tqdm
import torch
......@@ -5,7 +6,8 @@ from torchvision import datasets, transforms
from nni.algorithms.compression.v2.pytorch.pruning import AGPPruner
from examples.model_compress.models.cifar10.vgg import VGG
sys.path.append('../../models')
from cifar10.vgg import VGG
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
......
examples/model_compress/pruning/v2/scheduler_torch.py
View file @ 070df4a0
import sys
from tqdm import tqdm
import torch
......@@ -7,7 +8,8 @@ from nni.algorithms.compression.v2.pytorch.pruning import L1NormPruner
from nni.algorithms.compression.v2.pytorch.pruning.tools import AGPTaskGenerator
from nni.algorithms.compression.v2.pytorch.pruning.basic_scheduler import PruningScheduler
from examples.model_compress.models.cifar10.vgg import VGG
sys.path.append('../../models')
from cifar10.vgg import VGG
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
......@@ -90,6 +92,8 @@ if __name__ == '__main__':
# or the result with the highest score (given by evaluator) will be the best result.
# scheduler = PruningScheduler(pruner, task_generator, finetuner=finetuner, speed_up=True, dummy_input=dummy_input, evaluator=evaluator)
scheduler = PruningScheduler(pruner, task_generator, finetuner=finetuner, speed_up=True, dummy_input=dummy_input, evaluator=None)
scheduler = PruningScheduler(pruner, task_generator, finetuner=finetuner, speed_up=True, dummy_input=dummy_input, evaluator=None, reset_weight=False)
scheduler.compress()
_, model, masks, _, _ = scheduler.get_best_result()
examples/model_compress/pruning/v2/simple_pruning_torch.py
View file @ 070df4a0
import sys
from tqdm import tqdm
import torch
......@@ -6,7 +7,8 @@ from torchvision import datasets, transforms
from nni.algorithms.compression.v2.pytorch.pruning import L1NormPruner
from nni.compression.pytorch.speedup import ModelSpeedup
from examples.model_compress.models.cifar10.vgg import VGG
sys.path.append('../../models')
from cifar10.vgg import VGG
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
......@@ -72,7 +74,7 @@ if __name__ == '__main__':
evaluator(model)
pruner._unwrap_model()
ModelSpeedup(model, dummy_input=torch.rand(10, 3, 32, 32).to(device), masks_file='simple_masks.pth').speedup_model()
ModelSpeedup(model, dummy_input=torch.rand(10, 3, 32, 32).to(device), masks_file=masks).speedup_model()
print('\nThe accuracy after speed up:')
evaluator(model)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment