Skip to content

GitLab

"...references/git@developer.sourcefind.cn:change/sglang.git" did not exist on "909abb58f551b9b517587a97b66761fd73568718"
Unverified Commit 7620e7c5 authored by SparkSnail's avatar SparkSnail Committed by GitHub
Browse files

Merge pull request #214 from microsoft/master 

merge master
parents c037a7c1 187494aa
Show whitespace changes
Inline Side-by-side
Showing with 663 additions and 288 deletions
azure-pipelines.yml
View file @ 7620e7c5
...@@ -126,18 +126,10 @@ jobs: ...@@ -126,18 +126,10 @@ jobs:
cd test cd test
powershell.exe -file unittest.ps1 powershell.exe -file unittest.ps1
displayName: 'unit test' displayName: 'unit test'
- script: |
cd test
python naive_test.py
displayName: 'Naive test'
- script: | - script: |
cd test cd test
python tuner_test.py python tuner_test.py
displayName: 'Built-in tuners / assessors tests' displayName: 'Built-in tuners / assessors tests'
- script: |
cd test
python metrics_test.py
displayName: 'Trial job metrics test'
- script: | - script: |
cd test cd test
PATH=$HOME/.local/bin:$PATH python3 cli_test.py PATH=$HOME/.local/bin:$PATH python3 cli_test.py
......
docs/en_US/CommunitySharings/TuningSystems.md 0 → 100644
View file @ 7620e7c5
# Automatically tune systems with NNI
As computer systems and networking get increasingly complicated, optimizing them manually with explicit rules and heuristics becomes harder than ever before, sometimes impossible. Below are two examples of tuning systems with NNI. Anyone can easily tune their own systems by following them.
* [Tuning RocksDB with NNI](../TrialExample/RocksdbExamples.md)
* [Tuning parameters of SPTAG (Space Partition Tree And Graph) with NNI](SptagAutoTune.md)
Please see [this paper](https://dl.acm.org/citation.cfm?id=3352031) for more details:
Mike Liang, Chieh-Jan, et al. "The Case for Learning-and-System Co-design." ACM SIGOPS Operating Systems Review 53.1 (2019): 68-74.
docs/en_US/Compressor/Overview.md
View file @ 7620e7c5
...@@ -180,13 +180,55 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer): ...@@ -180,13 +180,55 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer):
def quantize_weight(self, weight, config, **kwargs): def quantize_weight(self, weight, config, **kwargs):
""" """
weight is the target weight tensor quantize should overload this method to quantize weight tensors.
config is the selected dict object in config_list for this layer This method is effectively hooked to :meth:`forward` of the model.
kwargs contains op, op_types, and op_name
design your quantizer and return new weight Parameters
----------
weight : Tensor
weight that needs to be quantized
config : dict
the configuration for weight quantization
""" """
# Put your code to generate `new_weight` here
return new_weight return new_weight
def quantize_output(self, output, config, **kwargs):
"""
quantize should overload this method to quantize output.
This method is effectively hooked to `:meth:`forward` of the model.
Parameters
----------
output : Tensor
output that needs to be quantized
config : dict
the configuration for output quantization
"""
# Put your code to generate `new_output` here
return new_output
def quantize_input(self, *inputs, config, **kwargs):
"""
quantize should overload this method to quantize input.
This method is effectively hooked to :meth:`forward` of the model.
Parameters
----------
inputs : Tensor
inputs that needs to be quantized
config : dict
the configuration for inputs quantization
"""
# Put your code to generate `new_input` here
return new_input
# note for pytorch version, there is no sess in input arguments # note for pytorch version, there is no sess in input arguments
def update_epoch(self, epoch_num, sess): def update_epoch(self, epoch_num, sess):
pass pass
...@@ -200,8 +242,6 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer): ...@@ -200,8 +242,6 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer):
pass pass
``` ```
__[TODO]__ Will add another member function `quantize_layer_output`, as some quantization algorithms also quantize layers' output.
### Usage of user customized compression algorithm ### Usage of user customized compression algorithm
__[TODO]__ ... __[TODO]__ ...
docs/en_US/TrialExample/GbdtExample.md
View file @ 7620e7c5
...@@ -44,7 +44,15 @@ Given the features and label in train data, we train a GBDT regression model and ...@@ -44,7 +44,15 @@ Given the features and label in train data, we train a GBDT regression model and
## 3. How to run in nni ## 3. How to run in nni
### 3.1 Prepare your trial code
### 3.1 Install all the requirments
```
pip install lightgbm
pip install pandas
```
### 3.2 Prepare your trial code
You need to prepare a basic code as following: You need to prepare a basic code as following:
...@@ -86,7 +94,7 @@ if __name__ == '__main__': ...@@ -86,7 +94,7 @@ if __name__ == '__main__':
run(lgb_train, lgb_eval, PARAMS, X_test, y_test) run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
``` ```
### 3.2 Prepare your search space. ### 3.3 Prepare your search space.
If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `bagging_freq`, you could write a [search_space.json](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/search_space.json) as follow: If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `bagging_freq`, you could write a [search_space.json](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/search_space.json) as follow:
```json ```json
...@@ -100,7 +108,7 @@ If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `baggi ...@@ -100,7 +108,7 @@ If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `baggi
More support variable type you could reference [here](../Tutorial/SearchSpaceSpec.md). More support variable type you could reference [here](../Tutorial/SearchSpaceSpec.md).
### 3.3 Add SDK of nni into your code. ### 3.4 Add SDK of nni into your code.
```diff ```diff
+import nni +import nni
...@@ -146,7 +154,7 @@ if __name__ == '__main__': ...@@ -146,7 +154,7 @@ if __name__ == '__main__':
run(lgb_train, lgb_eval, PARAMS, X_test, y_test) run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
``` ```
### 3.4 Write a config file and run it. ### 3.5 Write a config file and run it.
In the config file, you could set some settings including: In the config file, you could set some settings including:
......
docs/en_US/Tutorial/Contributing.md
View file @ 7620e7c5
...@@ -40,6 +40,9 @@ A person looking to contribute can take up an issue by claiming it as a comment/ ...@@ -40,6 +40,9 @@ A person looking to contribute can take up an issue by claiming it as a comment/
## Code Styles & Naming Conventions ## Code Styles & Naming Conventions
* We follow [PEP8](https://www.python.org/dev/peps/pep-0008/) for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as `flake8` or `pylint` * We follow [PEP8](https://www.python.org/dev/peps/pep-0008/) for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as `flake8` or `pylint`
* We also follow [NumPy Docstring Style](https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy) for Python Docstring Conventions. During the [documentation building](Contributing.md#documentation), we use [sphinx.ext.napoleon](https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html) to generate Python API documentation from Docstring. * We also follow [NumPy Docstring Style](https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy) for Python Docstring Conventions. During the [documentation building](Contributing.md#documentation), we use [sphinx.ext.napoleon](https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html) to generate Python API documentation from Docstring.
* For docstrings, please refer to [numpydoc docstring guide](https://numpydoc.readthedocs.io/en/latest/format.html) and [pandas docstring guide](https://python-sprints.github.io/pandas/guide/pandas_docstring.html)
* For function docstring, **description**, **Parameters**, and **Returns**/**Yields** are mandatory.
* For class docstring, **description**, **Attributes** are mandatory.
## Documentation ## Documentation
Our documentation is built with [sphinx](http://sphinx-doc.org/), supporting [Markdown](https://guides.github.com/features/mastering-markdown/) and [reStructuredText](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) format. All our documentations are placed under [docs/en_US](https://github.com/Microsoft/nni/tree/master/docs). Our documentation is built with [sphinx](http://sphinx-doc.org/), supporting [Markdown](https://guides.github.com/features/mastering-markdown/) and [reStructuredText](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) format. All our documentations are placed under [docs/en_US](https://github.com/Microsoft/nni/tree/master/docs).
......
docs/en_US/sdk_reference.rst
View file @ 7620e7c5
...@@ -24,10 +24,10 @@ Tuner ...@@ -24,10 +24,10 @@ Tuner
.. autoclass:: nni.evolution_tuner.evolution_tuner.EvolutionTuner .. autoclass:: nni.evolution_tuner.evolution_tuner.EvolutionTuner
:members: :members:
.. autoclass:: nni.smac_tuner.smac_tuner.SMACTuner .. autoclass:: nni.smac_tuner.SMACTuner
:members: :members:
.. autoclass:: nni.gridsearch_tuner.gridsearch_tuner.GridSearchTuner .. autoclass:: nni.gridsearch_tuner.GridSearchTuner
:members: :members:
.. autoclass:: nni.networkmorphism_tuner.networkmorphism_tuner.NetworkMorphismTuner .. autoclass:: nni.networkmorphism_tuner.networkmorphism_tuner.NetworkMorphismTuner
...@@ -36,6 +36,15 @@ Tuner ...@@ -36,6 +36,15 @@ Tuner
.. autoclass:: nni.metis_tuner.metis_tuner.MetisTuner .. autoclass:: nni.metis_tuner.metis_tuner.MetisTuner
:members: :members:
.. autoclass:: nni.ppo_tuner.PPOTuner
:members:
.. autoclass:: nni.batch_tuner.batch_tuner.BatchTuner
:members:
.. autoclass:: nni.gp_tuner.gp_tuner.GPTuner
:members:
Assessor Assessor
------------------------ ------------------------
.. autoclass:: nni.assessor.Assessor .. autoclass:: nni.assessor.Assessor
......
src/nni_manager/common/experimentStartupInfo.ts
View file @ 7620e7c5
...@@ -43,7 +43,7 @@ class ExperimentStartupInfo { ...@@ -43,7 +43,7 @@ class ExperimentStartupInfo {
this.initialized = true; this.initialized = true;
if (logDir !== undefined && logDir.length > 0) { if (logDir !== undefined && logDir.length > 0) {
this.logDir = path.join(logDir, getExperimentId()); this.logDir = path.join(path.normalize(logDir), getExperimentId());
} else { } else {
this.logDir = path.join(os.homedir(), 'nni', 'experiments', getExperimentId()); this.logDir = path.join(os.homedir(), 'nni', 'experiments', getExperimentId());
} }
......
src/nni_manager/training_service/common/util.ts
View file @ 7620e7c5
...@@ -70,11 +70,11 @@ export async function validateCodeDir(codeDir: string) : Promise<number> { ...@@ -70,11 +70,11 @@ export async function validateCodeDir(codeDir: string) : Promise<number> {
*/ */
export async function execMkdir(directory: string, share: boolean = false): Promise<void> { export async function execMkdir(directory: string, share: boolean = false): Promise<void> {
if (process.platform === 'win32') { if (process.platform === 'win32') {
await cpp.exec(`powershell.exe New-Item -Path ${directory} -ItemType "directory" -Force`); await cpp.exec(`powershell.exe New-Item -Path "${directory}" -ItemType "directory" -Force`);
} else if (share) { } else if (share) {
await cpp.exec(`(umask 0; mkdir -p ${directory})`); await cpp.exec(`(umask 0; mkdir -p '${directory}')`);
} else { } else {
await cpp.exec(`mkdir -p ${directory}`); await cpp.exec(`mkdir -p '${directory}'`);
} }
return Promise.resolve(); return Promise.resolve();
...@@ -87,9 +87,9 @@ export async function execMkdir(directory: string, share: boolean = false): Prom ...@@ -87,9 +87,9 @@ export async function execMkdir(directory: string, share: boolean = false): Prom
*/ */
export async function execCopydir(source: string, destination: string): Promise<void> { export async function execCopydir(source: string, destination: string): Promise<void> {
if (process.platform === 'win32') { if (process.platform === 'win32') {
await cpp.exec(`powershell.exe Copy-Item ${source} -Destination ${destination} -Recurse`); await cpp.exec(`powershell.exe Copy-Item "${source}" -Destination "${destination}" -Recurse`);
} else { } else {
await cpp.exec(`cp -r ${source} ${destination}`); await cpp.exec(`cp -r '${source}' '${destination}'`);
} }
return Promise.resolve(); return Promise.resolve();
...@@ -101,9 +101,9 @@ export async function execCopydir(source: string, destination: string): Promise< ...@@ -101,9 +101,9 @@ export async function execCopydir(source: string, destination: string): Promise<
*/ */
export async function execNewFile(filename: string): Promise<void> { export async function execNewFile(filename: string): Promise<void> {
if (process.platform === 'win32') { if (process.platform === 'win32') {
await cpp.exec(`powershell.exe New-Item -Path ${filename} -ItemType "file" -Force`); await cpp.exec(`powershell.exe New-Item -Path "${filename}" -ItemType "file" -Force`);
} else { } else {
await cpp.exec(`touch ${filename}`); await cpp.exec(`touch '${filename}'`);
} }
return Promise.resolve(); return Promise.resolve();
...@@ -115,9 +115,9 @@ export async function execNewFile(filename: string): Promise<void> { ...@@ -115,9 +115,9 @@ export async function execNewFile(filename: string): Promise<void> {
*/ */
export function runScript(filePath: string): cp.ChildProcess { export function runScript(filePath: string): cp.ChildProcess {
if (process.platform === 'win32') { if (process.platform === 'win32') {
return cp.exec(`powershell.exe -ExecutionPolicy Bypass -file ${filePath}`); return cp.exec(`powershell.exe -ExecutionPolicy Bypass -file "${filePath}"`);
} else { } else {
return cp.exec(`bash ${filePath}`); return cp.exec(`bash '${filePath}'`);
} }
} }
...@@ -128,9 +128,9 @@ export function runScript(filePath: string): cp.ChildProcess { ...@@ -128,9 +128,9 @@ export function runScript(filePath: string): cp.ChildProcess {
export async function execTail(filePath: string): Promise<cpp.childProcessPromise.Result> { export async function execTail(filePath: string): Promise<cpp.childProcessPromise.Result> {
let cmdresult: cpp.childProcessPromise.Result; let cmdresult: cpp.childProcessPromise.Result;
if (process.platform === 'win32') { if (process.platform === 'win32') {
cmdresult = await cpp.exec(`powershell.exe Get-Content ${filePath} -Tail 1`); cmdresult = await cpp.exec(`powershell.exe Get-Content "${filePath}" -Tail 1`);
} else { } else {
cmdresult = await cpp.exec(`tail -n 1 ${filePath}`); cmdresult = await cpp.exec(`tail -n 1 '${filePath}'`);
} }
return Promise.resolve(cmdresult); return Promise.resolve(cmdresult);
...@@ -142,9 +142,9 @@ export async function execTail(filePath: string): Promise<cpp.childProcessPromis ...@@ -142,9 +142,9 @@ export async function execTail(filePath: string): Promise<cpp.childProcessPromis
*/ */
export async function execRemove(directory: string): Promise<void> { export async function execRemove(directory: string): Promise<void> {
if (process.platform === 'win32') { if (process.platform === 'win32') {
await cpp.exec(`powershell.exe Remove-Item ${directory} -Recurse -Force`); await cpp.exec(`powershell.exe Remove-Item "${directory}" -Recurse -Force`);
} else { } else {
await cpp.exec(`rm -rf ${directory}`); await cpp.exec(`rm -rf '${directory}'`);
} }
return Promise.resolve(); return Promise.resolve();
...@@ -173,7 +173,7 @@ export function setEnvironmentVariable(variable: { key: string; value: string }) ...@@ -173,7 +173,7 @@ export function setEnvironmentVariable(variable: { key: string; value: string })
if (process.platform === 'win32') { if (process.platform === 'win32') {
return `$env:${variable.key}="${variable.value}"`; return `$env:${variable.key}="${variable.value}"`;
} else { } else {
return `export ${variable.key}=${variable.value}`; return `export ${variable.key}='${variable.value}'`;
} }
} }
......
src/nni_manager/training_service/local/localTrainingService.ts
View file @ 7620e7c5
...@@ -490,18 +490,18 @@ class LocalTrainingService implements TrainingService { ...@@ -490,18 +490,18 @@ class LocalTrainingService implements TrainingService {
const script: string[] = []; const script: string[] = [];
if (process.platform === 'win32') { if (process.platform === 'win32') {
script.push( script.push(
`cmd.exe /c ${localTrialConfig.command} 2>${path.join(workingDirectory, 'stderr')}`, `cmd.exe /c ${localTrialConfig.command} 2>"${path.join(workingDirectory, 'stderr')}"`,
`$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds`, `$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds`,
`$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()`, `$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()`,
`Write $LASTEXITCODE " " $NOW_DATE | Out-File ${path.join(workingDirectory, '.nni', 'state')} -NoNewline -encoding utf8`); `Write $LASTEXITCODE " " $NOW_DATE | Out-File "${path.join(workingDirectory, '.nni', 'state')}" -NoNewline -encoding utf8`);
} else { } else {
script.push(`eval ${localTrialConfig.command} 2>${path.join(workingDirectory, 'stderr')}`); script.push(`eval ${localTrialConfig.command} 2>"${path.join(workingDirectory, 'stderr')}"`);
if (process.platform === 'darwin') { if (process.platform === 'darwin') {
// https://superuser.com/questions/599072/how-to-get-bash-execution-time-in-milliseconds-under-mac-os-x // https://superuser.com/questions/599072/how-to-get-bash-execution-time-in-milliseconds-under-mac-os-x
// Considering the worst case, write 999 to avoid negative duration // Considering the worst case, write 999 to avoid negative duration
script.push(`echo $? \`date +%s999\` >${path.join(workingDirectory, '.nni', 'state')}`); script.push(`echo $? \`date +%s999\` >'${path.join(workingDirectory, '.nni', 'state')}'`);
} else { } else {
script.push(`echo $? \`date +%s%3N\` >${path.join(workingDirectory, '.nni', 'state')}`); script.push(`echo $? \`date +%s%3N\` >'${path.join(workingDirectory, '.nni', 'state')}'`);
} }
} }
...@@ -522,7 +522,7 @@ class LocalTrainingService implements TrainingService { ...@@ -522,7 +522,7 @@ class LocalTrainingService implements TrainingService {
if (process.platform !== 'win32') { if (process.platform !== 'win32') {
runScriptContent.push('#!/bin/bash'); runScriptContent.push('#!/bin/bash');
} }
runScriptContent.push(`cd ${this.localTrialConfig.codeDir}`); runScriptContent.push(`cd '${this.localTrialConfig.codeDir}'`);
for (const variable of variables) { for (const variable of variables) {
runScriptContent.push(setEnvironmentVariable(variable)); runScriptContent.push(setEnvironmentVariable(variable));
} }
......
src/sdk/pynni/nni/batch_tuner/batch_tuner.py
View file @ 7620e7c5
...@@ -31,22 +31,27 @@ TYPE = '_type' ...@@ -31,22 +31,27 @@ TYPE = '_type'
CHOICE = 'choice' CHOICE = 'choice'
VALUE = '_value' VALUE = '_value'
logger = logging.getLogger('batch_tuner_AutoML') LOGGER = logging.getLogger('batch_tuner_AutoML')
class BatchTuner(Tuner): class BatchTuner(Tuner):
""" """
BatchTuner is tuner will running all the configure that user want to run batchly. BatchTuner is tuner will running all the configure that user want to run batchly.
Examples
--------
The search space only be accepted like: The search space only be accepted like:
```
{ {
'combine_params': { '_type': 'choice', 'combine_params': { '_type': 'choice',
'_value': '[{...}, {...}, {...}]', '_value': '[{...}, {...}, {...}]',
} }
} }
```
""" """
def __init__(self): def __init__(self):
self.count = -1 self._count = -1
self.values = [] self._values = []
def is_valid(self, search_space): def is_valid(self, search_space):
""" """
...@@ -55,6 +60,11 @@ class BatchTuner(Tuner): ...@@ -55,6 +60,11 @@ class BatchTuner(Tuner):
Parameters Parameters
---------- ----------
search_space : dict search_space : dict
Returns
-------
None or list
If valid, return candidate values; else return None.
""" """
if not len(search_space) == 1: if not len(search_space) == 1:
raise RuntimeError('BatchTuner only supprt one combined-paramreters key.') raise RuntimeError('BatchTuner only supprt one combined-paramreters key.')
...@@ -62,11 +72,14 @@ class BatchTuner(Tuner): ...@@ -62,11 +72,14 @@ class BatchTuner(Tuner):
for param in search_space: for param in search_space:
param_type = search_space[param][TYPE] param_type = search_space[param][TYPE]
if not param_type == CHOICE: if not param_type == CHOICE:
raise RuntimeError('BatchTuner only supprt one combined-paramreters type is choice.') raise RuntimeError('BatchTuner only supprt \
else: one combined-paramreters type is choice.')
if isinstance(search_space[param][VALUE], list): if isinstance(search_space[param][VALUE], list):
return search_space[param][VALUE] return search_space[param][VALUE]
raise RuntimeError('The combined-paramreters value in BatchTuner is not a list.')
raise RuntimeError('The combined-paramreters \
value in BatchTuner is not a list.')
return None return None
def update_search_space(self, search_space): def update_search_space(self, search_space):
...@@ -76,7 +89,7 @@ class BatchTuner(Tuner): ...@@ -76,7 +89,7 @@ class BatchTuner(Tuner):
---------- ----------
search_space : dict search_space : dict
""" """
self.values = self.is_valid(search_space) self._values = self.is_valid(search_space)
def generate_parameters(self, parameter_id, **kwargs): def generate_parameters(self, parameter_id, **kwargs):
"""Returns a dict of trial (hyper-)parameters, as a serializable object. """Returns a dict of trial (hyper-)parameters, as a serializable object.
...@@ -84,41 +97,49 @@ class BatchTuner(Tuner): ...@@ -84,41 +97,49 @@ class BatchTuner(Tuner):
Parameters Parameters
---------- ----------
parameter_id : int parameter_id : int
Returns
-------
dict
A candidate parameter group.
""" """
self.count += 1 self._count += 1
if self.count > len(self.values) - 1: if self._count > len(self._values) - 1:
raise nni.NoMoreTrialError('no more parameters now.') raise nni.NoMoreTrialError('no more parameters now.')
return self.values[self.count] return self._values[self._count]
def receive_trial_result(self, parameter_id, parameters, value, **kwargs): def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
pass pass
def import_data(self, data): def import_data(self, data):
"""Import additional data for tuning """Import additional data for tuning
Parameters Parameters
---------- ----------
data: data:
a list of dictionarys, each of which has at least two keys, 'parameter' and 'value' a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
""" """
if not self.values: if not self._values:
logger.info("Search space has not been initialized, skip this data import") LOGGER.info("Search space has not been initialized, skip this data import")
return return
self.values = self.values[(self.count+1):] self._values = self._values[(self._count+1):]
self.count = -1 self._count = -1
_completed_num = 0 _completed_num = 0
for trial_info in data: for trial_info in data:
logger.info("Importing data, current processing progress %s / %s", _completed_num, len(data)) LOGGER .info("Importing data, current processing \
progress %s / %s", _completed_num, len(data))
# simply validate data format # simply validate data format
assert "parameter" in trial_info assert "parameter" in trial_info
_params = trial_info["parameter"] _params = trial_info["parameter"]
assert "value" in trial_info assert "value" in trial_info
_value = trial_info['value'] _value = trial_info['value']
if not _value: if not _value:
logger.info("Useless trial data, value is %s, skip this trial data.", _value) LOGGER.info("Useless trial data, value is %s, skip this trial data.", _value)
continue continue
_completed_num += 1 _completed_num += 1
if _params in self.values: if _params in self._values:
self.values.remove(_params) self._values.remove(_params)
logger.info("Successfully import data to batch tuner, total data: %d, imported data: %d.", len(data), _completed_num) LOGGER .info("Successfully import data to batch tuner, \
total data: %d, imported data: %d.", len(data), _completed_num)
src/sdk/pynni/nni/compression/torch/compressor.py
View file @ 7620e7c5
...@@ -32,21 +32,34 @@ class Compressor: ...@@ -32,21 +32,34 @@ class Compressor:
""" """
self.bound_model = model self.bound_model = model
self.config_list = config_list self.config_list = config_list
self.modules_to_compress = [] self.modules_to_compress = None
def compress(self): def detect_modules_to_compress(self):
""" """
Compress the model with algorithm implemented by subclass. detect all modules should be compressed, and save the result in `self.modules_to_compress`.
The model will be instrumented and user should never edit it after calling this method. The model will be instrumented and user should never edit it after calling this method.
`self.modules_to_compress` records all the to-be-compressed layers
""" """
if self.modules_to_compress is None:
self.modules_to_compress = []
for name, module in self.bound_model.named_modules(): for name, module in self.bound_model.named_modules():
layer = LayerInfo(name, module) layer = LayerInfo(name, module)
config = self.select_config(layer) config = self.select_config(layer)
if config is not None: if config is not None:
self._instrument_layer(layer, config)
self.modules_to_compress.append((layer, config)) self.modules_to_compress.append((layer, config))
return self.modules_to_compress
def compress(self):
"""
Compress the model with algorithm implemented by subclass.
The model will be instrumented and user should never edit it after calling this method.
`self.modules_to_compress` records all the to-be-compressed layers
"""
modules_to_compress = self.detect_modules_to_compress()
for layer, config in modules_to_compress:
self._instrument_layer(layer, config)
return self.bound_model return self.bound_model
def get_modules_to_compress(self): def get_modules_to_compress(self):
...@@ -55,7 +68,7 @@ class Compressor: ...@@ -55,7 +68,7 @@ class Compressor:
Returns Returns
------- -------
self.modules_to_compress : list list
a list of the layers, each of which is a tuple (`layer`, `config`), a list of the layers, each of which is a tuple (`layer`, `config`),
`layer` is `LayerInfo`, `config` is a `dict` `layer` is `LayerInfo`, `config` is a `dict`
""" """
...@@ -72,7 +85,7 @@ class Compressor: ...@@ -72,7 +85,7 @@ class Compressor:
Returns Returns
------- -------
ret : config or None config or None
the retrieved configuration for this layer, if None, this layer should the retrieved configuration for this layer, if None, this layer should
not be compressed not be compressed
""" """
...@@ -206,6 +219,8 @@ class Pruner(Compressor): ...@@ -206,6 +219,8 @@ class Pruner(Compressor):
""" """
assert model_path is not None, 'model_path must be specified' assert model_path is not None, 'model_path must be specified'
for name, m in self.bound_model.named_modules(): for name, m in self.bound_model.named_modules():
if name == "":
continue
mask = self.mask_dict.get(name) mask = self.mask_dict.get(name)
if mask is not None: if mask is not None:
mask_sum = mask.sum().item() mask_sum = mask.sum().item()
...@@ -238,26 +253,87 @@ class Quantizer(Compressor): ...@@ -238,26 +253,87 @@ class Quantizer(Compressor):
""" """
def quantize_weight(self, weight, config, op, op_type, op_name): def quantize_weight(self, weight, config, op, op_type, op_name):
"""user should know where dequantize goes and implement it in quantize method """
we now do not provide dequantize method quantize should overload this method to quantize weight.
This method is effectively hooked to :meth:`forward` of the model.
Parameters
----------
weight : Tensor
weight that needs to be quantized
config : dict
the configuration for weight quantization
""" """
raise NotImplementedError("Quantizer must overload quantize_weight()") raise NotImplementedError("Quantizer must overload quantize_weight()")
def quantize_output(self, output, config, op, op_type, op_name):
"""
quantize should overload this method to quantize output.
This method is effectively hooked to :meth:`forward` of the model.
Parameters
----------
output : Tensor
output that needs to be quantized
config : dict
the configuration for output quantization
"""
raise NotImplementedError("Quantizer must overload quantize_output()")
def quantize_input(self, *inputs, config, op, op_type, op_name):
"""
quantize should overload this method to quantize input.
This method is effectively hooked to :meth:`forward` of the model.
Parameters
----------
inputs : Tensor
inputs that needs to be quantized
config : dict
the configuration for inputs quantization
"""
raise NotImplementedError("Quantizer must overload quantize_input()")
def _instrument_layer(self, layer, config): def _instrument_layer(self, layer, config):
"""
Create a wrapper forward function to replace the original one.
Parameters
----------
layer : LayerInfo
the layer to instrument the mask
config : dict
the configuration for quantization
"""
assert layer._forward is None, 'Each model can only be compressed once' assert layer._forward is None, 'Each model can only be compressed once'
assert "quant_types" in config, 'must provide quant_types in config'
assert isinstance(config["quant_types"], list), 'quant_types must be list type'
if 'weight' in config["quant_types"]:
if not _check_weight(layer.module): if not _check_weight(layer.module):
_logger.warning('Module %s does not have parameter "weight"', layer.name) _logger.warning('Module %s does not have parameter "weight"', layer.name)
return
layer._forward = layer.module.forward layer._forward = layer.module.forward
def new_forward(*inputs): def new_forward(*inputs):
if 'input' in config["quant_types"]:
inputs = self.quantize_input(inputs, config=config, op=layer.module, op_type=layer.type, op_name=layer.name)
if 'weight' in config["quant_types"] and _check_weight(layer.module):
weight = layer.module.weight.data weight = layer.module.weight.data
new_weight = self.quantize_weight(weight, config, op=layer.module, op_type=layer.type, op_name=layer.name) new_weight = self.quantize_weight(weight, config, op=layer.module, op_type=layer.type, op_name=layer.name)
layer.module.weight.data = new_weight layer.module.weight.data = new_weight
return layer._forward(*inputs) result = layer._forward(*inputs)
layer.module.weight.data = weight
else:
result = layer._forward(*inputs)
layer.module.forward = new_forward if 'output' in config["quant_types"]:
result = self.quantize_output(result, config, op=layer.module, op_type=layer.type, op_name=layer.name)
return result
layer.module.forward = new_forward
def _check_weight(module): def _check_weight(module):
try: try:
......
src/sdk/pynni/nni/curvefitting_assessor/curvefitting_assessor.py
View file @ 7620e7c5
...@@ -29,13 +29,13 @@ class CurvefittingAssessor(Assessor): ...@@ -29,13 +29,13 @@ class CurvefittingAssessor(Assessor):
Parameters Parameters
---------- ----------
epoch_num: int epoch_num : int
The total number of epoch The total number of epoch
optimize_mode: str optimize_mode : str
optimize mode, 'maximize' or 'minimize' optimize mode, 'maximize' or 'minimize'
start_step: int start_step : int
only after receiving start_step number of reported intermediate results only after receiving start_step number of reported intermediate results
threshold: float threshold : float
The threshold that we decide to early stop the worse performance curve. The threshold that we decide to early stop the worse performance curve.
""" """
def __init__(self, epoch_num=20, optimize_mode='maximize', start_step=6, threshold=0.95, gap=1): def __init__(self, epoch_num=20, optimize_mode='maximize', start_step=6, threshold=0.95, gap=1):
...@@ -70,9 +70,9 @@ class CurvefittingAssessor(Assessor): ...@@ -70,9 +70,9 @@ class CurvefittingAssessor(Assessor):
Parameters Parameters
---------- ----------
trial_job_id: int trial_job_id : int
trial job id trial job id
success: bool success : bool
True if succssfully finish the experiment, False otherwise True if succssfully finish the experiment, False otherwise
""" """
if success: if success:
...@@ -90,9 +90,9 @@ class CurvefittingAssessor(Assessor): ...@@ -90,9 +90,9 @@ class CurvefittingAssessor(Assessor):
Parameters Parameters
---------- ----------
trial_job_id: int trial_job_id : int
trial job id trial job id
trial_history: list trial_history : list
The history performance matrix of each trial The history performance matrix of each trial
Returns Returns
...@@ -105,7 +105,6 @@ class CurvefittingAssessor(Assessor): ...@@ -105,7 +105,6 @@ class CurvefittingAssessor(Assessor):
Exception Exception
unrecognize exception in curvefitting_assessor unrecognize exception in curvefitting_assessor
""" """
trial_job_id = trial_job_id
self.trial_history = trial_history self.trial_history = trial_history
if not self.set_best_performance: if not self.set_best_performance:
return AssessResult.Good return AssessResult.Good
......
src/sdk/pynni/nni/curvefitting_assessor/curvefunctions.py
View file @ 7620e7c5
...@@ -14,7 +14,9 @@ ...@@ -14,7 +14,9 @@
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
A family of functions used by CurvefittingAssessor
"""
import numpy as np import numpy as np
all_models = {} all_models = {}
...@@ -29,10 +31,10 @@ def vap(x, a, b, c): ...@@ -29,10 +31,10 @@ def vap(x, a, b, c):
Parameters Parameters
---------- ----------
x: int x : int
a: float a : float
b: float b : float
c: float c : float
Returns Returns
------- -------
...@@ -50,10 +52,10 @@ def pow3(x, c, a, alpha): ...@@ -50,10 +52,10 @@ def pow3(x, c, a, alpha):
Parameters Parameters
---------- ----------
x: int x : int
c: float c : float
a: float a : float
alpha: float alpha : float
Returns Returns
------- -------
...@@ -71,9 +73,9 @@ def linear(x, a, b): ...@@ -71,9 +73,9 @@ def linear(x, a, b):
Parameters Parameters
---------- ----------
x: int x : int
a: float a : float
b: float b : float
Returns Returns
------- -------
...@@ -91,9 +93,9 @@ def logx_linear(x, a, b): ...@@ -91,9 +93,9 @@ def logx_linear(x, a, b):
Parameters Parameters
---------- ----------
x: int x : int
a: float a : float
b: float b : float
Returns Returns
------- -------
...@@ -112,10 +114,10 @@ def dr_hill_zero_background(x, theta, eta, kappa): ...@@ -112,10 +114,10 @@ def dr_hill_zero_background(x, theta, eta, kappa):
Parameters Parameters
---------- ----------
x: int x : int
theta: float theta : float
eta: float eta : float
kappa: float kappa : float
Returns Returns
------- -------
...@@ -133,10 +135,10 @@ def log_power(x, a, b, c): ...@@ -133,10 +135,10 @@ def log_power(x, a, b, c):
Parameters Parameters
---------- ----------
x: int x : int
a: float a : float
b: float b : float
c: float c : float
Returns Returns
------- -------
...@@ -154,11 +156,11 @@ def pow4(x, alpha, a, b, c): ...@@ -154,11 +156,11 @@ def pow4(x, alpha, a, b, c):
Parameters Parameters
---------- ----------
x: int x : int
alpha: float alpha : float
a: float a : float
b: float b : float
c: float c : float
Returns Returns
------- -------
...@@ -177,11 +179,11 @@ def mmf(x, alpha, beta, kappa, delta): ...@@ -177,11 +179,11 @@ def mmf(x, alpha, beta, kappa, delta):
Parameters Parameters
---------- ----------
x: int x : int
alpha: float alpha : float
beta: float beta : float
kappa: float kappa : float
delta: float delta : float
Returns Returns
------- -------
...@@ -199,11 +201,11 @@ def exp4(x, c, a, b, alpha): ...@@ -199,11 +201,11 @@ def exp4(x, c, a, b, alpha):
Parameters Parameters
---------- ----------
x: int x : int
c: float c : float
a: float a : float
b: float b : float
alpha: float alpha : float
Returns Returns
------- -------
...@@ -221,9 +223,9 @@ def ilog2(x, c, a): ...@@ -221,9 +223,9 @@ def ilog2(x, c, a):
Parameters Parameters
---------- ----------
x: int x : int
c: float c : float
a: float a : float
Returns Returns
------- -------
...@@ -242,11 +244,11 @@ def weibull(x, alpha, beta, kappa, delta): ...@@ -242,11 +244,11 @@ def weibull(x, alpha, beta, kappa, delta):
Parameters Parameters
---------- ----------
x: int x : int
alpha: float alpha : float
beta: float beta : float
kappa: float kappa : float
delta: float delta : float
Returns Returns
------- -------
...@@ -264,11 +266,11 @@ def janoschek(x, a, beta, k, delta): ...@@ -264,11 +266,11 @@ def janoschek(x, a, beta, k, delta):
Parameters Parameters
---------- ----------
x: int x : int
a: float a : float
beta: float beta : float
k: float k : float
delta: float delta : float
Returns Returns
------- -------
......
src/sdk/pynni/nni/curvefitting_assessor/model_factory.py
View file @ 7620e7c5
...@@ -40,7 +40,7 @@ class CurveModel: ...@@ -40,7 +40,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
target_pos: int target_pos : int
The point we need to predict The point we need to predict
""" """
def __init__(self, target_pos): def __init__(self, target_pos):
...@@ -120,14 +120,14 @@ class CurveModel: ...@@ -120,14 +120,14 @@ class CurveModel:
Parameters Parameters
---------- ----------
model: string model : string
name of the curve function model name of the curve function model
pos: int pos : int
the epoch number of the position you want to predict the epoch number of the position you want to predict
Returns Returns
------- -------
int: int
The expected matrix at pos The expected matrix at pos
""" """
if model_para_num[model] == 2: if model_para_num[model] == 2:
...@@ -143,9 +143,9 @@ class CurveModel: ...@@ -143,9 +143,9 @@ class CurveModel:
Parameters Parameters
---------- ----------
pos: int pos : int
the epoch number of the position you want to predict the epoch number of the position you want to predict
sample: list sample : list
sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk} sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
Returns Returns
...@@ -165,7 +165,7 @@ class CurveModel: ...@@ -165,7 +165,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
samples: list samples : list
a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix, a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}} representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
...@@ -187,7 +187,7 @@ class CurveModel: ...@@ -187,7 +187,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
sample: list sample : list
sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk} sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
Returns Returns
...@@ -206,9 +206,9 @@ class CurveModel: ...@@ -206,9 +206,9 @@ class CurveModel:
Parameters Parameters
---------- ----------
pos: int pos : int
the epoch number of the position you want to predict the epoch number of the position you want to predict
sample: list sample : list
sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk} sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
Returns Returns
...@@ -225,7 +225,7 @@ class CurveModel: ...@@ -225,7 +225,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
sample: list sample : list
sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk} sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
Returns Returns
...@@ -244,7 +244,7 @@ class CurveModel: ...@@ -244,7 +244,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
samples: list samples : list
a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix, a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}} representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
...@@ -267,7 +267,7 @@ class CurveModel: ...@@ -267,7 +267,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
samples: list samples : list
a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix, a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}} representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
...@@ -322,7 +322,7 @@ class CurveModel: ...@@ -322,7 +322,7 @@ class CurveModel:
Parameters Parameters
---------- ----------
trial_history: list trial_history : list
The history performance matrix of each trial. The history performance matrix of each trial.
Returns Returns
......
src/sdk/pynni/nni/evolution_tuner/evolution_tuner.py
View file @ 7620e7c5
...@@ -32,7 +32,9 @@ import nni.parameter_expressions as parameter_expressions ...@@ -32,7 +32,9 @@ import nni.parameter_expressions as parameter_expressions
def json2space(x, oldy=None, name=NodeType.ROOT): def json2space(x, oldy=None, name=NodeType.ROOT):
"""Change search space from json format to hyperopt format """
Change search space from json format to hyperopt format
""" """
y = list() y = list()
if isinstance(x, dict): if isinstance(x, dict):
...@@ -59,7 +61,9 @@ def json2space(x, oldy=None, name=NodeType.ROOT): ...@@ -59,7 +61,9 @@ def json2space(x, oldy=None, name=NodeType.ROOT):
return y return y
def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeType.ROOT): def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeType.ROOT):
"""Json to pramaters. """
Json to pramaters.
""" """
if isinstance(x, dict): if isinstance(x, dict):
if NodeType.TYPE in x.keys(): if NodeType.TYPE in x.keys():
...@@ -117,6 +121,17 @@ def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeTyp ...@@ -117,6 +121,17 @@ def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeTyp
class Individual: class Individual:
""" """
Indicidual class to store the indv info. Indicidual class to store the indv info.
Attributes
----------
config : str
Search space.
info : str
The str to save information of individual.
result : float
The final metric of a individual.
store_dir : str
save_dir : str
""" """
def __init__(self, config=None, info=None, result=None, save_dir=None): def __init__(self, config=None, info=None, result=None, save_dir=None):
...@@ -124,6 +139,7 @@ class Individual: ...@@ -124,6 +139,7 @@ class Individual:
Parameters Parameters
---------- ----------
config : str config : str
A config to represent a group of parameters.
info : str info : str
result : float result : float
save_dir : str save_dir : str
...@@ -140,6 +156,8 @@ class Individual: ...@@ -140,6 +156,8 @@ class Individual:
def mutation(self, config=None, info=None, save_dir=None): def mutation(self, config=None, info=None, save_dir=None):
""" """
Mutation by reset state information.
Parameters Parameters
---------- ----------
config : str config : str
...@@ -177,8 +195,11 @@ class EvolutionTuner(Tuner): ...@@ -177,8 +195,11 @@ class EvolutionTuner(Tuner):
self.population = None self.population = None
self.space = None self.space = None
def update_search_space(self, search_space): def update_search_space(self, search_space):
"""Update search space. """
Update search space.
Search_space contains the information that user pre-defined. Search_space contains the information that user pre-defined.
Parameters Parameters
...@@ -191,15 +212,19 @@ class EvolutionTuner(Tuner): ...@@ -191,15 +212,19 @@ class EvolutionTuner(Tuner):
self.random_state = np.random.RandomState() self.random_state = np.random.RandomState()
self.population = [] self.population = []
is_rand = dict() is_rand = dict()
for item in self.space: for item in self.space:
is_rand[item] = True is_rand[item] = True
for _ in range(self.population_size): for _ in range(self.population_size):
config = json2parameter( config = json2parameter(
self.searchspace_json, is_rand, self.random_state) self.searchspace_json, is_rand, self.random_state)
self.population.append(Individual(config=config)) self.population.append(Individual(config=config))
def generate_parameters(self, parameter_id, **kwargs): def generate_parameters(self, parameter_id, **kwargs):
"""Returns a dict of trial (hyper-)parameters, as a serializable object. """
This function will returns a dict of trial (hyper-)parameters, as a serializable object.
Parameters Parameters
---------- ----------
...@@ -207,15 +232,19 @@ class EvolutionTuner(Tuner): ...@@ -207,15 +232,19 @@ class EvolutionTuner(Tuner):
Returns Returns
------- -------
config : dict dict
A group of candaidte parameters that evolution tuner generated.
""" """
if not self.population: if not self.population:
raise RuntimeError('The population is empty') raise RuntimeError('The population is empty')
pos = -1 pos = -1
for i in range(len(self.population)): for i in range(len(self.population)):
if self.population[i].result is None: if self.population[i].result is None:
pos = i pos = i
break break
if pos != -1: if pos != -1:
indiv = copy.deepcopy(self.population[pos]) indiv = copy.deepcopy(self.population[pos])
self.population.pop(pos) self.population.pop(pos)
...@@ -230,6 +259,7 @@ class EvolutionTuner(Tuner): ...@@ -230,6 +259,7 @@ class EvolutionTuner(Tuner):
self.population[0].config) self.population[0].config)
is_rand = dict() is_rand = dict()
mutation_pos = space[random.randint(0, len(space)-1)] mutation_pos = space[random.randint(0, len(space)-1)]
for i in range(len(self.space)): for i in range(len(self.space)):
is_rand[self.space[i]] = (self.space[i] == mutation_pos) is_rand[self.space[i]] = (self.space[i] == mutation_pos)
config = json2parameter( config = json2parameter(
...@@ -238,21 +268,27 @@ class EvolutionTuner(Tuner): ...@@ -238,21 +268,27 @@ class EvolutionTuner(Tuner):
# remove "_index" from config and save params-id # remove "_index" from config and save params-id
total_config = config total_config = config
self.total_data[parameter_id] = total_config self.total_data[parameter_id] = total_config
config = split_index(total_config) config = split_index(total_config)
return config return config
def receive_trial_result(self, parameter_id, parameters, value, **kwargs): def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
'''Record the result from a trial """
Record the result from a trial
Parameters Parameters
---------- ----------
parameters: dict parameter_id : int
parameters : dict
value : dict/float value : dict/float
if value is dict, it should have "default" key. if value is dict, it should have "default" key.
value is final metrics of the trial. value is final metrics of the trial.
''' """
reward = extract_scalar_reward(value) reward = extract_scalar_reward(value)
if parameter_id not in self.total_data: if parameter_id not in self.total_data:
raise RuntimeError('Received parameter_id not in total_data.') raise RuntimeError('Received parameter_id not in total_data.')
# restore the paramsters contains "_index" # restore the paramsters contains "_index"
......
src/sdk/pynni/nni/gp_tuner/gp_tuner.py
View file @ 7620e7c5
...@@ -17,9 +17,11 @@ ...@@ -17,9 +17,11 @@
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
''' """
gp_tuner.py GPTuner is a Bayesian Optimization method where Gaussian Process is used for modeling loss functions.
'''
See :class:`GPTuner` for details.
"""
import warnings import warnings
import logging import logging
...@@ -38,18 +40,40 @@ logger = logging.getLogger("GP_Tuner_AutoML") ...@@ -38,18 +40,40 @@ logger = logging.getLogger("GP_Tuner_AutoML")
class GPTuner(Tuner): class GPTuner(Tuner):
''' """
GPTuner GPTuner is a Bayesian Optimization method where Gaussian Process is used for modeling loss functions.
'''
Parameters
----------
optimize_mode : str
optimize mode, 'maximize' or 'minimize', by default 'maximize'
utility : str
utility function (also called 'acquisition funcition') to use, which can be 'ei', 'ucb' or 'poi'. By default 'ei'.
kappa : float
value used by utility function 'ucb'. The bigger kappa is, the more the tuner will be exploratory. By default 5.
xi : float
used by utility function 'ei' and 'poi'. The bigger xi is, the more the tuner will be exploratory. By default 0.
nu : float
used to specify Matern kernel. The smaller nu, the less smooth the approximated function is. By default 2.5.
alpha : float
Used to specify Gaussian Process Regressor. Larger values correspond to increased noise level in the observations.
By default 1e-6.
cold_start_num : int
Number of random exploration to perform before Gaussian Process. By default 10.
selection_num_warm_up : int
Number of random points to evaluate for getting the point which maximizes the acquisition function. By default 100000
selection_num_starting_points : int
Number of times to run L-BFGS-B from a random starting point after the warmup. By default 250.
"""
def __init__(self, optimize_mode="maximize", utility='ei', kappa=5, xi=0, nu=2.5, alpha=1e-6, cold_start_num=10, def __init__(self, optimize_mode="maximize", utility='ei', kappa=5, xi=0, nu=2.5, alpha=1e-6, cold_start_num=10,
selection_num_warm_up=100000, selection_num_starting_points=250): selection_num_warm_up=100000, selection_num_starting_points=250):
self.optimize_mode = OptimizeMode(optimize_mode) self._optimize_mode = OptimizeMode(optimize_mode)
# utility function related # utility function related
self.utility = utility self._utility = utility
self.kappa = kappa self._kappa = kappa
self.xi = xi self._xi = xi
# target space # target space
self._space = None self._space = None
...@@ -72,30 +96,23 @@ class GPTuner(Tuner): ...@@ -72,30 +96,23 @@ class GPTuner(Tuner):
self._selection_num_starting_points = selection_num_starting_points self._selection_num_starting_points = selection_num_starting_points
# num of imported data # num of imported data
self.supplement_data_num = 0 self._supplement_data_num = 0
def update_search_space(self, search_space): def update_search_space(self, search_space):
"""Update the self.bounds and self.types by the search_space.json """
Update the self.bounds and self.types by the search_space.json file.
Parameters Override of the abstract method in :class:`~nni.tuner.Tuner`.
----------
search_space : dict
""" """
self._space = TargetSpace(search_space, self._random_state) self._space = TargetSpace(search_space, self._random_state)
def generate_parameters(self, parameter_id, **kwargs): def generate_parameters(self, parameter_id, **kwargs):
"""Generate next parameter for trial """
If the number of trial result is lower than cold start number, Method which provides one set of hyper-parameters.
gp will first randomly generate some parameters. If the number of trial result is lower than cold_start_number, GPTuner will first randomly generate some parameters.
Otherwise, choose the parameters by the Gussian Process Model Otherwise, choose the parameters by the Gussian Process Model.
Parameters
----------
parameter_id : int
Returns Override of the abstract method in :class:`~nni.tuner.Tuner`.
-------
result : dict
""" """
if self._space.len() < self._cold_start_num: if self._space.len() < self._cold_start_num:
results = self._space.random_sample() results = self._space.random_sample()
...@@ -107,7 +124,7 @@ class GPTuner(Tuner): ...@@ -107,7 +124,7 @@ class GPTuner(Tuner):
self._gp.fit(self._space.params, self._space.target) self._gp.fit(self._space.params, self._space.target)
util = UtilityFunction( util = UtilityFunction(
kind=self.utility, kappa=self.kappa, xi=self.xi) kind=self._utility, kappa=self._kappa, xi=self._xi)
results = acq_max( results = acq_max(
f_acq=util.utility, f_acq=util.utility,
...@@ -124,17 +141,13 @@ class GPTuner(Tuner): ...@@ -124,17 +141,13 @@ class GPTuner(Tuner):
return results return results
def receive_trial_result(self, parameter_id, parameters, value, **kwargs): def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
"""Tuner receive result from trial. """
Method invoked when a trial reports its final result.
Parameters Override of the abstract method in :class:`~nni.tuner.Tuner`.
----------
parameter_id : int
parameters : dict
value : dict/float
if value is dict, it should have "default" key.
""" """
value = extract_scalar_reward(value) value = extract_scalar_reward(value)
if self.optimize_mode == OptimizeMode.Minimize: if self._optimize_mode == OptimizeMode.Minimize:
value = -value value = -value
logger.info("Received trial result.") logger.info("Received trial result.")
...@@ -143,26 +156,27 @@ class GPTuner(Tuner): ...@@ -143,26 +156,27 @@ class GPTuner(Tuner):
self._space.register(parameters, value) self._space.register(parameters, value)
def import_data(self, data): def import_data(self, data):
"""Import additional data for tuning """
Parameters Import additional data for tuning.
----------
data: Override of the abstract method in :class:`~nni.tuner.Tuner`.
a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
""" """
_completed_num = 0 _completed_num = 0
for trial_info in data: for trial_info in data:
logger.info("Importing data, current processing progress %s / %s", _completed_num, len(data)) logger.info(
"Importing data, current processing progress %s / %s", _completed_num, len(data))
_completed_num += 1 _completed_num += 1
assert "parameter" in trial_info assert "parameter" in trial_info
_params = trial_info["parameter"] _params = trial_info["parameter"]
assert "value" in trial_info assert "value" in trial_info
_value = trial_info['value'] _value = trial_info['value']
if not _value: if not _value:
logger.info("Useless trial data, value is %s, skip this trial data.", _value) logger.info(
"Useless trial data, value is %s, skip this trial data.", _value)
continue continue
self.supplement_data_num += 1 self._supplement_data_num += 1
_parameter_id = '_'.join( _parameter_id = '_'.join(
["ImportData", str(self.supplement_data_num)]) ["ImportData", str(self._supplement_data_num)])
self.receive_trial_result( self.receive_trial_result(
parameter_id=_parameter_id, parameters=_params, value=_value) parameter_id=_parameter_id, parameters=_params, value=_value)
logger.info("Successfully import data to GP tuner.") logger.info("Successfully import data to GP tuner.")
src/sdk/pynni/nni/gp_tuner/target_space.py
View file @ 7620e7c5
...@@ -17,39 +17,51 @@ ...@@ -17,39 +17,51 @@
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
''' """
target_space.py Tool class to hold the param-space coordinates (X) and target values (Y).
''' """
import numpy as np import numpy as np
import nni.parameter_expressions as parameter_expressions import nni.parameter_expressions as parameter_expressions
def _hashable(params): def _hashable(params):
""" ensure that an point is hashable by a python dict """ """
Transform list params to tuple format. Ensure that an point is hashable by a python dict.
Parameters
----------
params : numpy array
array format of parameters
Returns
-------
tuple
tuple format of parameters
"""
return tuple(map(float, params)) return tuple(map(float, params))
class TargetSpace(): class TargetSpace():
""" """
Holds the param-space coordinates (X) and target values (Y) Holds the param-space coordinates (X) and target values (Y)
"""
def __init__(self, pbounds, random_state=None):
"""
Parameters Parameters
---------- ----------
pbounds : dict pbounds : dict
Dictionary with parameters names as keys and a tuple with minimum Dictionary with parameters names and legal values.
and maximum values.
random_state : int, RandomState, or None random_state : int, RandomState, or None
optionally specify a seed for a random number generator optionally specify a seed for a random number generator, by default None.
""" """
self.random_state = random_state
def __init__(self, pbounds, random_state=None):
self._random_state = random_state
# Get the name of the parameters # Get the name of the parameters
self._keys = sorted(pbounds) self._keys = sorted(pbounds)
# Create an array with parameters bounds # Create an array with parameters bounds
self._bounds = np.array( self._bounds = np.array(
[item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])] [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])]
...@@ -71,54 +83,100 @@ class TargetSpace(): ...@@ -71,54 +83,100 @@ class TargetSpace():
self._cache = {} self._cache = {}
def __contains__(self, params): def __contains__(self, params):
''' """
check if a parameter is already registered check if a parameter is already registered
'''
Parameters
----------
params : numpy array
Returns
-------
bool
True if the parameter is already registered, else false
"""
return _hashable(params) in self._cache return _hashable(params) in self._cache
def len(self): def len(self):
''' """
length of registered params and targets length of registered params and targets
'''
Returns
-------
int
"""
assert len(self._params) == len(self._target) assert len(self._params) == len(self._target)
return len(self._target) return len(self._target)
@property @property
def params(self): def params(self):
''' """
params: numpy array registered parameters
'''
Returns
-------
numpy array
"""
return self._params return self._params
@property @property
def target(self): def target(self):
''' """
target: numpy array registered target values
'''
Returns
-------
numpy array
"""
return self._target return self._target
@property @property
def dim(self): def dim(self):
''' """
dim: int dimension of parameters
length of keys
''' Returns
-------
int
"""
return len(self._keys) return len(self._keys)
@property @property
def keys(self): def keys(self):
''' """
keys: numpy array keys of parameters
'''
Returns
-------
numpy array
"""
return self._keys return self._keys
@property @property
def bounds(self): def bounds(self):
'''bounds''' """
bounds of parameters
Returns
-------
numpy array
"""
return self._bounds return self._bounds
def params_to_array(self, params): def params_to_array(self, params):
''' dict to array ''' """
dict to array
Parameters
----------
params : dict
dict format of parameters
Returns
-------
numpy array
array format of parameters
"""
try: try:
assert set(params) == set(self.keys) assert set(params) == set(self.keys)
except AssertionError: except AssertionError:
...@@ -129,11 +187,20 @@ class TargetSpace(): ...@@ -129,11 +187,20 @@ class TargetSpace():
return np.asarray([params[key] for key in self.keys]) return np.asarray([params[key] for key in self.keys])
def array_to_params(self, x): def array_to_params(self, x):
''' """
array to dict array to dict
maintain int type if the paramters is defined as int in search_space.json maintain int type if the paramters is defined as int in search_space.json
''' Parameters
----------
x : numpy array
array format of parameters
Returns
-------
dict
dict format of parameters
"""
try: try:
assert len(x) == len(self.keys) assert len(x) == len(self.keys)
except AssertionError: except AssertionError:
...@@ -159,15 +226,15 @@ class TargetSpace(): ...@@ -159,15 +226,15 @@ class TargetSpace():
Parameters Parameters
---------- ----------
x : dict params : dict
parameters
y : float target : float
target function value target function value
""" """
x = self.params_to_array(params) x = self.params_to_array(params)
if x in self: if x in self:
#raise KeyError('Data point {} is not unique'.format(x))
print('Data point {} is not unique'.format(x)) print('Data point {} is not unique'.format(x))
# Insert data into unique dictionary # Insert data into unique dictionary
...@@ -180,32 +247,43 @@ class TargetSpace(): ...@@ -180,32 +247,43 @@ class TargetSpace():
""" """
Creates a random point within the bounds of the space. Creates a random point within the bounds of the space.
Returns
-------
numpy array
one groupe of parameter
""" """
params = np.empty(self.dim) params = np.empty(self.dim)
for col, _bound in enumerate(self._bounds): for col, _bound in enumerate(self._bounds):
if _bound['_type'] == 'choice': if _bound['_type'] == 'choice':
params[col] = parameter_expressions.choice( params[col] = parameter_expressions.choice(
_bound['_value'], self.random_state) _bound['_value'], self._random_state)
elif _bound['_type'] == 'randint': elif _bound['_type'] == 'randint':
params[col] = self.random_state.randint( params[col] = self._random_state.randint(
_bound['_value'][0], _bound['_value'][1], size=1) _bound['_value'][0], _bound['_value'][1], size=1)
elif _bound['_type'] == 'uniform': elif _bound['_type'] == 'uniform':
params[col] = parameter_expressions.uniform( params[col] = parameter_expressions.uniform(
_bound['_value'][0], _bound['_value'][1], self.random_state) _bound['_value'][0], _bound['_value'][1], self._random_state)
elif _bound['_type'] == 'quniform': elif _bound['_type'] == 'quniform':
params[col] = parameter_expressions.quniform( params[col] = parameter_expressions.quniform(
_bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self.random_state) _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self._random_state)
elif _bound['_type'] == 'loguniform': elif _bound['_type'] == 'loguniform':
params[col] = parameter_expressions.loguniform( params[col] = parameter_expressions.loguniform(
_bound['_value'][0], _bound['_value'][1], self.random_state) _bound['_value'][0], _bound['_value'][1], self._random_state)
elif _bound['_type'] == 'qloguniform': elif _bound['_type'] == 'qloguniform':
params[col] = parameter_expressions.qloguniform( params[col] = parameter_expressions.qloguniform(
_bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self.random_state) _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self._random_state)
return params return params
def max(self): def max(self):
"""Get maximum target value found and corresponding parametes.""" """
Get maximum target value found and its corresponding parameters.
Returns
-------
dict
target value and parameters, empty dict if nothing registered
"""
try: try:
res = { res = {
'target': self.target.max(), 'target': self.target.max(),
...@@ -218,7 +296,14 @@ class TargetSpace(): ...@@ -218,7 +296,14 @@ class TargetSpace():
return res return res
def res(self): def res(self):
"""Get all target values found and corresponding parametes.""" """
Get all target values found and corresponding parameters.
Returns
-------
list
a list of target values and their corresponding parameters
"""
params = [dict(zip(self.keys, p)) for p in self.params] params = [dict(zip(self.keys, p)) for p in self.params]
return [ return [
......
src/sdk/pynni/nni/gp_tuner/util.py
View file @ 7620e7c5
...@@ -17,9 +17,9 @@ ...@@ -17,9 +17,9 @@
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
''' """
gp_tuner.py utility functions and classes for GPTuner
''' """
import warnings import warnings
import numpy as np import numpy as np
...@@ -28,9 +28,21 @@ from scipy.optimize import minimize ...@@ -28,9 +28,21 @@ from scipy.optimize import minimize
def _match_val_type(vals, bounds): def _match_val_type(vals, bounds):
''' """
Update values in the array, to match their corresponding type Update values in the array, to match their corresponding type, make sure the value is legal.
'''
Parameters
----------
vals : numpy array
values of parameters
bounds : numpy array
list of dictionary which stores parameters names and legal values.
Returns
-------
vals_new : list
The closest legal value to the original value
"""
vals_new = [] vals_new = []
for i, bound in enumerate(bounds): for i, bound in enumerate(bounds):
...@@ -52,32 +64,33 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points): ...@@ -52,32 +64,33 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points):
A function to find the maximum of the acquisition function A function to find the maximum of the acquisition function
It uses a combination of random sampling (cheap) and the 'L-BFGS-B' It uses a combination of random sampling (cheap) and the 'L-BFGS-B'
optimization method. First by sampling `n_warmup` (1e5) points at random, optimization method. First by sampling ``num_warmup`` points at random,
and then running L-BFGS-B from `n_iter` (250) random starting points. and then running L-BFGS-B from ``num_starting_points`` random starting points.
Parameters Parameters
---------- ----------
:param f_acq: f_acq : UtilityFunction.utility
The acquisition function object that return its point-wise value. The acquisition function object that return its point-wise value.
:param gp: gp : GaussianProcessRegressor
A gaussian process fitted to the relevant data. A gaussian process fitted to the relevant data.
:param y_max: y_max : float
The current maximum known value of the target function. The current maximum known value of the target function.
:param bounds: bounds : numpy array
The variables bounds to limit the search of the acq max. The variables bounds to limit the search of the acq max.
:param num_warmup: num_warmup : int
number of times to randomly sample the aquisition function number of times to randomly sample the aquisition function
:param num_starting_points: num_starting_points : int
number of times to run scipy.minimize number of times to run scipy.minimize
Returns Returns
------- -------
:return: x_max, The arg max of the acquisition function. numpy array
The parameter which achieves max of the acquisition function.
""" """
# Warm up with random points # Warm up with random points
...@@ -117,36 +130,70 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points): ...@@ -117,36 +130,70 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points):
class UtilityFunction(): class UtilityFunction():
""" """
An object to compute the acquisition functions. A class to compute different acquisition function values.
"""
def __init__(self, kind, kappa, xi): Parameters
""" ----------
If UCB is to be used, a constant kappa is needed. kind : string
specification of utility function to use
kappa : float
parameter usedd for 'ucb' acquisition function
xi : float
parameter usedd for 'ei' and 'poi' acquisition function
""" """
self.kappa = kappa
self.xi = xi def __init__(self, kind, kappa, xi):
self._kappa = kappa
self._xi = xi
if kind not in ['ucb', 'ei', 'poi']: if kind not in ['ucb', 'ei', 'poi']:
err = "The utility function " \ err = "The utility function " \
"{} has not been implemented, " \ "{} has not been implemented, " \
"please choose one of ucb, ei, or poi.".format(kind) "please choose one of ucb, ei, or poi.".format(kind)
raise NotImplementedError(err) raise NotImplementedError(err)
self.kind = kind self._kind = kind
def utility(self, x, gp, y_max): def utility(self, x, gp, y_max):
'''return utility function''' """
if self.kind == 'ucb': return utility function
return self._ucb(x, gp, self.kappa)
if self.kind == 'ei': Parameters
return self._ei(x, gp, y_max, self.xi) ----------
if self.kind == 'poi': x : numpy array
return self._poi(x, gp, y_max, self.xi) parameters
gp : GaussianProcessRegressor
y_max : float
maximum target value observed so far
Returns
-------
function
return corresponding function, return None if parameter is illegal
"""
if self._kind == 'ucb':
return self._ucb(x, gp, self._kappa)
if self._kind == 'ei':
return self._ei(x, gp, y_max, self._xi)
if self._kind == 'poi':
return self._poi(x, gp, y_max, self._xi)
return None return None
@staticmethod @staticmethod
def _ucb(x, gp, kappa): def _ucb(x, gp, kappa):
"""
Upper Confidence Bound (UCB) utility function
Parameters
----------
x : numpy array
parameters
gp : GaussianProcessRegressor
kappa : float
Returns
-------
float
"""
with warnings.catch_warnings(): with warnings.catch_warnings():
warnings.simplefilter("ignore") warnings.simplefilter("ignore")
mean, std = gp.predict(x, return_std=True) mean, std = gp.predict(x, return_std=True)
...@@ -155,6 +202,22 @@ class UtilityFunction(): ...@@ -155,6 +202,22 @@ class UtilityFunction():
@staticmethod @staticmethod
def _ei(x, gp, y_max, xi): def _ei(x, gp, y_max, xi):
"""
Expected Improvement (EI) utility function
Parameters
----------
x : numpy array
parameters
gp : GaussianProcessRegressor
y_max : float
maximum target value observed so far
xi : float
Returns
-------
float
"""
with warnings.catch_warnings(): with warnings.catch_warnings():
warnings.simplefilter("ignore") warnings.simplefilter("ignore")
mean, std = gp.predict(x, return_std=True) mean, std = gp.predict(x, return_std=True)
...@@ -164,6 +227,22 @@ class UtilityFunction(): ...@@ -164,6 +227,22 @@ class UtilityFunction():
@staticmethod @staticmethod
def _poi(x, gp, y_max, xi): def _poi(x, gp, y_max, xi):
"""
Possibility Of Improvement (POI) utility function
Parameters
----------
x : numpy array
parameters
gp : GaussianProcessRegressor
y_max : float
maximum target value observed so far
xi : float
Returns
-------
float
"""
with warnings.catch_warnings(): with warnings.catch_warnings():
warnings.simplefilter("ignore") warnings.simplefilter("ignore")
mean, std = gp.predict(x, return_std=True) mean, std = gp.predict(x, return_std=True)
......
src/sdk/pynni/nni/gridsearch_tuner/__init__.py
View file @ 7620e7c5
from .gridsearch_tuner import GridSearchTuner
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment