Merge pull request #214 from microsoft/master

merge master

README_zh_CN.md

@@ -359,4 +359,4 @@ You can use these commands to get more information about the experiment
 
 ## **许可协议**
 
-代码库遵循 [MIT 许可协议](LICENSE)
\ No newline at end of file
+代码库遵循 [MIT 许可协议](LICENSE)

azure-pipelines.yml

@@ -126,18 +126,10 @@ jobs:
       cd test
       powershell.exe -file unittest.ps1
     displayName: 'unit test'
-  - script: |
-      cd test
-      python naive_test.py
-    displayName: 'Naive test'
   - script: |
       cd test
       python tuner_test.py
     displayName: 'Built-in tuners / assessors tests'
-  - script: |
-      cd test
-      python metrics_test.py
-    displayName: 'Trial job metrics test'
   - script: |
       cd test
       PATH=$HOME/.local/bin:$PATH python3 cli_test.py

docs/en_US/CommunitySharings/TuningSystems.md

+# Automatically tune systems with NNI
+
+As computer systems and networking get increasingly complicated, optimizing them manually with explicit rules and heuristics becomes harder than ever before, sometimes impossible. Below are two examples of tuning systems with NNI. Anyone can easily tune their own systems by following them.
+
+* [Tuning RocksDB with NNI](../TrialExample/RocksdbExamples.md)
+* [Tuning parameters of SPTAG (Space Partition Tree And Graph) with NNI](SptagAutoTune.md)
+
+Please see [this paper](https://dl.acm.org/citation.cfm?id=3352031) for more details:
+
+Mike Liang, Chieh-Jan, et al. "The Case for Learning-and-System Co-design." ACM SIGOPS Operating Systems Review 53.1 (2019): 68-74.

docs/en_US/Compressor/Overview.md

@@ -180,12 +180,54 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer):
 
     def quantize_weight(self, weight, config, **kwargs):
         """
-        weight is the target weight tensor
-        config is the selected dict object in config_list for this layer
-        kwargs contains op, op_types, and op_name
-        design your quantizer and return new weight
+        quantize should overload this method to quantize weight tensors.
+        This method is effectively hooked to :meth:`forward` of the model.
+
+        Parameters
+        ----------
+        weight : Tensor
+            weight that needs to be quantized
+        config : dict
+            the configuration for weight quantization
         """
+
+        # Put your code to generate `new_weight` here
+
         return new_weight
+    
+    def quantize_output(self, output, config, **kwargs):
+        """
+        quantize should overload this method to quantize output.
+        This method is effectively hooked to `:meth:`forward` of the model.
+
+        Parameters
+        ----------
+        output : Tensor
+            output that needs to be quantized
+        config : dict
+            the configuration for output quantization
+        """
+
+        # Put your code to generate `new_output` here
+
+        return new_output
+
+    def quantize_input(self, *inputs, config, **kwargs):
+        """
+        quantize should overload this method to quantize input.
+        This method is effectively hooked to :meth:`forward` of the model.
+
+        Parameters
+        ----------
+        inputs : Tensor
+            inputs that needs to be quantized
+        config : dict
+            the configuration for inputs quantization
+        """
+
+        # Put your code to generate `new_input` here
+
+        return new_input
 
     # note for pytorch version, there is no sess in input arguments
     def update_epoch(self, epoch_num, sess):
@@ -200,8 +242,6 @@ class YourQuantizer(nni.compression.tensorflow.Quantizer):
         pass
 ```
 
-__[TODO]__ Will add another member function `quantize_layer_output`, as some quantization algorithms also quantize layers' output.
-
 ### Usage of user customized compression algorithm
 
 __[TODO]__ ...

docs/en_US/TrialExample/GbdtExample.md

@@ -44,7 +44,15 @@ Given the features and label in train data, we train a GBDT regression model and
 
 ## 3. How to run in nni
 
-### 3.1 Prepare your trial code
+
+### 3.1 Install all the requirments
+
+```
+pip install lightgbm
+pip install pandas
+```
+
+### 3.2 Prepare your trial code
 
 You need to prepare a basic code as following:
 
@@ -86,7 +94,7 @@ if __name__ == '__main__':
     run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
 ```
 
-### 3.2 Prepare your search space.
+### 3.3 Prepare your search space.
 If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `bagging_freq`, you could write a [search_space.json](https://github.com/Microsoft/nni/blob/master/examples/trials/auto-gbdt/search_space.json) as follow:
 
 ```json
@@ -100,7 +108,7 @@ If you like to tune `num_leaves`, `learning_rate`, `bagging_fraction` and `baggi
 
 More support variable type you could reference [here](../Tutorial/SearchSpaceSpec.md).
 
-### 3.3 Add SDK of nni into your code.
+### 3.4 Add SDK of nni into your code.
 
 ```diff
 +import nni
@@ -146,7 +154,7 @@ if __name__ == '__main__':
     run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
 ```
 
-### 3.4 Write a config file and run it.
+### 3.5 Write a config file and run it.
 
 In the config file, you could set some settings including:
 

docs/en_US/Tutorial/Contributing.md

@@ -40,6 +40,9 @@ A person looking to contribute can take up an issue by claiming it as a comment/
 ## Code Styles & Naming Conventions
 * We follow [PEP8](https://www.python.org/dev/peps/pep-0008/) for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as `flake8` or `pylint`
 * We also follow [NumPy Docstring Style](https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy) for Python Docstring Conventions. During the [documentation building](Contributing.md#documentation), we use [sphinx.ext.napoleon](https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html) to generate Python API documentation from Docstring.
+* For docstrings, please refer to [numpydoc docstring guide](https://numpydoc.readthedocs.io/en/latest/format.html) and [pandas docstring guide](https://python-sprints.github.io/pandas/guide/pandas_docstring.html)
+    * For function docstring, **description**, **Parameters**, and **Returns**/**Yields** are mandatory.
+    * For class docstring, **description**, **Attributes** are mandatory.
 
 ## Documentation
 Our documentation is built with [sphinx](http://sphinx-doc.org/), supporting [Markdown](https://guides.github.com/features/mastering-markdown/) and [reStructuredText](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) format. All our documentations are placed under [docs/en_US](https://github.com/Microsoft/nni/tree/master/docs).
@@ -48,4 +51,4 @@ Our documentation is built with [sphinx](http://sphinx-doc.org/), supporting [Ma
 
 * For links, please consider using __relative paths__ first. However, if the documentation is written in Markdown format, and:
     * It's an image link which needs to be formatted with embedded html grammar, please use global URL like `https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png`, which can be automatically generated by dragging picture onto [Github Issue](https://github.com/Microsoft/nni/issues/new) Box.
-    * It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at `https://github.com/Microsoft/nni/tree/master/` ([mnist.py](https://github.com/Microsoft/nni/blob/master/examples/trials/mnist/mnist.py) for example).
\ No newline at end of file
+    * It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at `https://github.com/Microsoft/nni/tree/master/` ([mnist.py](https://github.com/Microsoft/nni/blob/master/examples/trials/mnist/mnist.py) for example).

docs/en_US/sdk_reference.rst

@@ -24,10 +24,10 @@ Tuner
 ..  autoclass:: nni.evolution_tuner.evolution_tuner.EvolutionTuner
     :members:
 
-..  autoclass:: nni.smac_tuner.smac_tuner.SMACTuner
+..  autoclass:: nni.smac_tuner.SMACTuner
     :members:
 
-..  autoclass:: nni.gridsearch_tuner.gridsearch_tuner.GridSearchTuner
+..  autoclass:: nni.gridsearch_tuner.GridSearchTuner
     :members:
 
 ..  autoclass:: nni.networkmorphism_tuner.networkmorphism_tuner.NetworkMorphismTuner
@@ -36,6 +36,15 @@ Tuner
 ..  autoclass:: nni.metis_tuner.metis_tuner.MetisTuner
     :members:
 
+..  autoclass:: nni.ppo_tuner.PPOTuner
+    :members:
+
+..  autoclass:: nni.batch_tuner.batch_tuner.BatchTuner
+    :members:
+
+..  autoclass:: nni.gp_tuner.gp_tuner.GPTuner
+    :members:
+
 Assessor
 ------------------------
 ..  autoclass:: nni.assessor.Assessor

src/nni_manager/common/experimentStartupInfo.ts

@@ -43,7 +43,7 @@ class ExperimentStartupInfo {
         this.initialized = true;
 
         if (logDir !== undefined && logDir.length > 0) {
-            this.logDir = path.join(logDir, getExperimentId());
+            this.logDir = path.join(path.normalize(logDir), getExperimentId());
         } else {
             this.logDir = path.join(os.homedir(), 'nni', 'experiments', getExperimentId());
         }

src/nni_manager/training_service/common/util.ts

@@ -70,11 +70,11 @@ export async function validateCodeDir(codeDir: string) : Promise<number> {
  */
 export async function execMkdir(directory: string, share: boolean = false): Promise<void> {
     if (process.platform === 'win32') {
-        await cpp.exec(`powershell.exe New-Item -Path ${directory} -ItemType "directory" -Force`);
+        await cpp.exec(`powershell.exe New-Item -Path "${directory}" -ItemType "directory" -Force`);
     } else if (share) {
-        await cpp.exec(`(umask 0; mkdir -p ${directory})`);
+        await cpp.exec(`(umask 0; mkdir -p '${directory}')`);
     } else {
-        await cpp.exec(`mkdir -p ${directory}`);
+        await cpp.exec(`mkdir -p '${directory}'`);
     }
 
     return Promise.resolve();
@@ -87,9 +87,9 @@ export async function execMkdir(directory: string, share: boolean = false): Prom
  */
 export async function execCopydir(source: string, destination: string): Promise<void> {
     if (process.platform === 'win32') {
-        await cpp.exec(`powershell.exe Copy-Item ${source} -Destination ${destination} -Recurse`);
+        await cpp.exec(`powershell.exe Copy-Item "${source}" -Destination "${destination}" -Recurse`);
     } else {
-        await cpp.exec(`cp -r ${source} ${destination}`);
+        await cpp.exec(`cp -r '${source}' '${destination}'`);
     }
 
     return Promise.resolve();
@@ -101,9 +101,9 @@ export async function execCopydir(source: string, destination: string): Promise<
  */
 export async function execNewFile(filename: string): Promise<void> {
     if (process.platform === 'win32') {
-        await cpp.exec(`powershell.exe New-Item -Path ${filename} -ItemType "file" -Force`);
+        await cpp.exec(`powershell.exe New-Item -Path "${filename}" -ItemType "file" -Force`);
     } else {
-        await cpp.exec(`touch ${filename}`);
+        await cpp.exec(`touch '${filename}'`);
     }
 
     return Promise.resolve();
@@ -115,9 +115,9 @@ export async function execNewFile(filename: string): Promise<void> {
  */
 export function runScript(filePath: string): cp.ChildProcess {
     if (process.platform === 'win32') {
-        return cp.exec(`powershell.exe -ExecutionPolicy Bypass -file ${filePath}`);
+        return cp.exec(`powershell.exe -ExecutionPolicy Bypass -file "${filePath}"`);
     } else {
-        return cp.exec(`bash ${filePath}`);
+        return cp.exec(`bash '${filePath}'`);
     }
 }
 
@@ -128,9 +128,9 @@ export function runScript(filePath: string): cp.ChildProcess {
 export async function execTail(filePath: string): Promise<cpp.childProcessPromise.Result> {
     let cmdresult: cpp.childProcessPromise.Result;
     if (process.platform === 'win32') {
-        cmdresult = await cpp.exec(`powershell.exe Get-Content ${filePath} -Tail 1`);
+        cmdresult = await cpp.exec(`powershell.exe Get-Content "${filePath}" -Tail 1`);
     } else {
-        cmdresult = await cpp.exec(`tail -n 1 ${filePath}`);
+        cmdresult = await cpp.exec(`tail -n 1 '${filePath}'`);
     }
 
     return Promise.resolve(cmdresult);
@@ -142,9 +142,9 @@ export async function execTail(filePath: string): Promise<cpp.childProcessPromis
  */
 export async function execRemove(directory: string): Promise<void> {
     if (process.platform === 'win32') {
-        await cpp.exec(`powershell.exe Remove-Item ${directory} -Recurse -Force`);
+        await cpp.exec(`powershell.exe Remove-Item "${directory}" -Recurse -Force`);
     } else {
-        await cpp.exec(`rm -rf ${directory}`);
+        await cpp.exec(`rm -rf '${directory}'`);
     }
 
     return Promise.resolve();
@@ -173,7 +173,7 @@ export function setEnvironmentVariable(variable: { key: string; value: string })
     if (process.platform === 'win32') {
         return `$env:${variable.key}="${variable.value}"`;
     } else {
-        return `export ${variable.key}=${variable.value}`;
+        return `export ${variable.key}='${variable.value}'`;
     }
 }
 

src/nni_manager/training_service/local/localTrainingService.ts

@@ -490,18 +490,18 @@ class LocalTrainingService implements TrainingService {
         const script: string[] = [];
         if (process.platform === 'win32') {
             script.push(
-                `cmd.exe /c ${localTrialConfig.command} 2>${path.join(workingDirectory, 'stderr')}`,
+                `cmd.exe /c ${localTrialConfig.command} 2>"${path.join(workingDirectory, 'stderr')}"`,
                 `$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds`,
                 `$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()`,
-                `Write $LASTEXITCODE " " $NOW_DATE  | Out-File ${path.join(workingDirectory, '.nni', 'state')} -NoNewline -encoding utf8`);
+                `Write $LASTEXITCODE " " $NOW_DATE  | Out-File "${path.join(workingDirectory, '.nni', 'state')}" -NoNewline -encoding utf8`);
         } else {
-            script.push(`eval ${localTrialConfig.command} 2>${path.join(workingDirectory, 'stderr')}`);
+            script.push(`eval ${localTrialConfig.command} 2>"${path.join(workingDirectory, 'stderr')}"`);
             if (process.platform === 'darwin') {
                 // https://superuser.com/questions/599072/how-to-get-bash-execution-time-in-milliseconds-under-mac-os-x
                 // Considering the worst case, write 999 to avoid negative duration
-                script.push(`echo $? \`date +%s999\` >${path.join(workingDirectory, '.nni', 'state')}`);
+                script.push(`echo $? \`date +%s999\` >'${path.join(workingDirectory, '.nni', 'state')}'`);
             } else {
-                script.push(`echo $? \`date +%s%3N\` >${path.join(workingDirectory, '.nni', 'state')}`);
+                script.push(`echo $? \`date +%s%3N\` >'${path.join(workingDirectory, '.nni', 'state')}'`);
             }
         }
 
@@ -522,7 +522,7 @@ class LocalTrainingService implements TrainingService {
         if (process.platform !== 'win32') {
             runScriptContent.push('#!/bin/bash');
         }
-        runScriptContent.push(`cd ${this.localTrialConfig.codeDir}`);
+        runScriptContent.push(`cd '${this.localTrialConfig.codeDir}'`);
         for (const variable of variables) {
             runScriptContent.push(setEnvironmentVariable(variable));
         }

src/sdk/pynni/nni/batch_tuner/batch_tuner.py

@@ -31,22 +31,27 @@ TYPE = '_type'
 CHOICE = 'choice'
 VALUE = '_value'
 
-logger = logging.getLogger('batch_tuner_AutoML')
+LOGGER = logging.getLogger('batch_tuner_AutoML')
 
 class BatchTuner(Tuner):
     """
     BatchTuner is tuner will running all the configure that user want to run batchly.
+
+    Examples
+    --------
     The search space only be accepted like:
+    ```
     {
         'combine_params': { '_type': 'choice',
                              '_value': '[{...}, {...}, {...}]',
                           }
     }
+    ```
     """
 
     def __init__(self):
-        self.count = -1
-        self.values = []
+        self._count = -1
+        self._values = []
 
     def is_valid(self, search_space):
         """
@@ -55,6 +60,11 @@ class BatchTuner(Tuner):
         Parameters
         ----------
         search_space : dict
+
+        Returns
+        -------
+        None or list
+            If valid, return candidate values; else return None.
         """
         if not len(search_space) == 1:
             raise RuntimeError('BatchTuner only supprt one combined-paramreters key.')
@@ -62,11 +72,14 @@ class BatchTuner(Tuner):
         for param in search_space:
             param_type = search_space[param][TYPE]
             if not param_type == CHOICE:
-                raise RuntimeError('BatchTuner only supprt one combined-paramreters type is choice.')
-            else:
-                if isinstance(search_space[param][VALUE], list):
-                    return search_space[param][VALUE]
-                raise RuntimeError('The combined-paramreters value in BatchTuner is not a list.')
+                raise RuntimeError('BatchTuner only supprt \
+                                    one combined-paramreters type is choice.')
+
+            if isinstance(search_space[param][VALUE], list):
+                return search_space[param][VALUE]
+
+            raise RuntimeError('The combined-paramreters \
+                                value in BatchTuner is not a list.')
         return None
 
     def update_search_space(self, search_space):
@@ -76,7 +89,7 @@ class BatchTuner(Tuner):
         ----------
         search_space : dict
         """
-        self.values = self.is_valid(search_space)
+        self._values = self.is_valid(search_space)
 
     def generate_parameters(self, parameter_id, **kwargs):
         """Returns a dict of trial (hyper-)parameters, as a serializable object.
@@ -84,41 +97,49 @@ class BatchTuner(Tuner):
         Parameters
         ----------
         parameter_id : int
+
+        Returns
+        -------
+        dict
+            A candidate parameter group.
         """
-        self.count += 1
-        if self.count > len(self.values) - 1:
+        self._count += 1
+        if self._count > len(self._values) - 1:
             raise nni.NoMoreTrialError('no more parameters now.')
-        return self.values[self.count]
+        return self._values[self._count]
 
     def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
         pass
 
     def import_data(self, data):
         """Import additional data for tuning
+
         Parameters
         ----------
         data:
             a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
         """
-        if not self.values:
-            logger.info("Search space has not been initialized, skip this data import")
+        if not self._values:
+            LOGGER.info("Search space has not been initialized, skip this data import")
             return
 
-        self.values = self.values[(self.count+1):]
-        self.count = -1
+        self._values = self._values[(self._count+1):]
+        self._count = -1
 
         _completed_num = 0
         for trial_info in data:
-            logger.info("Importing data, current processing progress %s / %s", _completed_num, len(data))
+            LOGGER .info("Importing data, current processing \
+                            progress %s / %s", _completed_num, len(data))
             # simply validate data format
             assert "parameter" in trial_info
             _params = trial_info["parameter"]
             assert "value" in trial_info
             _value = trial_info['value']
             if not _value:
-                logger.info("Useless trial data, value is %s, skip this trial data.", _value)
+                LOGGER.info("Useless trial data, value is %s, skip this trial data.", _value)
                 continue
             _completed_num += 1
-            if _params in self.values:
-                self.values.remove(_params)
-        logger.info("Successfully import data to batch tuner, total data: %d, imported data: %d.", len(data), _completed_num)
+            if _params in self._values:
+                self._values.remove(_params)
+        LOGGER .info("Successfully import data to batch tuner, \
+                        total data: %d, imported data: %d.", len(data), _completed_num)

src/sdk/pynni/nni/compression/torch/compressor.py

@@ -32,7 +32,23 @@ class Compressor:
         """
         self.bound_model = model
         self.config_list = config_list
-        self.modules_to_compress = []
+        self.modules_to_compress = None
+
+    def detect_modules_to_compress(self):
+        """
+        detect all modules should be compressed, and save the result in `self.modules_to_compress`.
+
+        The model will be instrumented and user should never edit it after calling this method.
+        """
+        if self.modules_to_compress is None:
+            self.modules_to_compress = []
+            for name, module in self.bound_model.named_modules():
+                layer = LayerInfo(name, module)
+                config = self.select_config(layer)
+                if config is not None:
+                    self.modules_to_compress.append((layer, config))
+        return self.modules_to_compress
+
 
     def compress(self):
         """
@@ -41,12 +57,9 @@ class Compressor:
         The model will be instrumented and user should never edit it after calling this method.
         `self.modules_to_compress` records all the to-be-compressed layers
         """
-        for name, module in self.bound_model.named_modules():
-            layer = LayerInfo(name, module)
-            config = self.select_config(layer)
-            if config is not None:
-                self._instrument_layer(layer, config)
-                self.modules_to_compress.append((layer, config))
+        modules_to_compress = self.detect_modules_to_compress()
+        for layer, config in modules_to_compress:
+            self._instrument_layer(layer, config)
         return self.bound_model
 
     def get_modules_to_compress(self):
@@ -55,7 +68,7 @@ class Compressor:
 
         Returns
         -------
-        self.modules_to_compress : list
+        list
             a list of the layers, each of which is a tuple (`layer`, `config`),
             `layer` is `LayerInfo`, `config` is a `dict`
         """
@@ -72,7 +85,7 @@ class Compressor:
 
         Returns
         -------
-        ret : config or None
+        config or None
             the retrieved configuration for this layer, if None, this layer should
             not be compressed
         """
@@ -206,6 +219,8 @@ class Pruner(Compressor):
         """
         assert model_path is not None, 'model_path must be specified'
         for name, m in self.bound_model.named_modules():
+            if name == "":
+                continue
             mask = self.mask_dict.get(name)
             if mask is not None:
                 mask_sum = mask.sum().item()
@@ -238,26 +253,87 @@ class Quantizer(Compressor):
     """
 
     def quantize_weight(self, weight, config, op, op_type, op_name):
-        """user should know where dequantize goes and implement it in quantize method
-        we now do not provide dequantize method
+        """
+        quantize should overload this method to quantize weight.
+        This method is effectively hooked to :meth:`forward` of the model.
+
+        Parameters
+        ----------
+        weight : Tensor
+            weight that needs to be quantized
+        config : dict
+            the configuration for weight quantization
         """
         raise NotImplementedError("Quantizer must overload quantize_weight()")
 
+    def quantize_output(self, output, config, op, op_type, op_name):
+        """
+        quantize should overload this method to quantize output.
+        This method is effectively hooked to :meth:`forward` of the model.
+
+        Parameters
+        ----------
+        output : Tensor
+            output that needs to be quantized
+        config : dict
+            the configuration for output quantization
+        """
+        raise NotImplementedError("Quantizer must overload quantize_output()")
+
+    def quantize_input(self, *inputs, config, op, op_type, op_name):
+        """
+        quantize should overload this method to quantize input.
+        This method is effectively hooked to :meth:`forward` of the model.
+
+        Parameters
+        ----------
+        inputs : Tensor
+            inputs that needs to be quantized
+        config : dict
+            the configuration for inputs quantization
+        """
+        raise NotImplementedError("Quantizer must overload quantize_input()")
+
+
     def _instrument_layer(self, layer, config):
+        """
+        Create a wrapper forward function to replace the original one.
+
+        Parameters
+        ----------
+        layer : LayerInfo
+            the layer to instrument the mask
+        config : dict
+            the configuration for quantization
+        """
         assert layer._forward is None, 'Each model can only be compressed once'
-        if not _check_weight(layer.module):
-            _logger.warning('Module %s does not have parameter "weight"', layer.name)
-            return
+        assert "quant_types" in config, 'must provide quant_types in config'
+        assert isinstance(config["quant_types"], list), 'quant_types must be list type'
+
+        if 'weight' in config["quant_types"]:
+            if not _check_weight(layer.module):
+                _logger.warning('Module %s does not have parameter "weight"', layer.name)
         layer._forward = layer.module.forward
 
         def new_forward(*inputs):
-            weight = layer.module.weight.data
-            new_weight = self.quantize_weight(weight, config, op=layer.module, op_type=layer.type, op_name=layer.name)
-            layer.module.weight.data = new_weight
-            return layer._forward(*inputs)
+            if 'input' in config["quant_types"]:
+                inputs = self.quantize_input(inputs, config=config, op=layer.module, op_type=layer.type, op_name=layer.name)
+
+            if 'weight' in config["quant_types"] and _check_weight(layer.module):
+                weight = layer.module.weight.data
+                new_weight = self.quantize_weight(weight, config, op=layer.module, op_type=layer.type, op_name=layer.name)
+                layer.module.weight.data = new_weight
+                result = layer._forward(*inputs)
+                layer.module.weight.data = weight
+            else:
+                result = layer._forward(*inputs)
 
-        layer.module.forward = new_forward
+            if 'output' in config["quant_types"]:
+                result = self.quantize_output(result, config, op=layer.module, op_type=layer.type, op_name=layer.name)
 
+            return result
+
+        layer.module.forward = new_forward
 
 def _check_weight(module):
     try:

src/sdk/pynni/nni/curvefitting_assessor/curvefitting_assessor.py

@@ -29,13 +29,13 @@ class CurvefittingAssessor(Assessor):
 
     Parameters
     ----------
-    epoch_num: int
+    epoch_num : int
         The total number of epoch
-    optimize_mode: str
+    optimize_mode : str
         optimize mode, 'maximize' or 'minimize'
-    start_step: int
+    start_step : int
         only after receiving start_step number of reported intermediate results
-    threshold: float
+    threshold : float
         The threshold that we decide to early stop the worse performance curve.
     """
     def __init__(self, epoch_num=20, optimize_mode='maximize', start_step=6, threshold=0.95, gap=1):
@@ -70,9 +70,9 @@ class CurvefittingAssessor(Assessor):
 
         Parameters
         ----------
-        trial_job_id: int
+        trial_job_id : int
             trial job id
-        success: bool
+        success : bool
             True if succssfully finish the experiment, False otherwise
         """
         if success:
@@ -90,9 +90,9 @@ class CurvefittingAssessor(Assessor):
 
         Parameters
         ----------
-        trial_job_id: int
+        trial_job_id : int
             trial job id
-        trial_history: list
+        trial_history : list
             The history performance matrix of each trial
 
         Returns
@@ -105,7 +105,6 @@ class CurvefittingAssessor(Assessor):
         Exception
             unrecognize exception in curvefitting_assessor
         """
-        trial_job_id = trial_job_id
         self.trial_history = trial_history
         if not self.set_best_performance:
             return AssessResult.Good

src/sdk/pynni/nni/curvefitting_assessor/curvefunctions.py

@@ -14,7 +14,9 @@
 # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
 # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
+"""
+A family of functions used by CurvefittingAssessor
+"""
 import numpy as np
 
 all_models = {}
@@ -29,10 +31,10 @@ def vap(x, a, b, c):
 
     Parameters
     ----------
-    x: int
-    a: float
-    b: float
-    c: float
+    x : int
+    a : float
+    b : float
+    c : float
 
     Returns
     -------
@@ -50,10 +52,10 @@ def pow3(x, c, a, alpha):
 
     Parameters
     ----------
-    x: int
-    c: float
-    a: float
-    alpha: float
+    x : int
+    c : float
+    a : float
+    alpha : float
 
     Returns
     -------
@@ -71,9 +73,9 @@ def linear(x, a, b):
 
     Parameters
     ----------
-    x: int
-    a: float
-    b: float
+    x : int
+    a : float
+    b : float
 
     Returns
     -------
@@ -91,9 +93,9 @@ def logx_linear(x, a, b):
 
     Parameters
     ----------
-    x: int
-    a: float
-    b: float
+    x : int
+    a : float
+    b : float
 
     Returns
     -------
@@ -112,10 +114,10 @@ def dr_hill_zero_background(x, theta, eta, kappa):
 
     Parameters
     ----------
-    x: int
-    theta: float
-    eta: float
-    kappa: float
+    x : int
+    theta : float
+    eta : float
+    kappa : float
 
     Returns
     -------
@@ -133,10 +135,10 @@ def log_power(x, a, b, c):
 
     Parameters
     ----------
-    x: int
-    a: float
-    b: float
-    c: float
+    x : int
+    a : float
+    b : float
+    c : float
 
     Returns
     -------
@@ -154,11 +156,11 @@ def pow4(x, alpha, a, b, c):
 
     Parameters
     ----------
-    x: int
-    alpha: float
-    a: float
-    b: float
-    c: float
+    x : int
+    alpha : float
+    a : float
+    b : float
+    c : float
 
     Returns
     -------
@@ -177,11 +179,11 @@ def mmf(x, alpha, beta, kappa, delta):
 
     Parameters
     ----------
-    x: int
-    alpha: float
-    beta: float
-    kappa: float
-    delta: float
+    x : int
+    alpha : float
+    beta : float
+    kappa : float
+    delta : float
 
     Returns
     -------
@@ -199,11 +201,11 @@ def exp4(x, c, a, b, alpha):
 
     Parameters
     ----------
-    x: int
-    c: float
-    a: float
-    b: float
-    alpha: float
+    x : int
+    c : float
+    a : float
+    b : float
+    alpha : float
 
     Returns
     -------
@@ -221,9 +223,9 @@ def ilog2(x, c, a):
 
     Parameters
     ----------
-    x: int
-    c: float
-    a: float
+    x : int
+    c : float
+    a : float
 
     Returns
     -------
@@ -242,11 +244,11 @@ def weibull(x, alpha, beta, kappa, delta):
 
     Parameters
     ----------
-    x: int
-    alpha: float
-    beta: float
-    kappa: float
-    delta: float
+    x : int
+    alpha : float
+    beta : float
+    kappa : float
+    delta : float
 
     Returns
     -------
@@ -264,11 +266,11 @@ def janoschek(x, a, beta, k, delta):
 
     Parameters
     ----------
-    x: int
-    a: float
-    beta: float
-    k: float
-    delta: float
+    x : int
+    a : float
+    beta : float
+    k : float
+    delta : float
 
     Returns
     -------

src/sdk/pynni/nni/curvefitting_assessor/model_factory.py

@@ -40,7 +40,7 @@ class CurveModel:
 
     Parameters
     ----------
-    target_pos: int
+    target_pos : int
         The point we need to predict
     """
     def __init__(self, target_pos):
@@ -120,14 +120,14 @@ class CurveModel:
 
         Parameters
         ----------
-        model: string
+        model : string
             name of the curve function model
-        pos: int
+        pos : int
             the epoch number of the position you want to predict
 
         Returns
         -------
-        int:
+        int
             The expected matrix at pos
         """
         if model_para_num[model] == 2:
@@ -143,9 +143,9 @@ class CurveModel:
 
         Parameters
         ----------
-        pos: int
+        pos : int
             the epoch number of the position you want to predict
-        sample: list
+        sample : list
             sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
 
         Returns
@@ -165,7 +165,7 @@ class CurveModel:
 
         Parameters
         ----------
-        samples: list
+        samples : list
             a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
             representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
 
@@ -187,7 +187,7 @@ class CurveModel:
 
         Parameters
         ----------
-        sample: list
+        sample : list
             sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
 
         Returns
@@ -206,9 +206,9 @@ class CurveModel:
 
         Parameters
         ----------
-        pos: int
+        pos : int
             the epoch number of the position you want to predict
-        sample: list
+        sample : list
             sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
 
         Returns
@@ -225,7 +225,7 @@ class CurveModel:
 
         Parameters
         ----------
-        sample: list
+        sample : list
             sample is a (1 * NUM_OF_FUNCTIONS) matrix, representing{w1, w2, ... wk}
 
         Returns
@@ -244,7 +244,7 @@ class CurveModel:
 
         Parameters
         ----------
-        samples: list
+        samples : list
             a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
             representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
 
@@ -267,7 +267,7 @@ class CurveModel:
 
         Parameters
         ----------
-        samples: list
+        samples : list
             a collection of sample, it's a (NUM_OF_INSTANCE * NUM_OF_FUNCTIONS) matrix,
             representing{{w11, w12, ..., w1k}, {w21, w22, ... w2k}, ...{wk1, wk2,..., wkk}}
 
@@ -322,7 +322,7 @@ class CurveModel:
 
         Parameters
         ----------
-        trial_history: list
+        trial_history : list
             The history performance matrix of each trial.
 
         Returns

src/sdk/pynni/nni/evolution_tuner/evolution_tuner.py

@@ -32,7 +32,9 @@ import nni.parameter_expressions as parameter_expressions
 
 
 def json2space(x, oldy=None, name=NodeType.ROOT):
-    """Change search space from json format to hyperopt format
+    """
+    Change search space from json format to hyperopt format
+
     """
     y = list()
     if isinstance(x, dict):
@@ -59,7 +61,9 @@ def json2space(x, oldy=None, name=NodeType.ROOT):
     return y
 
 def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeType.ROOT):
-    """Json to pramaters.
+    """
+    Json to pramaters.
+
     """
     if isinstance(x, dict):
         if NodeType.TYPE in x.keys():
@@ -117,6 +121,17 @@ def json2parameter(x, is_rand, random_state, oldy=None, Rand=False, name=NodeTyp
 class Individual:
     """
     Indicidual class to store the indv info.
+
+    Attributes
+    ----------
+    config : str
+        Search space.
+    info : str
+        The str to save information of individual.
+    result : float
+        The final metric of a individual.
+    store_dir : str
+    save_dir : str
     """
 
     def __init__(self, config=None, info=None, result=None, save_dir=None):
@@ -124,6 +139,7 @@ class Individual:
         Parameters
         ----------
         config : str
+            A config to represent a group of parameters.
         info : str
         result : float
         save_dir : str
@@ -140,6 +156,8 @@ class Individual:
 
     def mutation(self, config=None, info=None, save_dir=None):
         """
+        Mutation by reset state information.
+
         Parameters
         ----------
         config : str
@@ -177,8 +195,11 @@ class EvolutionTuner(Tuner):
         self.population = None
         self.space = None
 
+
     def update_search_space(self, search_space):
-        """Update search space.
+        """
+        Update search space.
+
         Search_space contains the information that user pre-defined.
 
         Parameters
@@ -191,15 +212,19 @@ class EvolutionTuner(Tuner):
         self.random_state = np.random.RandomState()
         self.population = []
         is_rand = dict()
+
         for item in self.space:
             is_rand[item] = True
+
         for _ in range(self.population_size):
             config = json2parameter(
                 self.searchspace_json, is_rand, self.random_state)
             self.population.append(Individual(config=config))
 
+
     def generate_parameters(self, parameter_id, **kwargs):
-        """Returns a dict of trial (hyper-)parameters, as a serializable object.
+        """
+        This function will returns a dict of trial (hyper-)parameters, as a serializable object.
 
         Parameters
         ----------
@@ -207,15 +232,19 @@ class EvolutionTuner(Tuner):
 
         Returns
         -------
-        config : dict
+        dict
+            A group of candaidte parameters that evolution tuner generated.
         """
         if not self.population:
             raise RuntimeError('The population is empty')
+
         pos = -1
+
         for i in range(len(self.population)):
             if self.population[i].result is None:
                 pos = i
                 break
+
         if pos != -1:
             indiv = copy.deepcopy(self.population[pos])
             self.population.pop(pos)
@@ -230,6 +259,7 @@ class EvolutionTuner(Tuner):
                                self.population[0].config)
             is_rand = dict()
             mutation_pos = space[random.randint(0, len(space)-1)]
+
             for i in range(len(self.space)):
                 is_rand[self.space[i]] = (self.space[i] == mutation_pos)
             config = json2parameter(
@@ -238,21 +268,27 @@ class EvolutionTuner(Tuner):
             # remove "_index" from config and save params-id
 
             total_config = config
+
         self.total_data[parameter_id] = total_config
         config = split_index(total_config)
+
         return config
 
+
     def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
-        '''Record the result from a trial
+        """
+        Record the result from a trial
 
         Parameters
         ----------
-        parameters: dict
+        parameter_id : int
+        parameters : dict
         value : dict/float
             if value is dict, it should have "default" key.
             value is final metrics of the trial.
-        '''
+        """
         reward = extract_scalar_reward(value)
+
         if parameter_id not in self.total_data:
             raise RuntimeError('Received parameter_id not in total_data.')
         # restore the paramsters contains "_index"

src/sdk/pynni/nni/gp_tuner/gp_tuner.py

@@ -17,9 +17,11 @@
 # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
 # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-'''
-gp_tuner.py
-'''
+"""
+GPTuner is a Bayesian Optimization method where Gaussian Process is used for modeling loss functions.
+
+See :class:`GPTuner` for details.
+"""
 
 import warnings
 import logging
@@ -38,18 +40,40 @@ logger = logging.getLogger("GP_Tuner_AutoML")
 
 
 class GPTuner(Tuner):
-    '''
-    GPTuner
-    '''
+    """
+    GPTuner is a Bayesian Optimization method where Gaussian Process is used for modeling loss functions.
+
+    Parameters
+    ----------
+    optimize_mode : str
+        optimize mode, 'maximize' or 'minimize', by default 'maximize'
+    utility : str
+        utility function (also called 'acquisition funcition') to use, which can be 'ei', 'ucb' or 'poi'. By default 'ei'.
+    kappa : float
+        value used by utility function 'ucb'. The bigger kappa is, the more the tuner will be exploratory. By default 5.
+    xi : float
+        used by utility function 'ei' and 'poi'. The bigger xi is, the more the tuner will be exploratory. By default 0.
+    nu : float
+        used to specify Matern kernel. The smaller nu, the less smooth the approximated function is. By default 2.5.
+    alpha : float
+        Used to specify Gaussian Process Regressor. Larger values correspond to increased noise level in the observations.
+        By default 1e-6.
+    cold_start_num : int
+        Number of random exploration to perform before Gaussian Process. By default 10.
+    selection_num_warm_up : int
+        Number of random points to evaluate for getting the point which maximizes the acquisition function. By default 100000
+    selection_num_starting_points : int
+        Number of times to run L-BFGS-B from a random starting point after the warmup. By default 250.
+    """
 
     def __init__(self, optimize_mode="maximize", utility='ei', kappa=5, xi=0, nu=2.5, alpha=1e-6, cold_start_num=10,
                  selection_num_warm_up=100000, selection_num_starting_points=250):
-        self.optimize_mode = OptimizeMode(optimize_mode)
+        self._optimize_mode = OptimizeMode(optimize_mode)
 
         # utility function related
-        self.utility = utility
-        self.kappa = kappa
-        self.xi = xi
+        self._utility = utility
+        self._kappa = kappa
+        self._xi = xi
 
         # target space
         self._space = None
@@ -72,30 +96,23 @@ class GPTuner(Tuner):
         self._selection_num_starting_points = selection_num_starting_points
 
         # num of imported data
-        self.supplement_data_num = 0
+        self._supplement_data_num = 0
 
     def update_search_space(self, search_space):
-        """Update the self.bounds and self.types by the search_space.json
+        """
+        Update the self.bounds and self.types by the search_space.json file.
 
-        Parameters
-        ----------
-        search_space : dict
+        Override of the abstract method in :class:`~nni.tuner.Tuner`.
         """
         self._space = TargetSpace(search_space, self._random_state)
 
     def generate_parameters(self, parameter_id, **kwargs):
-        """Generate next parameter for trial
-        If the number of trial result is lower than cold start number,
-        gp will first randomly generate some parameters.
-        Otherwise, choose the parameters by the Gussian Process Model
-
-        Parameters
-        ----------
-        parameter_id : int
-
-        Returns
-        -------
-        result : dict
+        """
+        Method which provides one set of hyper-parameters.
+        If the number of trial result is lower than cold_start_number, GPTuner will first randomly generate some parameters.
+        Otherwise, choose the parameters by the Gussian Process Model.
+
+        Override of the abstract method in :class:`~nni.tuner.Tuner`.
         """
         if self._space.len() < self._cold_start_num:
             results = self._space.random_sample()
@@ -107,7 +124,7 @@ class GPTuner(Tuner):
                 self._gp.fit(self._space.params, self._space.target)
 
             util = UtilityFunction(
-                kind=self.utility, kappa=self.kappa, xi=self.xi)
+                kind=self._utility, kappa=self._kappa, xi=self._xi)
 
             results = acq_max(
                 f_acq=util.utility,
@@ -124,17 +141,13 @@ class GPTuner(Tuner):
         return results
 
     def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
-        """Tuner receive result from trial.
-
-        Parameters
-        ----------
-        parameter_id : int
-        parameters : dict
-        value : dict/float
-            if value is dict, it should have "default" key.
+        """
+        Method invoked when a trial reports its final result.
+
+        Override of the abstract method in :class:`~nni.tuner.Tuner`.
         """
         value = extract_scalar_reward(value)
-        if self.optimize_mode == OptimizeMode.Minimize:
+        if self._optimize_mode == OptimizeMode.Minimize:
             value = -value
 
         logger.info("Received trial result.")
@@ -143,26 +156,27 @@ class GPTuner(Tuner):
         self._space.register(parameters, value)
 
     def import_data(self, data):
-        """Import additional data for tuning
-        Parameters
-        ----------
-        data:
-            a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
+        """
+        Import additional data for tuning.
+
+        Override of the abstract method in :class:`~nni.tuner.Tuner`.
         """
         _completed_num = 0
         for trial_info in data:
-            logger.info("Importing data, current processing progress %s / %s", _completed_num, len(data))
+            logger.info(
+                "Importing data, current processing progress %s / %s", _completed_num, len(data))
             _completed_num += 1
             assert "parameter" in trial_info
             _params = trial_info["parameter"]
             assert "value" in trial_info
             _value = trial_info['value']
             if not _value:
-                logger.info("Useless trial data, value is %s, skip this trial data.", _value)
+                logger.info(
+                    "Useless trial data, value is %s, skip this trial data.", _value)
                 continue
-            self.supplement_data_num += 1
+            self._supplement_data_num += 1
             _parameter_id = '_'.join(
-                ["ImportData", str(self.supplement_data_num)])
+                ["ImportData", str(self._supplement_data_num)])
             self.receive_trial_result(
                 parameter_id=_parameter_id, parameters=_params, value=_value)
         logger.info("Successfully import data to GP tuner.")

src/sdk/pynni/nni/gp_tuner/target_space.py

@@ -17,39 +17,51 @@
 # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
 # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-'''
-target_space.py
-'''
+"""
+Tool class to hold the param-space coordinates (X) and target values (Y).
+"""
+
 
 import numpy as np
 import nni.parameter_expressions as parameter_expressions
 
 
 def _hashable(params):
-    """ ensure that an point is hashable by a python dict """
+    """
+    Transform list params to tuple format. Ensure that an point is hashable by a python dict.
+
+    Parameters
+    ----------
+    params : numpy array
+        array format of parameters
+
+    Returns
+    -------
+    tuple
+        tuple format of parameters
+    """
     return tuple(map(float, params))
 
 
 class TargetSpace():
     """
     Holds the param-space coordinates (X) and target values (Y)
+
+    Parameters
+    ----------
+    pbounds : dict
+        Dictionary with parameters names and legal values.
+
+    random_state : int, RandomState, or None
+        optionally specify a seed for a random number generator, by default None.
     """
 
     def __init__(self, pbounds, random_state=None):
-        """
-        Parameters
-        ----------
-        pbounds : dict
-            Dictionary with parameters names as keys and a tuple with minimum
-            and maximum values.
-
-        random_state : int, RandomState, or None
-            optionally specify a seed for a random number generator
-        """
-        self.random_state = random_state
+        self._random_state = random_state
 
         # Get the name of the parameters
         self._keys = sorted(pbounds)
+
         # Create an array with parameters bounds
         self._bounds = np.array(
             [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])]
@@ -71,54 +83,100 @@ class TargetSpace():
         self._cache = {}
 
     def __contains__(self, params):
-        '''
+        """
         check if a parameter is already registered
-        '''
+
+        Parameters
+        ----------
+        params : numpy array
+
+        Returns
+        -------
+        bool
+            True if the parameter is already registered, else false
+        """
         return _hashable(params) in self._cache
 
     def len(self):
-        '''
+        """
         length of registered params and targets
-        '''
+
+        Returns
+        -------
+        int
+        """
         assert len(self._params) == len(self._target)
         return len(self._target)
 
     @property
     def params(self):
-        '''
-        params: numpy array
-        '''
+        """
+        registered parameters
+
+        Returns
+        -------
+        numpy array
+        """
         return self._params
 
     @property
     def target(self):
-        '''
-        target: numpy array
-        '''
+        """
+        registered target values
+
+        Returns
+        -------
+        numpy array
+        """
         return self._target
 
     @property
     def dim(self):
-        '''
-        dim: int
-            length of keys
-        '''
+        """
+        dimension of parameters
+
+        Returns
+        -------
+        int
+        """
         return len(self._keys)
 
     @property
     def keys(self):
-        '''
-        keys: numpy array
-        '''
+        """
+        keys of parameters
+
+        Returns
+        -------
+        numpy array
+        """
         return self._keys
 
     @property
     def bounds(self):
-        '''bounds'''
+        """
+        bounds of parameters
+
+        Returns
+        -------
+        numpy array
+        """
         return self._bounds
 
     def params_to_array(self, params):
-        ''' dict to array '''
+        """
+        dict to array
+
+        Parameters
+        ----------
+        params : dict
+            dict format of parameters
+
+        Returns
+        -------
+        numpy array
+            array format of parameters
+        """
         try:
             assert set(params) == set(self.keys)
         except AssertionError:
@@ -129,11 +187,20 @@ class TargetSpace():
         return np.asarray([params[key] for key in self.keys])
 
     def array_to_params(self, x):
-        '''
+        """
         array to dict
 
         maintain int type if the paramters is defined as int in search_space.json
-        '''
+        Parameters
+        ----------
+        x : numpy array
+            array format of parameters
+
+        Returns
+        -------
+        dict
+            dict format of parameters
+        """
         try:
             assert len(x) == len(self.keys)
         except AssertionError:
@@ -159,15 +226,15 @@ class TargetSpace():
 
         Parameters
         ----------
-        x : dict
+        params : dict
+            parameters
 
-        y : float
+        target : float
             target function value
         """
 
         x = self.params_to_array(params)
         if x in self:
-            #raise KeyError('Data point {} is not unique'.format(x))
             print('Data point {} is not unique'.format(x))
 
         # Insert data into unique dictionary
@@ -180,32 +247,43 @@ class TargetSpace():
         """
         Creates a random point within the bounds of the space.
 
+        Returns
+        -------
+        numpy array
+            one groupe of parameter
         """
         params = np.empty(self.dim)
         for col, _bound in enumerate(self._bounds):
             if _bound['_type'] == 'choice':
                 params[col] = parameter_expressions.choice(
-                    _bound['_value'], self.random_state)
+                    _bound['_value'], self._random_state)
             elif _bound['_type'] == 'randint':
-                params[col] = self.random_state.randint(
+                params[col] = self._random_state.randint(
                     _bound['_value'][0], _bound['_value'][1], size=1)
             elif _bound['_type'] == 'uniform':
                 params[col] = parameter_expressions.uniform(
-                    _bound['_value'][0], _bound['_value'][1], self.random_state)
+                    _bound['_value'][0], _bound['_value'][1], self._random_state)
             elif _bound['_type'] == 'quniform':
                 params[col] = parameter_expressions.quniform(
-                    _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self.random_state)
+                    _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self._random_state)
             elif _bound['_type'] == 'loguniform':
                 params[col] = parameter_expressions.loguniform(
-                    _bound['_value'][0], _bound['_value'][1], self.random_state)
+                    _bound['_value'][0], _bound['_value'][1], self._random_state)
             elif _bound['_type'] == 'qloguniform':
                 params[col] = parameter_expressions.qloguniform(
-                    _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self.random_state)
+                    _bound['_value'][0], _bound['_value'][1], _bound['_value'][2], self._random_state)
 
         return params
 
     def max(self):
-        """Get maximum target value found and corresponding parametes."""
+        """
+        Get maximum target value found and its corresponding parameters.
+
+        Returns
+        -------
+        dict
+            target value and parameters, empty dict if nothing registered
+        """
         try:
             res = {
                 'target': self.target.max(),
@@ -218,7 +296,14 @@ class TargetSpace():
         return res
 
     def res(self):
-        """Get all target values found and corresponding parametes."""
+        """
+        Get all target values found and corresponding parameters.
+
+        Returns
+        -------
+        list
+            a list of target values and their corresponding parameters
+        """
         params = [dict(zip(self.keys, p)) for p in self.params]
 
         return [

src/sdk/pynni/nni/gp_tuner/util.py

@@ -17,9 +17,9 @@
 # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
 # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-'''
-gp_tuner.py
-'''
+"""
+utility functions and classes for GPTuner
+"""
 
 import warnings
 import numpy as np
@@ -28,9 +28,21 @@ from scipy.optimize import minimize
 
 
 def _match_val_type(vals, bounds):
-    '''
-    Update values in the array, to match their corresponding type
-    '''
+    """
+    Update values in the array, to match their corresponding type, make sure the value is legal.
+
+    Parameters
+    ----------
+    vals : numpy array
+        values of parameters
+    bounds : numpy array
+        list of dictionary which stores parameters names and legal values.
+
+    Returns
+    -------
+    vals_new : list
+        The closest legal value to the original value
+    """
     vals_new = []
 
     for i, bound in enumerate(bounds):
@@ -52,32 +64,33 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points):
     A function to find the maximum of the acquisition function
 
     It uses a combination of random sampling (cheap) and the 'L-BFGS-B'
-    optimization method. First by sampling `n_warmup` (1e5) points at random,
-    and then running L-BFGS-B from `n_iter` (250) random starting points.
+    optimization method. First by sampling ``num_warmup`` points at random,
+    and then running L-BFGS-B from ``num_starting_points`` random starting points.
 
     Parameters
     ----------
-    :param f_acq:
+    f_acq : UtilityFunction.utility
         The acquisition function object that return its point-wise value.
 
-    :param gp:
+    gp : GaussianProcessRegressor
         A gaussian process fitted to the relevant data.
 
-    :param y_max:
+    y_max : float
         The current maximum known value of the target function.
 
-    :param bounds:
+    bounds : numpy array
         The variables bounds to limit the search of the acq max.
 
-    :param num_warmup:
+    num_warmup : int
         number of times to randomly sample the aquisition function
 
-    :param num_starting_points:
+    num_starting_points : int
         number of times to run scipy.minimize
 
     Returns
     -------
-    :return: x_max, The arg max of the acquisition function.
+    numpy array
+        The parameter which achieves max of the acquisition function.
     """
 
     # Warm up with random points
@@ -117,36 +130,70 @@ def acq_max(f_acq, gp, y_max, bounds, space, num_warmup, num_starting_points):
 
 class UtilityFunction():
     """
-    An object to compute the acquisition functions.
+    A class to compute different acquisition function values.
+
+    Parameters
+    ----------
+    kind : string
+        specification of utility function to use
+    kappa : float
+        parameter usedd for 'ucb' acquisition function
+    xi : float
+        parameter usedd for 'ei' and 'poi' acquisition function
     """
 
     def __init__(self, kind, kappa, xi):
-        """
-        If UCB is to be used, a constant kappa is needed.
-        """
-        self.kappa = kappa
-
-        self.xi = xi
+        self._kappa = kappa
+        self._xi = xi
 
         if kind not in ['ucb', 'ei', 'poi']:
             err = "The utility function " \
                 "{} has not been implemented, " \
                 "please choose one of ucb, ei, or poi.".format(kind)
             raise NotImplementedError(err)
-        self.kind = kind
+        self._kind = kind
 
     def utility(self, x, gp, y_max):
-        '''return utility function'''
-        if self.kind == 'ucb':
-            return self._ucb(x, gp, self.kappa)
-        if self.kind == 'ei':
-            return self._ei(x, gp, y_max, self.xi)
-        if self.kind == 'poi':
-            return self._poi(x, gp, y_max, self.xi)
+        """
+        return utility function
+
+        Parameters
+        ----------
+        x : numpy array
+            parameters
+        gp : GaussianProcessRegressor
+        y_max : float
+            maximum target value observed so far
+
+        Returns
+        -------
+        function
+            return corresponding function, return None if parameter is illegal
+        """
+        if self._kind == 'ucb':
+            return self._ucb(x, gp, self._kappa)
+        if self._kind == 'ei':
+            return self._ei(x, gp, y_max, self._xi)
+        if self._kind == 'poi':
+            return self._poi(x, gp, y_max, self._xi)
         return None
 
     @staticmethod
     def _ucb(x, gp, kappa):
+        """
+        Upper Confidence Bound (UCB) utility function
+
+        Parameters
+        ----------
+        x : numpy array
+            parameters
+        gp : GaussianProcessRegressor
+        kappa : float
+
+        Returns
+        -------
+        float
+        """
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             mean, std = gp.predict(x, return_std=True)
@@ -155,6 +202,22 @@ class UtilityFunction():
 
     @staticmethod
     def _ei(x, gp, y_max, xi):
+        """
+        Expected Improvement (EI) utility function
+
+        Parameters
+        ----------
+        x : numpy array
+            parameters
+        gp : GaussianProcessRegressor
+        y_max : float
+            maximum target value observed so far
+        xi : float
+
+        Returns
+        -------
+        float
+        """
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             mean, std = gp.predict(x, return_std=True)
@@ -164,6 +227,22 @@ class UtilityFunction():
 
     @staticmethod
     def _poi(x, gp, y_max, xi):
+        """
+        Possibility Of Improvement (POI) utility function
+
+        Parameters
+        ----------
+        x : numpy array
+            parameters
+        gp : GaussianProcessRegressor
+        y_max : float
+            maximum target value observed so far
+        xi : float
+
+        Returns
+        -------
+        float
+        """
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             mean, std = gp.predict(x, return_std=True)

src/sdk/pynni/nni/gridsearch_tuner/__init__.py

+from .gridsearch_tuner import GridSearchTuner
\ No newline at end of file