`@nni.report_intermediate_result` is used to report intermediate result, whose usage is the same as `nni.report_intermediate_result` in [Trials.md](Trials.md)
`@nni.report_intermediate_result` is used to report intermediate result, whose usage is the same as `nni.report_intermediate_result` in [Trials.md](../TrialExample/Trials.md)
### 4. Annotate final result
### 4. Annotate final result
`'''@nni.report_final_result(metrics)'''`
`'''@nni.report_final_result(metrics)'''`
`@nni.report_final_result` is used to report the final result of the current trial, whose usage is the same as `nni.report_final_result` in [Trials.md](Trials.md)
`@nni.report_final_result` is used to report the final result of the current trial, whose usage is the same as `nni.report_final_result` in [Trials.md](../TrialExample/Trials.md)
@@ -6,7 +6,7 @@ Firstly, if you are unsure or afraid of anything, just ask or submit the issue o
...
@@ -6,7 +6,7 @@ Firstly, if you are unsure or afraid of anything, just ask or submit the issue o
However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover all the points we're looking for in your contributions, raising your chances of quickly merging or addressing your contributions.
However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover all the points we're looking for in your contributions, raising your chances of quickly merging or addressing your contributions.
Looking for a quickstart, get acquainted with our [Get Started](./QuickStart.md) guide.
Looking for a quickstart, get acquainted with our [Get Started](QuickStart.md) guide.
There are a few simple guidelines that you need to follow before providing your hacks.
There are a few simple guidelines that you need to follow before providing your hacks.
__authorName__ is the name of the author who create the experiment.
__authorName__ is the name of the author who create the experiment.
TBD: add default value
TBD: add default value
* __experimentName__
* __experimentName__
* Description
* Description
__experimentName__ is the name of the experiment created.
__experimentName__ is the name of the experiment created.
TBD: add default value
TBD: add default value
* __trialConcurrency__
* __trialConcurrency__
...
@@ -174,9 +188,9 @@ machineList:
...
@@ -174,9 +188,9 @@ machineList:
* __remote__ submit trial jobs to remote ubuntu machines, and __machineList__ field should be filed in order to set up SSH connection to remote machine.
* __remote__ submit trial jobs to remote ubuntu machines, and __machineList__ field should be filed in order to set up SSH connection to remote machine.
* __pai__ submit trial jobs to [OpenPai](https://github.com/Microsoft/pai) of Microsoft. For more details of pai configuration, please reference [PAIMOdeDoc](./PaiMode.md)
* __pai__ submit trial jobs to [OpenPai](https://github.com/Microsoft/pai) of Microsoft. For more details of pai configuration, please reference [PAIMOdeDoc](../TrainingService/PaiMode.md)
* __kubeflow__ submit trial jobs to [kubeflow](https://www.kubeflow.org/docs/about/kubeflow/), NNI support kubeflow based on normal kubernetes and [azure kubernetes](https://azure.microsoft.com/en-us/services/kubernetes-service/).
* __kubeflow__ submit trial jobs to [kubeflow](https://www.kubeflow.org/docs/about/kubeflow/), NNI support kubeflow based on normal kubernetes and [azure kubernetes](https://azure.microsoft.com/en-us/services/kubernetes-service/). Detail please reference [KubeflowDoc](../TrainingService/KubeflowMode.md)
* __searchSpacePath__
* __searchSpacePath__
* Description
* Description
...
@@ -192,6 +206,16 @@ machineList:
...
@@ -192,6 +206,16 @@ machineList:
Note: if set useAnnotation=True, the searchSpacePath field should be removed.
Note: if set useAnnotation=True, the searchSpacePath field should be removed.
__multiThread__ enable multi-thread mode for dispatcher, if multiThread is set to `true`, dispatcher will start a thread to process each command from NNI Manager.
There are three parts that might have logs in NNI. They are nnimanager, dispatcher and trial. Here we will introduce them succinctly. More information please refer to [Overview](Overview.md).
There are three parts that might have logs in NNI. They are nnimanager, dispatcher and trial. Here we will introduce them succinctly. More information please refer to [Overview](../Overview.md).
-**NNI controller**: NNI controller (nnictl) is the nni command-line tool that is used to manage experiments (e.g., start an experiment).
-**NNI controller**: NNI controller (nnictl) is the nni command-line tool that is used to manage experiments (e.g., start an experiment).
-**nnimanager**: nnimanager is the core of NNI, whose log is important when the whole experiment fails (e.g., no webUI or training service fails)
-**nnimanager**: nnimanager is the core of NNI, whose log is important when the whole experiment fails (e.g., no webUI or training service fails)
...
@@ -57,7 +57,7 @@ Dispatcher fails. Usually, for some new users of NNI, it means that tuner fails.
...
@@ -57,7 +57,7 @@ Dispatcher fails. Usually, for some new users of NNI, it means that tuner fails.
Take the later situation as an example. If you write a customized tuner who's \_\_init\_\_ function has an argument called `optimize_mode`, which you do not provide in your configuration file, NNI will fail to run your tuner so the experiment fails. You can see errors in the webUI like:
Take the later situation as an example. If you write a customized tuner who's \_\_init\_\_ function has an argument called `optimize_mode`, which you do not provide in your configuration file, NNI will fail to run your tuner so the experiment fails. You can see errors in the webUI like:
Here we can see it is a dispatcher error. So we can check dispatcher's log, which might look like:
Here we can see it is a dispatcher error. So we can check dispatcher's log, which might look like:
...
@@ -82,7 +82,7 @@ It means your trial code (which is run by NNI) fails. This kind of error is stro
...
@@ -82,7 +82,7 @@ It means your trial code (which is run by NNI) fails. This kind of error is stro
A common example of this would be run the mnist example without installing tensorflow. Surely there is an Import Error (that is, not installing tensorflow but trying to import it in your trial code) and thus every trial fails.
A common example of this would be run the mnist example without installing tensorflow. Surely there is an Import Error (that is, not installing tensorflow but trying to import it in your trial code) and thus every trial fails.
As it shows, every trial has a log path, where you can find trial'log and stderr.
As it shows, every trial has a log path, where you can find trial'log and stderr.
@@ -47,7 +47,7 @@ After you prepare NNI's environment, you could start a new experiment using `nni
...
@@ -47,7 +47,7 @@ After you prepare NNI's environment, you could start a new experiment using `nni
## Using docker in remote platform
## Using docker in remote platform
NNI support starting experiments in [remoteTrainingService](RemoteMachineMode.md), and run trial jobs in remote machines. As docker could start an independent Ubuntu system as SSH server, docker container could be used as the remote machine in NNI's remot mode.
NNI support starting experiments in [remoteTrainingService](../TrainingService/RemoteMachineMode.md), and run trial jobs in remote machines. As docker could start an independent Ubuntu system as SSH server, docker container could be used as the remote machine in NNI's remot mode.
### Step 1: Setting docker environment
### Step 1: Setting docker environment
...
@@ -78,7 +78,7 @@ If you use your own docker image as remote server, please make sure that this im
...
@@ -78,7 +78,7 @@ If you use your own docker image as remote server, please make sure that this im
### Step3: Run NNI experiments
### Step3: Run NNI experiments
You could set your config file as remote platform, and setting the `machineList` configuration to connect your docker SSH server, [refer](RemoteMachineMode.md). Note that you should set correct `port`,`username` and `passwd` or `sshKeyPath` of your host machine.
You could set your config file as remote platform, and setting the `machineList` configuration to connect your docker SSH server, [refer](../TrainingService/RemoteMachineMode.md). Note that you should set correct `port`,`username` and `passwd` or `sshKeyPath` of your host machine.
`port:` The host machine's port, mapping to docker's SSH port.
`port:` The host machine's port, mapping to docker's SSH port.
@@ -367,9 +368,36 @@ Debug mode will disable version check function in Trialkeeper.
...
@@ -367,9 +368,36 @@ Debug mode will disable version check function in Trialkeeper.
* Usage
* Usage
```bash
```bash
nnictl experiment list
nnictl experiment list [OPTIONS]
```
```
* Options
|Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|--all| False| |list all of experiments|
* __nnictl experiment delete__
* Description
Delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
* Usage
```bash
nnictl experiment delete [OPTIONS]
```
* Options
|Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|id| False| |ID of the experiment|
|--all| False| |delete all of experiments|
<aname="export"></a>
<aname="export"></a>
* __nnictl experiment export__
* __nnictl experiment export__
...
@@ -435,7 +463,7 @@ Debug mode will disable version check function in Trialkeeper.
...
@@ -435,7 +463,7 @@ Debug mode will disable version check function in Trialkeeper.
Currently, following tuner and advisor support import data:
Currently, following tuner and advisor support import data:
It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
* Note
if the target platform is being used by other users, it may cause unexpected errors to others.
* Usage
```bash
nnictl platform clean [OPTIONS]
```
* Options
|Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|--config| True| |the path of yaml config file used when create an experiment|
@@ -53,7 +53,7 @@ The above code can only try one set of parameters at a time, if we want to tune
...
@@ -53,7 +53,7 @@ The above code can only try one set of parameters at a time, if we want to tune
NNI is born for helping user do the tuning jobs, the NNI working process is presented below:
NNI is born for helping user do the tuning jobs, the NNI working process is presented below:
```pseudo
```
input: search space, trial code, config file
input: search space, trial code, config file
output: one optimal hyperparameter configuration
output: one optimal hyperparameter configuration
...
@@ -201,28 +201,28 @@ Click the tab "Overview".
...
@@ -201,28 +201,28 @@ Click the tab "Overview".
Information about this experiment will be shown in the WebUI, including the experiment trial profile and search space message. NNI also support `download these information and parameters` through the **Download** button. You can download the experiment result anytime in the middle for the running or at the end of the execution, etc.
Information about this experiment will be shown in the WebUI, including the experiment trial profile and search space message. NNI also support `download these information and parameters` through the **Download** button. You can download the experiment result anytime in the middle for the running or at the end of the execution, etc.
Top 10 trials will be listed in the Overview page, you can browse all the trials in "Trials Detail" page.
Top 10 trials will be listed in the Overview page, you can browse all the trials in "Trials Detail" page.
#### View trials detail page
#### View trials detail page
Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
Click the tab "Hyper Parameter" to see the parallel graph.
Click the tab "Hyper Parameter" to see the parallel graph.
* You can select the percentage to see top trials.
* You can select the percentage to see top trials.
* Choose two axis to swap its positions
* Choose two axis to swap its positions
Click the tab "Trial Duration" to see the bar graph.
Click the tab "Trial Duration" to see the bar graph.
Below is the status of the all trials. Specifically:
Below is the status of the all trials. Specifically:
...
@@ -231,20 +231,20 @@ Below is the status of the all trials. Specifically:
...
@@ -231,20 +231,20 @@ Below is the status of the all trials. Specifically:
* Kill: you can kill a job that status is running.
* Kill: you can kill a job that status is running.
* Support to search for a specific trial.
* Support to search for a specific trial.
* Intermediate Result Graph
* Intermediate Result Graph
## Related Topic
## Related Topic
*[Try different Tuners](BuiltinTuner.md)
*[Try different Tuners](../Tuner/BuiltinTuner.md)
*[Try different Assessors](BuiltinAssessors.md)
*[Try different Assessors](../Assessor/BuiltinAssessor.md)
*[How to use command line tool nnictl](Nnictl.md)
*[How to use command line tool nnictl](Nnictl.md)
*[How to write a trial](Trials.md)
*[How to write a trial](../TrialExample/Trials.md)
*[How to run an experiment on local (with multiple GPUs)?](LocalMode.md)
*[How to run an experiment on local (with multiple GPUs)?](../TrainingService/LocalMode.md)
*[How to run an experiment on multiple machines?](RemoteMachineMode.md)
*[How to run an experiment on multiple machines?](../TrainingService/RemoteMachineMode.md)
*[How to run an experiment on OpenPAI?](PaiMode.md)
*[How to run an experiment on OpenPAI?](../TrainingService/PaiMode.md)
*[How to run an experiment on Kubernetes through Kubeflow?](KubeflowMode.md)
*[How to run an experiment on Kubernetes through Kubeflow?](../TrainingService/KubeflowMode.md)
*[How to run an experiment on Kubernetes through FrameworkController?](FrameworkControllerMode.md)
*[How to run an experiment on Kubernetes through FrameworkController?](../TrainingService/FrameworkControllerMode.md)
* Only Random Search/TPE/Anneal/Evolution tuner supports nested search space
* Only Random Search/TPE/Anneal/Evolution tuner supports nested search space
* We do not support nested search space "Hyper Parameter" parallel graph now, the enhancement is being considered in #1110(https://github.com/microsoft/nni/issues/1110), any suggestions or discussions or contributions are warmly welcomed
* We do not support nested search space "Hyper Parameter" in visualization now, the enhancement is being considered in #1110(https://github.com/microsoft/nni/issues/1110), any suggestions or discussions or contributions are warmly welcomed
* If you have any question, you can click "Feedback" to report it.
* If you have any question, you can click "Feedback" to report it.
* If your experiment have more than 1000 trials, you can change the refresh interval on here.
* If your experiment have more than 1000 trials, you can change the refresh interval on here.
* See good performance trials.
* See good performance trials.
## View job default metric
## View job default metric
Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
## View hyper parameter
## View hyper parameter
...
@@ -28,24 +28,24 @@ Click the tab "Hyper Parameter" to see the parallel graph.
...
@@ -28,24 +28,24 @@ Click the tab "Hyper Parameter" to see the parallel graph.
* You can select the percentage to see top trials.
* You can select the percentage to see top trials.
* Choose two axis to swap its positions
* Choose two axis to swap its positions
## View Trial Duration
## View Trial Duration
Click the tab "Trial Duration" to see the bar graph.
Click the tab "Trial Duration" to see the bar graph.
## View Trial Intermediate Result Graph
## View Trial Intermediate Result Graph
Click the tab "Intermediate Result" to see the lines graph.
Click the tab "Intermediate Result" to see the lines graph.
The graph has a filter function. You can open the filter button. And then enter your focus point
The graph has a filter function. You can open the filter button. And then enter your focus point
in the scape input. Simultaneously, intermediate result inputs can limit the intermediate's range.
in the scape input. Simultaneously, intermediate result inputs can limit the intermediate's range.
## View trials status
## View trials status
...
@@ -53,27 +53,27 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically:
...
@@ -53,27 +53,27 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically:
* Trial detail: trial's id, trial's duration, start time, end time, status, accuracy and search space file.
* Trial detail: trial's id, trial's duration, start time, end time, status, accuracy and search space file.
* The button named "Add column" can select which column to show in the table. If you run an experiment that final result is dict, you can see other keys in the table.
* The button named "Add column" can select which column to show in the table. If you run an experiment that final result is dict, you can see other keys in the table.
* If you want to compare some trials, you can select them and then click "Compare" to see the results.
* If you want to compare some trials, you can select them and then click "Compare" to see the results.
* You can use the button named "Copy as python" to copy trial's parameters.
* You can use the button named "Copy as python" to copy trial's parameters.
* If you run on OpenPAI or Kubeflow platform, you can also see the hdfsLog.
* If you run on OpenPAI or Kubeflow platform, you can also see the hdfsLog.
* Kill: you can kill a job that status is running.
* Kill: you can kill a job that status is running.
* Support to search for a specific trial.
* Support to search for a specific trial.
* Intermediate Result Graph: you can see default and other keys in this graph.
* Intermediate Result Graph: you can see default and other keys in this graph.
