Skip to content

GitLab

Commit e31839cc authored by Lee's avatar Lee Committed by xuehui
Browse files

Dev networkmorphism (#413) 

* Quick fix nnictl config logic (#289)

* fix nnictl bug

* fix install.sh

* add desc for Dockerfile.build.base

* update document for Dockerfile

* update

* refactor port detect

* update

* refactor NNICTLDOC.md

* add document for pai and nnictl

* add default value for port

* add exception handling in trial_keeper.py

* fix port bug

* fix resume

* fix nnictl resume and fix nnictl stop

* fix document

* update

* refactor nnictl

* update

* update doc

* update

* update nnictl

* fix comment

* revert dockerfile

* update

* update

* update

* fix nnictl error hit

* fix comments

* fix bash-completion

* fix paramiko install

* quick fix resume logic

* update

* quick fix nnictl

* PR merge to 0.3 (#297)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* update doc

* fix links

* fix links

* fix links

* merge

* fix links and doc errors

* merge

* merge

* merge

* merge

* Update README.md (#288)

added License badge

* merge

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* fix doc mistakes and broken links. (#271)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* Update README.md

* Fix misspelling in examples/trials/ga_squad/README.md

* revise the installation cmd to v0.2

* revise to install v0.2

* remove enas readme (#292)

* Fix datastore performance issue (#301)

* Fix nnictl in v0.3 (#299)

Fix old version of config file
fix sklearn requirements
Fix resume log logic

* add basic tuner and trial for network morphism

* Complete basic receive_trial_result() and generate_parameters(). Use onnx  as the intermediate representation ( But it cannot convert to pytorch model )

* add tensorflow cifar10 for network morphism

* add unit test for tuner and its function

* use temporary torch_model

* fix request bug and program can communicate nni

* add basic pickle support for graph and train successful in pytorch

* Update unittest for networkmorphism_tuner

* Network Morphism add multi-gpu trial training support

* Format code with black tool

* change intermediate representation from pickle file to json we defined

* successfully pass the unittest for test_graph_json_transform

* add README for network morphism and it works fine in both Pytorch and Keras.

* separate the original Readme.md in network-morphism into two parts (tuner and trial)

* change the openpai image path

* beautify the file structure of network_morphism and add a fashion_mnist keras example

* pretty the source and add some docstring for funtion in order to pass the pylint.

* remove unused module import and add some docstring

* add some details for the application scenario Network Morphism Tuner

* follow the advice and modify the doc file

* add the config file for each task in the examples trial of network morphism

* change default python interpreter from python to python3
parent 04257666
Hide whitespace changes
Inline Side-by-side
Showing with 1517 additions and 143 deletions
.gitignore
View file @ e31839cc
...@@ -66,4 +66,4 @@ typings/ ...@@ -66,4 +66,4 @@ typings/
# Python cache files # Python cache files
__pycache__ __pycache__
build build
*.egg-info *.egg-info
\ No newline at end of file
docs/HowToChooseTuner.md
View file @ e31839cc
...@@ -10,6 +10,7 @@ For now, NNI has supported the following tuner algorithms. Note that NNI install ...@@ -10,6 +10,7 @@ For now, NNI has supported the following tuner algorithms. Note that NNI install
- [Batch Tuner](#Batch) - [Batch Tuner](#Batch)
- [Grid Search](#Grid) - [Grid Search](#Grid)
- [Hyperband](#Hyperband) - [Hyperband](#Hyperband)
- [Network Morphism](#NetworkMorphism)
## Supported tuner algorithms ## Supported tuner algorithms
...@@ -22,11 +23,11 @@ We will introduce some basic knowledge about the tuner algorithms, suggested sce ...@@ -22,11 +23,11 @@ We will introduce some basic knowledge about the tuner algorithms, suggested sce
The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.
The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities.
This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization][1]. This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization][1].
​
_Suggested scenario_: TPE, as a black-box optimization, can be used in various scenarios, and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search. _Suggested scenario_: TPE, as a black-box optimization, can be used in various scenarios, and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: TPE builtinTunerName: TPE
...@@ -43,7 +44,7 @@ In [Random Search for Hyper-Parameter Optimization][2] show that Random Search m ...@@ -43,7 +44,7 @@ In [Random Search for Hyper-Parameter Optimization][2] show that Random Search m
_Suggested scenario_: Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm. _Suggested scenario_: Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: Random builtinTunerName: Random
...@@ -57,7 +58,7 @@ This simple annealing algorithm begins by sampling from the prior, but tends ove ...@@ -57,7 +58,7 @@ This simple annealing algorithm begins by sampling from the prior, but tends ove
_Suggested scenario_: Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution. _Suggested scenario_: Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: Anneal builtinTunerName: Anneal
...@@ -74,7 +75,7 @@ Naive Evolution comes from [Large-Scale Evolution of Image Classifiers][3]. It r ...@@ -74,7 +75,7 @@ Naive Evolution comes from [Large-Scale Evolution of Image Classifiers][3]. It r
_Suggested scenario_: Its requirement of computation resource is relatively high. Specifically, it requires large inital population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress. _Suggested scenario_: Its requirement of computation resource is relatively high. Specifically, it requires large inital population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: Evolution builtinTunerName: Evolution
...@@ -93,7 +94,7 @@ Note that SMAC on nni only supports a subset of the types in [search space spec] ...@@ -93,7 +94,7 @@ Note that SMAC on nni only supports a subset of the types in [search space spec]
_Suggested scenario_: Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete. _Suggested scenario_: Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: SMAC builtinTunerName: SMAC
...@@ -110,14 +111,14 @@ Batch tuner allows users to simply provide several configurations (i.e., choices ...@@ -110,14 +111,14 @@ Batch tuner allows users to simply provide several configurations (i.e., choices
_Suggested sceanrio_: If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner. _Suggested sceanrio_: If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: BatchTuner builtinTunerName: BatchTuner
``` ```
Note that the search space that BatchTuner supported like: Note that the search space that BatchTuner supported like:
``` ```json
{ {
"combine_params": "combine_params":
{ {
...@@ -133,7 +134,6 @@ Note that the search space that BatchTuner supported like: ...@@ -133,7 +134,6 @@ Note that the search space that BatchTuner supported like:
``` ```
The search space file including the high-level key `combine_params`. The type of params in search space must be `choice` and the `values` including all the combined-params value. The search space file including the high-level key `combine_params`. The type of params in search space must be `choice` and the `values` including all the combined-params value.
<a name="Grid"></a> <a name="Grid"></a>
**Grid Search** **Grid Search**
...@@ -143,7 +143,7 @@ Note that the only acceptable types of search space are `choice`, `quniform`, `q ...@@ -143,7 +143,7 @@ Note that the only acceptable types of search space are `choice`, `quniform`, `q
_Suggested scenario_: It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space. _Suggested scenario_: It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
tuner: tuner:
builtinTunerName: GridSearch builtinTunerName: GridSearch
...@@ -152,12 +152,12 @@ _Usage_: ...@@ -152,12 +152,12 @@ _Usage_:
<a name="Hyperband"></a> <a name="Hyperband"></a>
**Hyperband** **Hyperband**
[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be refered to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md) [Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be referred to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md).
_Suggested scenario_: It is suggested when you have limited computation resource but have relatively large search space. It performs good in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent. _Suggested scenario_: It is suggested when you have limited computation resource but have relatively large search space. It performs good in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent.
_Usage_: _Usage_:
``` ```yaml
# config.yaml # config.yaml
advisor: advisor:
builtinAdvisorName: Hyperband builtinAdvisorName: Hyperband
...@@ -171,6 +171,31 @@ _Usage_: ...@@ -171,6 +171,31 @@ _Usage_:
eta: 3 eta: 3
``` ```
<a name="NetworkMorphism"></a>
**Network Morphism**
[Network Morphism](7) provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md).
_Suggested scenario_: It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](../examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the cv domain.
_Usage_:
```yaml
# config.yaml
tuner:
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
#for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 32
#input image channel
input_channel: 3
#number of classes
n_output_node: 10
```
# How to use Assessor that NNI supports? # How to use Assessor that NNI supports?
...@@ -184,12 +209,12 @@ For now, NNI has supported the following assessor algorithms. ...@@ -184,12 +209,12 @@ For now, NNI has supported the following assessor algorithms.
<a name="Medianstop"></a> <a name="Medianstop"></a>
**Medianstop** **Medianstop**
Medianstop is a simple early stopping rule mentioned in the [paper][7]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S. Medianstop is a simple early stopping rule mentioned in the [paper][8]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
_Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. _Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
_Usage_: _Usage_:
``` ```yaml
assessor: assessor:
builtinAssessorName: Medianstop builtinAssessorName: Medianstop
classArgs: classArgs:
...@@ -201,10 +226,11 @@ _Usage_: ...@@ -201,10 +226,11 @@ _Usage_:
start_step: 5 start_step: 5
``` ```
[1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf [1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
[2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf [2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
[3]: https://arxiv.org/pdf/1703.01041.pdf [3]: https://arxiv.org/pdf/1703.01041.pdf
[4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf [4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
[5]: https://github.com/automl/SMAC3 [5]: https://github.com/automl/SMAC3
[6]: https://arxiv.org/pdf/1603.06560.pdf [6]: https://arxiv.org/pdf/1603.06560.pdf
[7]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf [7]: https://arxiv.org/abs/1806.10282
\ No newline at end of file [8]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
\ No newline at end of file
docs/KubeflowMode.md
View file @ e31839cc
**Run an Experiment on Kubeflow** **Run an Experiment on Kubeflow**
=== ===
Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/kubeflow), called kubeflow mode. Before starting to use NNI kubeflow mode, you should have a kubernetes cluster, either on-prem or [Azure Kubernetes Service(AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/), a Ubuntu machine on which [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is installed and configured to connect to your kubernetes cluster. If you are not familiar with kubernetes, [here](https://kubernetes.io/docs/tutorials/kubernetes-basics/) is a goot start. In kubeflow mode, your trial program will run as kubeflow job in kubernetes cluster. Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/kubeflow), called kubeflow mode. Before starting to use NNI kubeflow mode, you should have a kubernetes cluster, either on-prem or [Azure Kubernetes Service(AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/), a Ubuntu machine on which [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is installed and configured to connect to your kubernetes cluster. If you are not familiar with kubernetes, [here](https://kubernetes.io/docs/tutorials/kubernetes-basics/) is a goot start. In kubeflow mode, your trial program will run as kubeflow job in kubernetes cluster.
## Prerequisite for on-premises Kubernetes Service ## Prerequisite for on-premises Kubernetes Service
1. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this [guideline](https://kubernetes.io/docs/setup/) to set up Kubernetes 1. A **Kubernetes** cluster using Kubernetes 1.8 or later. Follow this [guideline](https://kubernetes.io/docs/setup/) to set up Kubernetes
2. Download, set up, and deploy **Kubelow** to your Kubernetes cluster. Follow this [guideline](https://www.kubeflow.org/docs/started/getting-started/) to set up Kubeflow 2. Download, set up, and deploy **Kubelow** to your Kubernetes cluster. Follow this [guideline](https://www.kubeflow.org/docs/started/getting-started/) to set up Kubeflow
3. Install **kubectl**, and configure to connect to your Kubernetes API server. Follow this [guideline](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to install kubectl on Ubuntu 3. Install **kubectl**, and configure to connect to your Kubernetes API server. Follow this [guideline](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to install kubectl on Ubuntu
4. If your NNI trial job needs GPU resource, you should follow this [guideline](https://github.com/NVIDIA/k8s-device-plugin) to configure **Nvidia device plugin for Kubernetes**. 4. If your NNI trial job needs GPU resource, you should follow this [guideline](https://github.com/NVIDIA/k8s-device-plugin) to configure **Nvidia device plugin for Kubernetes**.
5. Install **NFS server** and export a general purpose mount (we recommend to map your NFS server path in `root_squash option`, otherwise permission issue may raise when nni copy files to NFS. Refer this [page](https://linux.die.net/man/5/exports) to learn what root_squash option is), or **Azure File Storage**. 5. Install **NFS server** and export a general purpose mount (we recommend to map your NFS server path in `root_squash option`, otherwise permission issue may raise when nni copy files to NFS. Refer this [page](https://linux.die.net/man/5/exports) to learn what root_squash option is), or **Azure File Storage**.
6. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client: 6. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
``` ```
apt-get install nfs-common apt-get install nfs-common
``` ```
7. Install **NNI**, follow the install guide [here](GetStarted.md). 7. Install **NNI**, follow the install guide [here](GetStarted.md).
## Prerequisite for Azure Kubernetes Service ## Prerequisite for Azure Kubernetes Service
1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service. 1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service.
2. Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and __kubectl__. Use `az login` to set azure account, and connect kubectl client to AKS, [refer](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#connect-to-the-cluster). 2. Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and __kubectl__. Use `az login` to set azure account, and connect kubectl client to AKS, [refer](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#connect-to-the-cluster).
3. Deploy kubeflow on Azure Kubernetes Service, follow the [guideline](https://www.kubeflow.org/docs/started/getting-started/). 3. Deploy kubeflow on Azure Kubernetes Service, follow the [guideline](https://www.kubeflow.org/docs/started/getting-started/).
4. Follow the [guideline](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal) to create azure file storage account. If you use Azure Kubernetes Service, nni need Azure Storage Service to store code files and the output files. 4. Follow the [guideline](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal) to create azure file storage account. If you use Azure Kubernetes Service, nni need Azure Storage Service to store code files and the output files.
5. To access Azure storage service, nni need the access key of the storage account, and nni use [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) Service to protect your private key. Set up Azure Key Vault Service, add a secret to Key Vault to store the access key of Azure storage account. Follow this [guideline](https://docs.microsoft.com/en-us/azure/key-vault/quick-create-cli) to store the access key. 5. To access Azure storage service, nni need the access key of the storage account, and nni use [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) Service to protect your private key. Set up Azure Key Vault Service, add a secret to Key Vault to store the access key of Azure storage account. Follow this [guideline](https://docs.microsoft.com/en-us/azure/key-vault/quick-create-cli) to store the access key.
## Design ## Design
TODO TODO
## Run an experiment ## Run an experiment
Use `examples/trials/mnist` as an example. The nni config yaml file's content is like: Use `examples/trials/mnist` as an example. The nni config yaml file's content is like:
``` ```
authorName: your_name authorName: your_name
experimentName: example_mnist experimentName: example_mnist
# how many trials could be concurrently running # how many trials could be concurrently running
trialConcurrency: 4 trialConcurrency: 4
# maximum experiment running duration # maximum experiment running duration
maxExecDuration: 3h maxExecDuration: 3h
# empty means never stop # empty means never stop
maxTrialNum: 100 maxTrialNum: 100
# choice: local, remote, pai, kubeflow # choice: local, remote, pai, kubeflow
trainingServicePlatform: kubeflow trainingServicePlatform: kubeflow
# choice: true, false # choice: true, false
useAnnotation: false useAnnotation: false
tuner: tuner:
builtinTunerName: TPE builtinTunerName: TPE
classArgs: classArgs:
#choice: maximize, minimize #choice: maximize, minimize
optimize_mode: maximize optimize_mode: maximize
trial: trial:
codeDir: ~/nni/examples/trials/mnist codeDir: ~/nni/examples/trials/mnist
ps: ps:
replicas: 1 replicas: 1
command: python mnist-keras.py command: python mnist-keras.py
gpuNum: 0 gpuNum: 0
cpuNum: 1 cpuNum: 1
memoryMB: 8196 memoryMB: 8196
image: {your_docker_image_for_tensorflow_ps} image: {your_docker_image_for_tensorflow_ps}
worker: worker:
replicas: 1 replicas: 1
command: python mnist-keras.py command: python mnist-keras.py
gpuNum: 2 gpuNum: 2
cpuNum: 1 cpuNum: 1
memoryMB: 8196 memoryMB: 8196
image: {your_docker_image_for_tensorflow_worker} image: {your_docker_image_for_tensorflow_worker}
kubeflowConfig: kubeflowConfig:
operator: tf-operator operator: tf-operator
storage: nfs storage: nfs
nfs: nfs:
server: {your_nfs_server} server: {your_nfs_server}
path: {your_nfs_server_exported_path} path: {your_nfs_server_exported_path}
``` ```
If you use Azure Kubernetes Service, you should set `kubeflowConfig` in your config yaml file as follows: If you use Azure Kubernetes Service, you should set `kubeflowConfig` in your config yaml file as follows:
``` ```
kubeflowConfig: kubeflowConfig:
operator: tf-operator operator: tf-operator
storage: azureStorage storage: azureStorage
keyVault: keyVault:
vaultName: {your_vault_name} vaultName: {your_vault_name}
name: {your_secert_name} name: {your_secert_name}
azureStorage: azureStorage:
accountName: {your_storage_account_name} accountName: {your_storage_account_name}
azureShare: {your_azure_share_name} azureShare: {your_azure_share_name}
``` ```
Note: You should explicitly set `trainingServicePlatform: kubeflow` in nni config yaml file if you want to start experiment in kubeflow mode. Note: You should explicitly set `trainingServicePlatform: kubeflow` in nni config yaml file if you want to start experiment in kubeflow mode.
Trial configuration in kubeflow mode have the following configuration keys: Trial configuration in kubeflow mode have the following configuration keys:
* codeDir * codeDir
* code directory, where you put training code and config files * code directory, where you put training code and config files
* worker (required). This config section is used to configure tensorflow worker role * worker (required). This config section is used to configure tensorflow worker role
* replicas * replicas
* Required key. Should be positive number depends on how many replication your want to run for tensorflow worker role. * Required key. Should be positive number depends on how many replication your want to run for tensorflow worker role.
* command * command
* Required key. Command to launch your trial job, like ```python mnist.py``` * Required key. Command to launch your trial job, like ```python mnist.py```
* memoryMB * memoryMB
* Required key. Should be positive number based on your trial program's memory requirement * Required key. Should be positive number based on your trial program's memory requirement
* cpuNum * cpuNum
* gpuNum * gpuNum
* image * image
* Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run. * Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run.
* We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it. * We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
* ps (optional). This config section is used to configure tensorflow parameter server role. * ps (optional). This config section is used to configure tensorflow parameter server role.
Once complete to fill nni experiment config file and save (for example, save as exp_kubeflow.yaml), then run the following command Once complete to fill nni experiment config file and save (for example, save as exp_kubeflow.yaml), then run the following command
``` ```
nnictl create --config exp_kubeflow.yaml nnictl create --config exp_kubeflow.yaml
``` ```
to start the experiment in kubeflow mode. NNI will create Kubeflow tfjob for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`. to start the experiment in kubeflow mode. NNI will create Kubeflow tfjob for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`.
You can see the kubeflow tfjob created by NNI in your Kubernetes dashboard. You can see the kubeflow tfjob created by NNI in your Kubernetes dashboard.
Notice: In kubeflow mode, NNIManager will start a rest server and listen on a port which is your NNI WebUI's port plus 1. For example, if your WebUI port is `8080`, the rest server will listen on `8081`, to receive metrics from trial job running in Kubernetes. So you should `enable 8081` TCP port in your firewall rule to allow incoming traffic. Notice: In kubeflow mode, NNIManager will start a rest server and listen on a port which is your NNI WebUI's port plus 1. For example, if your WebUI port is `8080`, the rest server will listen on `8081`, to receive metrics from trial job running in Kubernetes. So you should `enable 8081` TCP port in your firewall rule to allow incoming traffic.
Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information. Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information.
Any problems when using NNI in kubeflow mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni), or send mail to nni@microsoft.com Any problems when using NNI in kubeflow mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni), or send mail to nni@microsoft.com
examples/trials/cifar10_pytorch/search_space.json
View file @ e31839cc
{ {
"lr":{"_type":"choice", "_value":[0.1, 0.01, 0.001, 0.0001]}, "lr":{"_type":"choice", "_value":[0.1, 0.01, 0.001, 0.0001]},
"optimizer":{"_type":"choice", "_value":["SGD", "Adadelta", "Adagrad", "Adam", "Adamax"]}, "optimizer":{"_type":"choice", "_value":["SGD", "Adadelta", "Adagrad", "Adam", "Adamax"]},
"model":{"_type":"choice", "_value":["vgg", "resnet18", "googlenet", "densenet121", "mobilenet", "dpn92", "senet18"]} "model":{"_type":"choice", "_value":["vgg", "resnet18", "googlenet", "densenet121", "mobilenet", "dpn92", "senet18"]}
} }
examples/trials/ga_squad/README.md
View file @ e31839cc
...@@ -159,7 +159,7 @@ The trial has a lot of different files, functions and classes. Here we will only ...@@ -159,7 +159,7 @@ The trial has a lot of different files, functions and classes. Here we will only
Among those files, `trial.py` and `graph_to_tf.py` is special. Among those files, `trial.py` and `graph_to_tf.py` is special.
`graph_to_tf.py` has a function named as `graph_to_network`, here is its skelton code: `graph_to_tf.py` has a function named as `graph_to_network`, here is its skeleton code:
``` ```
def graph_to_network(input1, def graph_to_network(input1,
......
examples/trials/network_morphism/FashionMNIST/FashionMNIST_keras.py 0 → 100644
View file @ e31839cc
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import logging
import os
import tensorflow as tf
import keras
from keras.callbacks import EarlyStopping, TensorBoard
from keras.datasets import fashion_mnist
from keras.optimizers import SGD, Adadelta, Adagrad, Adam, Adamax, RMSprop
from keras.utils import multi_gpu_model, to_categorical
import keras.backend.tensorflow_backend as KTF
import nni
from nni.networkmorphism_tuner.graph import json_to_graph
# set the logger format
log_format = "%(asctime)s %(message)s"
logging.basicConfig(
filename="networkmorphism.log",
filemode="a",
level=logging.INFO,
format=log_format,
datefmt="%m/%d %I:%M:%S %p",
)
# set the logger format
logger = logging.getLogger("fashion_mnist-network-morphism-keras")
# restrict gpu usage background
config = tf.ConfigProto()
# pylint: disable=E1101,W0603
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
KTF.set_session(sess)
def get_args():
""" get args from command line
"""
parser = argparse.ArgumentParser("fashion_mnist")
parser.add_argument("--batch_size", type=int, default=128, help="batch size")
parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
parser.add_argument(
"--learning_rate", type=float, default=0.001, help="learning rate"
)
parser.add_argument(
"--weight_decay",
type=float,
default=1e-5,
help="weight decay of the learning rate",
)
return parser.parse_args()
trainloader = None
testloader = None
net = None
args = get_args()
TENSORBOARD_DIR = os.environ["NNI_OUTPUT_DIR"]
def build_graph_from_json(ir_model_json):
"""build model from json representation
"""
graph = json_to_graph(ir_model_json)
logging.debug(graph.operation_history)
model = graph.produce_keras_model()
return model
def parse_rev_args(receive_msg):
""" parse reveive msgs to global variable
"""
global trainloader
global testloader
global net
# Loading Data
logger.debug("Preparing data..")
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)
x_train = x_train.reshape(x_train.shape+(1,)).astype("float32")
x_test = x_test.reshape(x_test.shape+(1,)).astype("float32")
x_train /= 255.0
x_test /= 255.0
trainloader = (x_train, y_train)
testloader = (x_test, y_test)
# Model
logger.debug("Building model..")
net = build_graph_from_json(receive_msg)
# parallel model
try:
available_devices = os.environ["CUDA_VISIBLE_DEVICES"]
gpus = len(available_devices.split(","))
if gpus > 1:
net = multi_gpu_model(net, gpus)
except KeyError:
logger.debug("parallel model not support in this config settings")
if args.optimizer == "SGD":
optimizer = SGD(lr=args.learning_rate, momentum=0.9, decay=args.weight_decay)
if args.optimizer == "Adadelta":
optimizer = Adadelta(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adagrad":
optimizer = Adagrad(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adam":
optimizer = Adam(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adamax":
optimizer = Adamax(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "RMSprop":
optimizer = RMSprop(lr=args.learning_rate, decay=args.weight_decay)
# Compile the model
net.compile(
loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"]
)
return 0
class SendMetrics(keras.callbacks.Callback):
"""
Keras callback to send metrics to NNI framework
"""
def on_epoch_end(self, epoch, logs=None):
"""
Run on end of each epoch
"""
if logs is None:
logs = dict()
logger.debug(logs)
nni.report_intermediate_result(logs["acc"])
# Training
def train_eval():
""" train and eval the model
"""
global trainloader
global testloader
global net
(x_train, y_train) = trainloader
(x_test, y_test) = testloader
# train procedure
net.fit(
x=x_train,
y=y_train,
batch_size=args.batch_size,
validation_data=(x_test, y_test),
epochs=args.epochs,
shuffle=True,
callbacks=[
SendMetrics(),
EarlyStopping(min_delta=0.001, patience=10),
TensorBoard(log_dir=TENSORBOARD_DIR),
],
)
# trial report final acc to tuner
_, acc = net.evaluate(x_test, y_test)
logger.debug("Final result is: %d", acc)
nni.report_final_result(acc)
if __name__ == "__main__":
try:
# trial get next parameter from network morphism tuner
RCV_CONFIG = nni.get_next_parameter()
logger.debug(RCV_CONFIG)
parse_rev_args(RCV_CONFIG)
train_eval()
except Exception as exception:
logger.exception(exception)
raise
examples/trials/network_morphism/FashionMNIST/FashionMNIST_pytorch.py 0 → 100644
View file @ e31839cc
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import logging
import sys
import nni
from nni.networkmorphism_tuner.graph import json_to_graph
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
sys.path.append("../")
from network_morphism import utils
# set the logger format
log_format = "%(asctime)s %(message)s"
logging.basicConfig(
filename="networkmorphism.log",
filemode="a",
level=logging.INFO,
format=log_format,
datefmt="%m/%d %I:%M:%S %p",
)
# pylint: disable=W0603
# set the logger format
logger = logging.getLogger("FashionMNIST-network-morphism")
def get_args():
""" get args from command line
"""
parser = argparse.ArgumentParser("FashionMNIST")
parser.add_argument("--batch_size", type=int, default=128, help="batch size")
parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
parser.add_argument(
"--learning_rate", type=float, default=0.001, help="learning rate"
)
parser.add_argument("--cutout", action="store_true", default=False, help="use cutout")
parser.add_argument("--cutout_length", type=int, default=8, help="cutout length")
parser.add_argument(
"--model_path", type=str, default="./", help="Path to save the destination model"
)
return parser.parse_args()
trainloader = None
testloader = None
net = None
criterion = None
optimizer = None
device = "cuda" if torch.cuda.is_available() else "cpu"
best_acc = 0.0
args = get_args()
def build_graph_from_json(ir_model_json):
"""build model from json representation
"""
graph = json_to_graph(ir_model_json)
logging.debug(graph.operation_history)
model = graph.produce_torch_model()
return model
def parse_rev_args(receive_msg):
""" parse reveive msgs to global variable
"""
global trainloader
global testloader
global net
global criterion
global optimizer
# Loading Data
logger.debug("Preparing data..")
raw_train_data = torchvision.datasets.FashionMNIST(
root="./data", train=True, download=True
)
dataset_mean, dataset_std = (
[raw_train_data.train_data.float().mean() / 255],
[raw_train_data.train_data.float().std() / 255],
)
transform_train, transform_test = utils.data_transforms_mnist(
args, dataset_mean, dataset_std
)
trainset = torchvision.datasets.FashionMNIST(
root="./data", train=True, download=True, transform=transform_train
)
trainloader = torch.utils.data.DataLoader(
trainset, batch_size=args.batch_size, shuffle=True, num_workers=2
)
testset = torchvision.datasets.FashionMNIST(
root="./data", train=False, download=True, transform=transform_test
)
testloader = torch.utils.data.DataLoader(
testset, batch_size=args.batch_size, shuffle=False, num_workers=2
)
# Model
logger.debug("Building model..")
net = build_graph_from_json(receive_msg)
net = net.to(device)
criterion = nn.CrossEntropyLoss()
if args.optimizer == "SGD":
optimizer = optim.SGD(
net.parameters(), lr=args.learning_rate, momentum=0.9, weight_decay=5e-4
)
if args.optimizer == "Adadelta":
optimizer = optim.Adadelta(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adagrad":
optimizer = optim.Adagrad(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adam":
optimizer = optim.Adam(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adamax":
optimizer = optim.Adamax(net.parameters(), lr=args.learning_rate)
if args.optimizer == "RMSprop":
optimizer = optim.RMSprop(net.parameters(), lr=args.learning_rate)
return 0
# Training
def train(epoch):
""" train model on each epoch in trainset
"""
global trainloader
global testloader
global net
global criterion
global optimizer
logger.debug("Epoch: %d", epoch)
net.train()
train_loss = 0
correct = 0
total = 0
for batch_idx, (inputs, targets) in enumerate(trainloader):
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()
train_loss += loss.item()
_, predicted = outputs.max(1)
total += targets.size(0)
correct += predicted.eq(targets).sum().item()
acc = 100.0 * correct / total
logger.debug(
"Loss: %.3f | Acc: %.3f%% (%d/%d)",
train_loss / (batch_idx + 1),
100.0 * correct / total,
correct,
total,
)
return acc
def test(epoch):
""" eval model on each epoch in testset
"""
global best_acc
global trainloader
global testloader
global net
global criterion
global optimizer
logger.debug("Eval on epoch: %d", epoch)
net.eval()
test_loss = 0
correct = 0
total = 0
with torch.no_grad():
for batch_idx, (inputs, targets) in enumerate(testloader):
inputs, targets = inputs.to(device), targets.to(device)
outputs = net(inputs)
loss = criterion(outputs, targets)
test_loss += loss.item()
_, predicted = outputs.max(1)
total += targets.size(0)
correct += predicted.eq(targets).sum().item()
acc = 100.0 * correct / total
logger.debug(
"Loss: %.3f | Acc: %.3f%% (%d/%d)",
test_loss / (batch_idx + 1),
100.0 * correct / total,
correct,
total,
)
acc = 100.0 * correct / total
if acc > best_acc:
best_acc = acc
return acc, best_acc
if __name__ == "__main__":
try:
# trial get next parameter from network morphism tuner
RCV_CONFIG = nni.get_next_parameter()
logger.debug(RCV_CONFIG)
parse_rev_args(RCV_CONFIG)
train_acc = 0.0
best_acc = 0.0
early_stop = utils.EarlyStopping(mode="max")
for ep in range(args.epochs):
train_acc = train(ep)
test_acc, best_acc = test(ep)
nni.report_intermediate_result(test_acc)
logger.debug(test_acc)
if early_stop.step(test_acc):
break
# trial report best_acc to tuner
nni.report_final_result(best_acc)
except Exception as exception:
logger.exception(exception)
raise
examples/trials/network_morphism/FashionMNIST/config.yml 0 → 100644
View file @ e31839cc
authorName: default
experimentName: example_FashionMNIST-network-morphism
trialConcurrency: 4
maxExecDuration: 48h
maxTrialNum: 200
#choice: local, remote, pai
trainingServicePlatform: local
#searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
#SMAC (SMAC should be installed through nnictl)
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
#for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 28
#input image channel
input_channel: 1
#number of classes
n_output_node: 10
trial:
command: python3 FashionMNIST_keras.py
codeDir: .
gpuNum: 1
examples/trials/network_morphism/FashionMNIST/config_pai.yml 0 → 100644
View file @ e31839cc
authorName: default
experimentName: example_FashionMNIST-network-morphism
trialConcurrency: 1
maxExecDuration: 24h
maxTrialNum: 10
#choice: local, remote, pai
trainingServicePlatform: pai
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
#SMAC (SMAC should be installed through nnictl)
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
# for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 28
#input image channel
input_channel: 1
#number of classes
n_output_node: 10
trial:
command: python3 FashionMNIST_keras.py
codeDir: .
gpuNum: 1
cpuNum: 1
memoryMB: 8196
#The docker image to run nni job on pai
image: msranni/nni:latest
#The hdfs directory to store data on pai, format 'hdfs://host:port/directory'
dataDir: hdfs://10.10.10.10:9000/username/nni
#The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
outputDir: hdfs://10.10.10.10:9000/username/nni
paiConfig:
#The username to login pai
userName: username
#The password to login pai
passWord: password
#The host of restful server of pai
host: 10.10.10.10
\ No newline at end of file
examples/trials/network_morphism/README.md 0 → 100644
View file @ e31839cc
# Network Morphism for Automatic Model Architecture Search in NNI
The Network Morphism is a build-in Tuner using network morphism techniques to search and evaluate the new network architecture. This example shows us how to use it to find good model architectures for deep learning.
## How to run this example?
### 1. Training framework support
The network morphism now is framework-based, and we have not implemented the framework-free methods. The training frameworks which we have supported yet are Pytorch and Keras. If you get familiar with the intermediate JSON format, you can build your own model in your own training framework. In the future, we will change to intermediate format from JSON to ONNX in order to get a [standard intermediate representation spec](https://github.com/onnx/onnx/blob/master/docs/IR.md).
### 2. Install the requirements
```bash
# install the requirements packages
cd examples/trials/network_morphism/
pip install -r requirements.txt
```
### 3. Update configuration
Modify `examples/trials/network_morphism/cifar10/config.yaml` to fit your own task, note that searchSpacePath is not required in our configuration. Here is the default configuration:
```yaml
authorName: default
experimentName: example_cifar10-network-morphism
trialConcurrency: 1
maxExecDuration: 48h
maxTrialNum: 200
#choice: local, remote, pai
trainingServicePlatform: local
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
#SMAC (SMAC should be installed through nnictl)
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
#for now, this tuner only supports cv domain
task: cv
#modify to fit your input image width
input_width: 32
#modify to fit your input image channel
input_channel: 3
#modify to fit your number of classes
n_output_node: 10
trial:
# your own command here
command: python3 cifar10_keras.py
codeDir: .
gpuNum: 0
```
In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
`trialConcurrency` is the number of trials running concurrently, which is the number of GPUs you want to use, if you are setting `gpuNum` to 1.
### 4. Call "json\_to\_graph()" function in your own code
Modify your code and call "json\_to\_graph()" function to build a pytorch model or keras model from received json string. Here is the simple example.
```python
import nni
from nni.networkmorphism_tuner.graph import json_to_graph
def build_graph_from_json(ir_model_json):
"""build a pytorch model from json representation
"""
graph = json_to_graph(ir_model_json)
model = graph.produce_torch_model()
return model
# trial get next parameter from network morphism tuner
RCV_CONFIG = nni.get_next_parameter()
# call the function to build pytorch model or keras model
net = build_graph_from_json(RCV_CONFIG)
# training procedure
# ....
# report the final accuracy to nni
nni.report_final_result(best_acc)
```
### 5. Submit this job
```bash
# You can use nni command tool "nnictl" to create the a job which submit to the nni
# finally you successfully commit a Network Morphism Job to nni
nnictl create --config config.yaml
```
## Trial Examples
The trial has some examples which can guide you which located in `examples/trials/network_morphism/`. You can refer to it and modify to your own task. Hope this will help you to build your code.
### FashionMNIST
`Fashion-MNIST` is a dataset of [Zalando](https://jobs.zalando.com/tech/)'s article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. It is a modern image classification dataset widely used to replacing MNIST as a baseline dataset, because the dataset MNIST is too easy and overused.
There are two examples, [FashionMNIST-keras.py](./FashionMNIST/FashionMNIST_keras.py) and [FashionMNIST-pytorch.py](./FashionMNIST/FashionMNIST_pytorch.py). Attention, you should change the `input_width` to 28 and `input_channel` to 1 in `config.yaml ` for this dataset.
### Cifar10
The `CIFAR-10` dataset [Canadian Institute For Advanced Research](https://www.cifar.ca/) is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10 different classes.
There are two examples, [cifar10-keras.py](./cifar10/cifar10_keras.py) and [cifar10-pytorch.py](./cifar10/cifar10_pytorch.py). The value `input_width` is 32 and the value `input_channel` is 3 in `config.yaml ` for this dataset.
\ No newline at end of file
examples/trials/network_morphism/cifar10/cifar10_keras.py 0 → 100644
View file @ e31839cc
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import logging
import os
import tensorflow as tf
import keras
from keras.callbacks import EarlyStopping, TensorBoard
from keras.datasets import cifar10
from keras.optimizers import SGD, Adadelta, Adagrad, Adam, Adamax, RMSprop
from keras.utils import multi_gpu_model, to_categorical
import keras.backend.tensorflow_backend as KTF
import nni
from nni.networkmorphism_tuner.graph import json_to_graph
# set the logger format
log_format = "%(asctime)s %(message)s"
logging.basicConfig(
filename="networkmorphism.log",
filemode="a",
level=logging.INFO,
format=log_format,
datefmt="%m/%d %I:%M:%S %p",
)
# set the logger format
logger = logging.getLogger("cifar10-network-morphism-keras")
# restrict gpu usage background
config = tf.ConfigProto()
# pylint: disable=E1101,W0603
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
KTF.set_session(sess)
def get_args():
""" get args from command line
"""
parser = argparse.ArgumentParser("cifar10")
parser.add_argument("--batch_size", type=int, default=128, help="batch size")
parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
parser.add_argument(
"--learning_rate", type=float, default=0.001, help="learning rate"
)
parser.add_argument(
"--weight_decay",
type=float,
default=1e-5,
help="weight decay of the learning rate",
)
return parser.parse_args()
trainloader = None
testloader = None
net = None
args = get_args()
TENSORBOARD_DIR = os.environ["NNI_OUTPUT_DIR"]
def build_graph_from_json(ir_model_json):
"""build model from json representation
"""
graph = json_to_graph(ir_model_json)
logging.debug(graph.operation_history)
model = graph.produce_keras_model()
return model
def parse_rev_args(receive_msg):
""" parse reveive msgs to global variable
"""
global trainloader
global testloader
global net
# Loading Data
logger.debug("Preparing data..")
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)
x_train = x_train.astype("float32")
x_test = x_test.astype("float32")
x_train /= 255.0
x_test /= 255.0
trainloader = (x_train, y_train)
testloader = (x_test, y_test)
# Model
logger.debug("Building model..")
net = build_graph_from_json(receive_msg)
# parallel model
try:
available_devices = os.environ["CUDA_VISIBLE_DEVICES"]
gpus = len(available_devices.split(","))
if gpus > 1:
net = multi_gpu_model(net, gpus)
except KeyError:
logger.debug("parallel model not support in this config settings")
if args.optimizer == "SGD":
optimizer = SGD(lr=args.learning_rate, momentum=0.9, decay=args.weight_decay)
if args.optimizer == "Adadelta":
optimizer = Adadelta(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adagrad":
optimizer = Adagrad(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adam":
optimizer = Adam(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "Adamax":
optimizer = Adamax(lr=args.learning_rate, decay=args.weight_decay)
if args.optimizer == "RMSprop":
optimizer = RMSprop(lr=args.learning_rate, decay=args.weight_decay)
# Compile the model
net.compile(
loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"]
)
return 0
class SendMetrics(keras.callbacks.Callback):
"""
Keras callback to send metrics to NNI framework
"""
def on_epoch_end(self, epoch, logs=None):
"""
Run on end of each epoch
"""
if logs is None:
logs = dict()
logger.debug(logs)
nni.report_intermediate_result(logs["acc"])
# Training
def train_eval():
""" train and eval the model
"""
global trainloader
global testloader
global net
(x_train, y_train) = trainloader
(x_test, y_test) = testloader
# train procedure
net.fit(
x=x_train,
y=y_train,
batch_size=args.batch_size,
validation_data=(x_test, y_test),
epochs=args.epochs,
shuffle=True,
callbacks=[
SendMetrics(),
EarlyStopping(min_delta=0.001, patience=10),
TensorBoard(log_dir=TENSORBOARD_DIR),
],
)
# trial report final acc to tuner
_, acc = net.evaluate(x_test, y_test)
logger.debug("Final result is: %d", acc)
nni.report_final_result(acc)
if __name__ == "__main__":
try:
# trial get next parameter from network morphism tuner
RCV_CONFIG = nni.get_next_parameter()
logger.debug(RCV_CONFIG)
parse_rev_args(RCV_CONFIG)
train_eval()
except Exception as exception:
logger.exception(exception)
raise
examples/trials/network_morphism/cifar10/cifar10_pytorch.py 0 → 100644
View file @ e31839cc
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import logging
import sys
import nni
from nni.networkmorphism_tuner.graph import json_to_graph
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
sys.path.append("../")
from network_morphism import utils
# set the logger format
log_format = "%(asctime)s %(message)s"
logging.basicConfig(
filename="networkmorphism.log",
filemode="a",
level=logging.INFO,
format=log_format,
datefmt="%m/%d %I:%M:%S %p",
)
# pylint: disable=W0603
# set the logger format
logger = logging.getLogger("cifar10-network-morphism-pytorch")
def get_args():
""" get args from command line
"""
parser = argparse.ArgumentParser("cifar10")
parser.add_argument("--batch_size", type=int, default=128, help="batch size")
parser.add_argument("--optimizer", type=str, default="SGD", help="optimizer")
parser.add_argument("--epochs", type=int, default=200, help="epoch limit")
parser.add_argument(
"--learning_rate", type=float, default=0.001, help="learning rate"
)
parser.add_argument("--cutout", action="store_true", default=False, help="use cutout")
parser.add_argument("--cutout_length", type=int, default=8, help="cutout length")
parser.add_argument(
"--model_path", type=str, default="./", help="Path to save the destination model"
)
return parser.parse_args()
trainloader = None
testloader = None
net = None
criterion = None
optimizer = None
device = "cuda" if torch.cuda.is_available() else "cpu"
best_acc = 0.0
args = get_args()
def build_graph_from_json(ir_model_json):
"""build model from json representation
"""
graph = json_to_graph(ir_model_json)
logging.debug(graph.operation_history)
model = graph.produce_torch_model()
return model
def parse_rev_args(receive_msg):
""" parse reveive msgs to global variable
"""
global trainloader
global testloader
global net
global criterion
global optimizer
# Loading Data
logger.debug("Preparing data..")
transform_train, transform_test = utils.data_transforms_cifar10(args)
trainset = torchvision.datasets.CIFAR10(
root="./data", train=True, download=True, transform=transform_train
)
trainloader = torch.utils.data.DataLoader(
trainset, batch_size=args.batch_size, shuffle=True, num_workers=2
)
testset = torchvision.datasets.CIFAR10(
root="./data", train=False, download=True, transform=transform_test
)
testloader = torch.utils.data.DataLoader(
testset, batch_size=args.batch_size, shuffle=False, num_workers=2
)
# Model
logger.debug("Building model..")
net = build_graph_from_json(receive_msg)
net = net.to(device)
criterion = nn.CrossEntropyLoss()
if device == "cuda" and torch.cuda.device_count() > 1:
net = torch.nn.DataParallel(net)
if args.optimizer == "SGD":
optimizer = optim.SGD(
net.parameters(), lr=args.learning_rate, momentum=0.9, weight_decay=5e-4
)
if args.optimizer == "Adadelta":
optimizer = optim.Adadelta(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adagrad":
optimizer = optim.Adagrad(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adam":
optimizer = optim.Adam(net.parameters(), lr=args.learning_rate)
if args.optimizer == "Adamax":
optimizer = optim.Adamax(net.parameters(), lr=args.learning_rate)
if args.optimizer == "RMSprop":
optimizer = optim.RMSprop(net.parameters(), lr=args.learning_rate)
return 0
# Training
def train(epoch):
""" train model on each epoch in trainset
"""
global trainloader
global testloader
global net
global criterion
global optimizer
logger.debug("Epoch: %d", epoch)
net.train()
train_loss = 0
correct = 0
total = 0
for batch_idx, (inputs, targets) in enumerate(trainloader):
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()
train_loss += loss.item()
_, predicted = outputs.max(1)
total += targets.size(0)
correct += predicted.eq(targets).sum().item()
acc = 100.0 * correct / total
logger.debug(
"Loss: %.3f | Acc: %.3f%% (%d/%d)",
train_loss / (batch_idx + 1),
100.0 * correct / total,
correct,
total,
)
return acc
def test(epoch):
""" eval model on each epoch in testset
"""
global best_acc
global trainloader
global testloader
global net
global criterion
global optimizer
logger.debug("Eval on epoch: %d", epoch)
net.eval()
test_loss = 0
correct = 0
total = 0
with torch.no_grad():
for batch_idx, (inputs, targets) in enumerate(testloader):
inputs, targets = inputs.to(device), targets.to(device)
outputs = net(inputs)
loss = criterion(outputs, targets)
test_loss += loss.item()
_, predicted = outputs.max(1)
total += targets.size(0)
correct += predicted.eq(targets).sum().item()
acc = 100.0 * correct / total
logger.debug(
"Loss: %.3f | Acc: %.3f%% (%d/%d)",
test_loss / (batch_idx + 1),
100.0 * correct / total,
correct,
total,
)
acc = 100.0 * correct / total
if acc > best_acc:
best_acc = acc
return acc, best_acc
if __name__ == "__main__":
try:
# trial get next parameter from network morphism tuner
RCV_CONFIG = nni.get_next_parameter()
logger.debug(RCV_CONFIG)
parse_rev_args(RCV_CONFIG)
train_acc = 0.0
best_acc = 0.0
early_stop = utils.EarlyStopping(mode="max")
for ep in range(args.epochs):
train_acc = train(ep)
test_acc, best_acc = test(ep)
nni.report_intermediate_result(test_acc)
logger.debug(test_acc)
if early_stop.step(test_acc):
break
# trial report best_acc to tuner
nni.report_final_result(best_acc)
except Exception as exception:
logger.exception(exception)
raise
examples/trials/network_morphism/cifar10/config.yml 0 → 100644
View file @ e31839cc
authorName: default
experimentName: example_cifar10-network-morphism
trialConcurrency: 4
maxExecDuration: 48h
maxTrialNum: 200
#choice: local, remote, pai
trainingServicePlatform: local
#searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
#SMAC (SMAC should be installed through nnictl)
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
#for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 32
#input image channel
input_channel: 3
#number of classes
n_output_node: 10
trial:
command: python3 cifar10_keras.py
codeDir: .
gpuNum: 1
examples/trials/network_morphism/cifar10/config_pai.yml 0 → 100644
View file @ e31839cc
authorName: default
experimentName: example_cifar10-network-morphism
trialConcurrency: 1
maxExecDuration: 24h
maxTrialNum: 10
#choice: local, remote, pai
trainingServicePlatform: pai
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism
#SMAC (SMAC should be installed through nnictl)
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
# for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 32
#input image channel
input_channel: 3
#number of classes
n_output_node: 10
trial:
command: python3 cifar10_keras.py
codeDir: .
gpuNum: 1
cpuNum: 1
memoryMB: 8196
#The docker image to run nni job on pai
image: msranni/nni:latest
#The hdfs directory to store data on pai, format 'hdfs://host:port/directory'
dataDir: hdfs://10.10.10.10:9000/username/nni
#The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
outputDir: hdfs://10.10.10.10:9000/username/nni
paiConfig:
#The username to login pai
userName: username
#The password to login pai
passWord: password
#The host of restful server of pai
host: 10.10.10.10
\ No newline at end of file
examples/trials/network_morphism/requirements.txt 0 → 100644
View file @ e31839cc
numpy==1.14.2
tensorflow==1.12.0
torchvision==0.2.1
Keras==2.2.2
nni==0.3.0
torch==0.4.1
examples/trials/network_morphism/utils.py 0 → 100644
View file @ e31839cc
"""Some helper functions for PyTorch, including:
- get_mean_and_std: calculate the mean and std value of dataset.
- msr_init: net parameter initialization.
- progress_bar: progress bar mimic xlua.progress.
"""
import numpy as np
import torch
import torch.nn as nn
import torch.nn.init as init
import torchvision.transforms as transforms
class EarlyStopping:
""" EarlyStopping class to keep NN from overfitting
"""
# pylint: disable=E0202
def __init__(self, mode="min", min_delta=0, patience=10, percentage=False):
self.mode = mode
self.min_delta = min_delta
self.patience = patience
self.best = None
self.num_bad_epochs = 0
self.is_better = None
self._init_is_better(mode, min_delta, percentage)
if patience == 0:
self.is_better = lambda a, b: True
self.step = lambda a: False
def step(self, metrics):
""" EarlyStopping step on each epoch
Arguments:
metrics {float} -- metric value
"""
if self.best is None:
self.best = metrics
return False
if np.isnan(metrics):
return True
if self.is_better(metrics, self.best):
self.num_bad_epochs = 0
self.best = metrics
else:
self.num_bad_epochs += 1
if self.num_bad_epochs >= self.patience:
return True
return False
def _init_is_better(self, mode, min_delta, percentage):
if mode not in {"min", "max"}:
raise ValueError("mode " + mode + " is unknown!")
if not percentage:
if mode == "min":
self.is_better = lambda a, best: a < best - min_delta
if mode == "max":
self.is_better = lambda a, best: a > best + min_delta
else:
if mode == "min":
self.is_better = lambda a, best: a < best - (best * min_delta / 100)
if mode == "max":
self.is_better = lambda a, best: a > best + (best * min_delta / 100)
class Cutout:
"""Randomly mask out one or more patches from an image.
Args:
n_holes (int): Number of patches to cut out of each image.
length (int): The length (in pixels) of each square patch.
"""
def __init__(self, length):
self.length = length
def __call__(self, img):
"""
Args:
img (Tensor): Tensor image of size (C, H, W).
Returns:
Tensor: Image with n_holes of dimension length x length cut out of it.
"""
h_img, w_img = img.size(1), img.size(2)
mask = np.ones((h_img, w_img), np.float32)
y_img = np.random.randint(h_img)
x_img = np.random.randint(w_img)
y1_img = np.clip(y_img - self.length // 2, 0, h_img)
y2_img = np.clip(y_img + self.length // 2, 0, h_img)
x1_img = np.clip(x_img - self.length // 2, 0, w_img)
x2_img = np.clip(x_img + self.length // 2, 0, w_img)
mask[y1_img:y2_img, x1_img:x2_img] = 0.0
mask = torch.from_numpy(mask)
mask = mask.expand_as(img)
img *= mask
return img
def data_transforms_cifar10(args):
""" data_transforms for cifar10 dataset
"""
cifar_mean = [0.49139968, 0.48215827, 0.44653124]
cifar_std = [0.24703233, 0.24348505, 0.26158768]
train_transform = transforms.Compose(
[
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(cifar_mean, cifar_std),
]
)
if args.cutout:
train_transform.transforms.append(Cutout(args.cutout_length))
valid_transform = transforms.Compose(
[transforms.ToTensor(), transforms.Normalize(cifar_mean, cifar_std)]
)
return train_transform, valid_transform
def data_transforms_mnist(args, mnist_mean=None, mnist_std=None):
""" data_transforms for mnist dataset
"""
if mnist_mean is None:
mnist_mean = [0.5]
if mnist_std is None:
mnist_std = [0.5]
train_transform = transforms.Compose(
[
transforms.RandomCrop(28, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(mnist_mean, mnist_std),
]
)
if args.cutout:
train_transform.transforms.append(Cutout(args.cutout_length))
valid_transform = transforms.Compose(
[transforms.ToTensor(), transforms.Normalize(mnist_mean, mnist_std)]
)
return train_transform, valid_transform
def get_mean_and_std(dataset):
"""Compute the mean and std value of dataset."""
dataloader = torch.utils.data.DataLoader(
dataset, batch_size=1, shuffle=True, num_workers=2
)
mean = torch.zeros(3)
std = torch.zeros(3)
print("==> Computing mean and std..")
for inputs, _ in dataloader:
for i in range(3):
mean[i] += inputs[:, i, :, :].mean()
std[i] += inputs[:, i, :, :].std()
mean.div_(len(dataset))
std.div_(len(dataset))
return mean, std
def init_params(net):
"""Init layer parameters."""
for module in net.modules():
if isinstance(module, nn.Conv2d):
init.kaiming_normal(module.weight, mode="fan_out")
if module.bias:
init.constant(module.bias, 0)
elif isinstance(module, nn.BatchNorm2d):
init.constant(module.weight, 1)
init.constant(module.bias, 0)
elif isinstance(module, nn.Linear):
init.normal(module.weight, std=1e-3)
if module.bias:
init.constant(module.bias, 0)
pylintrc
View file @ e31839cc
...@@ -16,3 +16,6 @@ const-naming-style=any ...@@ -16,3 +16,6 @@ const-naming-style=any
disable=duplicate-code, disable=duplicate-code,
super-init-not-called super-init-not-called
# List of members which are set dynamically and missed by pylint inference
generated-members=numpy.*,torch.*
src/nni_manager/rest_server/restValidationSchemas.ts
View file @ e31839cc
...@@ -118,7 +118,7 @@ export namespace ValidationSchemas { ...@@ -118,7 +118,7 @@ export namespace ValidationSchemas {
checkpointDir: joi.string() checkpointDir: joi.string()
}), }),
tuner: joi.object({ tuner: joi.object({
builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch'), builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch', 'NetworkMorphism'),
codeDir: joi.string(), codeDir: joi.string(),
classFileName: joi.string(), classFileName: joi.string(),
className: joi.string(), className: joi.string(),
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment