RELEASE.md 13 KB
Newer Older
Yan Ni's avatar
Yan Ni committed
1
2
# ChangeLog

chicm-ms's avatar
chicm-ms committed
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
## Release 0.7 - 4/29/2018

### Major Features
* [Support NNI on Windows](./WindowsLocalMode.md)
    * NNI running on windows for local mode
* [New advisor: BOHB](./bohbAdvisor.md)
    * Support a new advisor BOHB, which is a robust and efficient hyperparameter tuning algorithm, combines the advantages of Bayesian optimization and Hyperband
* [Support import and export experiment data through nnictl](./NNICTLDOC.md#experiment)
    * Generate analysis results report after the experiment execution
    * Support import data to tuner and advisor for tuning
* [Designated gpu devices for NNI trial jobs](./ExperimentConfig.md#localConfig)
    * Specify GPU devices for NNI trial jobs by gpuIndices configuration, if gpuIndices is set in experiment configuration file, only the specified GPU devices are used for NNI trial jobs.
* Web Portal enhancement
    * Decimal format of metrics other than default on the Web UI
    * Hints in WebUI about Multi-phase
    * Enable copy/paste for hyperparameters as python dict
    * Enable early stopped trials data for tuners.
* NNICTL provide better error message
    * nnictl provide more meaningful error message for yaml file format error

### Bug fix
* Unable to kill all python threads after nnictl stop in async dispatcher mode
* nnictl --version does not work with make dev-instal
* All trail jobs status stays on 'waiting' for long time on PAI platform

## Release 0.6 - 4/2/2019
### Major Features
* [Version checking](https://github.com/Microsoft/nni/blob/master/docs/en_US/PAIMode.md#version-check)
	* check whether the version is consistent between nniManager and trialKeeper
* [Report final metrics for early stop job](https://github.com/Microsoft/nni/issues/776)
	* If includeIntermediateResults is true, the last intermediate result of the trial that is early stopped by assessor is sent to tuner as final result. The default value of includeIntermediateResults is false.
* [Separate Tuner/Assessor](https://github.com/Microsoft/nni/issues/841)
	* Adds two pipes to separate message receiving channels for tuner and assessor.
* Make log collection feature configurable
* Add intermediate result graph for all trials

### Bug fix
* [Add shmMB config key for PAI](https://github.com/Microsoft/nni/issues/842)
* Fix the bug that doesn't show any result if metrics is dict
* Fix the number calculation issue for float types in hyperband
* Fix a bug in the search space conversion in SMAC tuner
* Fix the WebUI issue when parsing experiment.json with illegal format
* Fix cold start issue in Metis Tuner

chicm-ms's avatar
chicm-ms committed
47
48
49
50
51
52
53
54
55
56
57
58
59
60
## Release 0.5.2 - 3/4/2019
### Improvements
* Curve fitting assessor performance improvement.

### Documentation
* Chinese version document: https://nni.readthedocs.io/zh/latest/
* Debuggability/serviceability document: https://nni.readthedocs.io/en/latest/HowToDebug.html
* Tuner assessor reference: https://nni.readthedocs.io/en/latest/sdk_reference.html#tuner

### Bug Fixes and Other Changes
* Fix a race condition bug that does not store trial job cancel status correctly.
* Fix search space parsing error when using SMAC tuner.
* Fix cifar10 example broken pipe issue.
* Add unit test cases for nnimanager and local training service.
61
62
* Add integration test azure pipelines for remote machine, OpenPAI and kubeflow training services.
* Support Pylon in OpenPAI webhdfs client.
chicm-ms's avatar
chicm-ms committed
63
64


65
66
## Release 0.5.1 - 1/31/2018
### Improvements
Yan Ni's avatar
Yan Ni committed
67
68
* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md) configurable
* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md), making it easier for debugging 
69
70
71
72
73
74

### Documentation
* Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/

### Bug Fixes and Other Changes
* Fix the bug of installation in python virtualenv, and refactor the installation logic
75
* Fix the bug of HDFS access failure on OpenPAI mode after OpenPAI is upgraded. 
76
77
78
* Fix the bug that sometimes in-place flushed stdout makes experiment crash


Yan Ni's avatar
Yan Ni committed
79
## Release 0.5.0 - 01/14/2019
80

Yan Ni's avatar
Yan Ni committed
81
### Major Features
82

Yan Ni's avatar
Yan Ni committed
83
#### New tuner and assessor supports
Yan Ni's avatar
Yan Ni committed
84

Yan Ni's avatar
Yan Ni committed
85
* Support [Metis tuner](metisTuner.md) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
86
* Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
Yan Ni's avatar
Yan Ni committed
87
* Support [Curve fitting assessor](curvefittingAssessor.md) for early stop policy using learning curve extrapolation.
88
* Advanced Support of [Weight Sharing](./AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS.
xuehui's avatar
xuehui committed
89

Yan Ni's avatar
Yan Ni committed
90
#### Training Service Enhancement
91

xuehui's avatar
xuehui committed
92
* [FrameworkController Training service](./FrameworkControllerMode.md): Support run experiments using frameworkcontroller on kubernetes
93
94
95
  * FrameworkController is a Controller on kubernetes that is general enough to run (distributed) jobs with various machine learning frameworks, such as tensorflow, pytorch, MXNet.
  * NNI provides unified and simple specification for job definition.
  * MNIST example for how to use FrameworkController.
xuehui's avatar
xuehui committed
96

Yan Ni's avatar
Yan Ni committed
97
#### User Experience improvements
98

99
* A better trial logging support for NNI experiments in OpenPAI, Kubeflow and FrameworkController mode:
100
101
102
  * An improved logging architecture to send stdout/stderr of trials to NNI manager via Http post. NNI manager will store trial's stdout/stderr messages in local log file.
  * Show the link for trial log file on WebUI.
* Support to show final result's all key-value pairs.
xuehui's avatar
xuehui committed
103

Yan Ni's avatar
Yan Ni committed
104
## Release 0.4.1 - 12/14/2018
105

Yan Ni's avatar
Yan Ni committed
106
### Major Features
107

Yan Ni's avatar
Yan Ni committed
108
#### New tuner supports
109

Yan Ni's avatar
Yan Ni committed
110
* Support [network morphism](networkmorphismTuner.md) as a new tuner
xuehui's avatar
xuehui committed
111

Yan Ni's avatar
Yan Ni committed
112
#### Training Service improvements
113

114
* Migrate [Kubeflow training service](KubeflowMode.md)'s dependency from kubectl CLI to [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) client
115
116
117
* [Pytorch-operator](https://github.com/kubeflow/pytorch-operator) support for Kubeflow training service
* Improvement on local code files uploading to OpenPAI HDFS
* Fixed OpenPAI integration WebUI bug: WebUI doesn't show latest trial job status, which is caused by OpenPAI token expiration
xuehui's avatar
xuehui committed
118

Yan Ni's avatar
Yan Ni committed
119
#### NNICTL improvements
120
121

* Show version information both in nnictl and WebUI. You can run **nnictl -v** to show your current installed NNI version
xuehui's avatar
xuehui committed
122

Yan Ni's avatar
Yan Ni committed
123
#### WebUI improvements
124
125
126
127
128
129
130

* Enable modify concurrency number during experiment
* Add feedback link to NNI github 'create issue' page
* Enable customize top 10 trials regarding to metric numbers (largest or smallest)
* Enable download logs for dispatcher & nnimanager
* Enable automatic scaling of axes for metric number
* Update annotation to support displaying real choice in searchspace
xuehui's avatar
xuehui committed
131

Yan Ni's avatar
Yan Ni committed
132
### New examples
133
134
135
136

* [FashionMnist](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism), work together with network morphism tuner
* [Distributed MNIST example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed-pytorch) written in PyTorch

Yan Ni's avatar
Yan Ni committed
137
## Release 0.4 - 12/6/2018
138

Yan Ni's avatar
Yan Ni committed
139
### Major Features
140
141
142

* [Kubeflow Training service](./KubeflowMode.md)
  * Support tf-operator
Yan Ni's avatar
Yan Ni committed
143
  * [Distributed trial example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow
Yan Ni's avatar
Yan Ni committed
144
145
* [Grid search tuner](gridsearchTuner.md) 
* [Hyperband tuner](hyperbandAdvisor.md)
146
147
148
149
150
151
152
153
154
155
* Support launch NNI experiment on MAC
* WebUI
  * UI support for hyperband tuner
  * Remove tensorboard button
  * Show experiment error message
  * Show line numbers in search space and trial profile
  * Support search a specific trial by trial number
  * Show trial's hdfsLogPath
  * Download experiment parameters

Yan Ni's avatar
Yan Ni committed
156
### Others
157
158
159
160
161

* Asynchronous dispatcher
* Docker file update, add pytorch library 
* Refactor 'nnictl stop' process, send SIGTERM to nni manager process, rather than calling stop Rest API. 
* OpenPAI training service bug fix
162
  * Support NNI Manager IP configuration(nniManagerIp) in OpenPAI cluster config file, to fix the issue that user’s machine has no eth0 device 
163
  * File number in codeDir is capped to 1000 now, to avoid user mistakenly fill root dir for codeDir
164
  * Don’t print useless ‘metrics is empty’ log in OpenPAI job’s stdout. Only print useful message once new metrics are recorded, to reduce confusion when user checks OpenPAI trial’s output for debugging purpose
165
  * Add timestamp at the beginning of each log entry in trial keeper.
166

Yan Ni's avatar
Yan Ni committed
167
## Release 0.3.0 - 11/2/2018
168

Yan Ni's avatar
Yan Ni committed
169
### NNICTL new features and updates
170

171
172
173
174
175
176
177
* Support running multiple experiments simultaneously.

  Before v0.3, NNI only supports running single experiment once a time. After this realse, users are able to run multiple experiments simultaneously. Each experiment will require a unique port, the 1st experiment will be set to the default port as previous versions. You can specify a unique port for the rest experiments as below:

  ```bash
  nnictl create --port 8081 --config <config file path>
  ```
chicm-ms's avatar
chicm-ms committed
178

179
* Support updating max trial number.
180
  use `nnictl update --help` to learn more. Or refer to [NNICTL Spec](NNICTLDOC.md) for the fully usage of NNICTL.
chicm-ms's avatar
chicm-ms committed
181

Yan Ni's avatar
Yan Ni committed
182
### API new features and updates
183

184
* <span style="color:red">**breaking change**</span>: nn.get_parameters() is refactored to nni.get_next_parameter. All examples of prior releases can not run on v0.3, please clone nni repo to get new examples. If you had applied NNI to your own codes, please update the API accordingly.
chicm-ms's avatar
chicm-ms committed
185

186
* New API **nni.get_sequence_id()**. 
187
188
189
190
191
192
193
  Each trial job is allocated a unique sequence number, which can be retrieved by nni.get_sequence_id() API.

  ```bash
  git clone -b v0.3 https://github.com/Microsoft/nni.git
  ```

* **nni.report_final_result(result)** API supports more data types for result parameter.
194

195
196
197
198
  It can be of following types:
  * int
  * float
  * A python dict containing 'default' key, the value of 'default' key should be of type int or float. The dict can contain any other key value pairs.
chicm-ms's avatar
chicm-ms committed
199

Yan Ni's avatar
Yan Ni committed
200
### New tuner support
201

202
* **Batch Tuner** which iterates all parameter combination, can be used to submit batch trial jobs.
chicm-ms's avatar
chicm-ms committed
203

Yan Ni's avatar
Yan Ni committed
204
### New examples
205

206
* A NNI Docker image for public usage:
207
208
209
210
211

  ```bash
  docker pull msranni/nni:latest
  ```

212
213
* New trial example: [NNI Sklearn Example](https://github.com/Microsoft/nni/tree/master/examples/trials/sklearn)
* New competition example: [Kaggle Competition TGS Salt Example](https://github.com/Microsoft/nni/tree/master/examples/trials/kaggle-tgs-salt)
214

Yan Ni's avatar
Yan Ni committed
215
### Others
216

217
218
219
* UI refactoring, refer to [WebUI doc](WebUI.md) for how to work with the new UI.
* Continuous Integration: NNI had switched to Azure pipelines
* [Known Issues in release 0.3.0](https://github.com/Microsoft/nni/labels/nni030knownissues).
chicm-ms's avatar
chicm-ms committed
220

Yan Ni's avatar
Yan Ni committed
221
## Release 0.2.0 - 9/29/2018
222

Yan Ni's avatar
Yan Ni committed
223
### Major Features
224

225
* Support [OpenPAI](https://github.com/Microsoft/pai) Training Platform (See [here](./PAIMode.md) for instructions about how to submit NNI job in pai mode)
226
227
  * Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster
  * NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking
Yan Ni's avatar
Yan Ni committed
228
* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](smacTuner.md) for instructions about how to use SMAC tuner)
229
230
231
232
233
  * [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3)
* Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment
* Others
  * Update ga squad example and related documentation
  * WebUI UX small enhancement and bug fix
fishyds's avatar
fishyds committed
234

Yan Ni's avatar
Yan Ni committed
235
### Known Issues
236

fishyds's avatar
fishyds committed
237
238
[Known Issues in release 0.2.0](https://github.com/Microsoft/nni/labels/nni020knownissues).

Yan Ni's avatar
Yan Ni committed
239
## Release 0.1.0 - 9/10/2018 (initial release)
240
241
242

Initial release of Neural Network Intelligence (NNI).

Yan Ni's avatar
Yan Ni committed
243
### Major Features
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259

* Installation and Deployment
  * Support pip install and source codes install
  * Support training services on local mode(including Multi-GPU mode) as well as multi-machines mode
* Tuners, Assessors and Trial
  * Support AutoML algorithms including:  hyperopt_tpe, hyperopt_annealing, hyperopt_random, and evolution_tuner
  * Support assessor(early stop) algorithms including: medianstop algorithm
  * Provide Python API for user defined tuners and assessors
  * Provide Python API for user to wrap trial code as NNI deployable codes
* Experiments
  * Provide a command line toolkit 'nnictl' for experiments management
  * Provide a WebUI for viewing experiments details and managing experiments
* Continuous Integration
  * Support CI by providing out-of-box integration with [travis-ci](https://github.com/travis-ci) on ubuntu
* Others
  * Support simple GPU job scheduling
260

Yan Ni's avatar
Yan Ni committed
261
### Known Issues
262

Scarlett Li's avatar
Scarlett Li committed
263
[Known Issues in release 0.1.0](https://github.com/Microsoft/nni/labels/nni010knownissues).