GitLab

[WebUI] Default metric is latest or final result, !local, show trial log hyperlink or address (#913)

* Add intermediate column

* update

* show trial log link or address

* fix some comments

* update

* Optimize job query performance

* Fix hyperband command queues

1. Route tuner and assessor commands to 2 seperate queues  issue #841
2. Allow tuner to leverage intermediate result when trial is early stopped.  issue #843

If user set remoteloggingType in config file, log content will not be transmitted from trialkeeper

* [WebUI] Add intermediate result and optimize hyper-parameter graph

* update

* fix comment

Wrong spell in INFO log.

check nni version in trialkeeper, to make sure the version of trialkeeper is consistent with trainingService
add a debug mode in config file

Support intermediate is dict

SSH client has a max number of open channels for a connection, if we set the number of trialCurrency too big, our ssh client will exec command using ssh frequently, then we will meet the error of Error: (SSH) Channel open failure: open failed.
Refactor the code, set one connection has a max trial concurrency, when the number of trial reach the ssh connection restriction, will create a new ssh connection to exec trial commands.

* fix remote bug

* add document

* add document

* fix remote issue

* fix forEach

* update doc according to comments

* remove 'any more'

* set nniManager.log and dispatcher.log time format to local time

* add epsilon

* add epsilon for ceil

* fix annotation, resolve annotation's key err bug, refactor the design

* add different tuner config files for config_test

* change MetisTuner config test due to no lightgbm python module in integration test

* install smac package in azure-pipelines

* SMAC need swig to be installed

* Try to install swig from source code

* remove SMAC test because the dependency can not be installed

* use sudo to install the swig

* sleep 10s to make sure the port has been released

* remove tuner test for networkmorphism because it uses more than 30s to release the tcp port

* word "down" to "done"

* add config test for Curvefitting assessor

* change file name

* Fix data type not match bug

* Optimize MetisTunner

* pretty the code

* Follow the review comment

* add exploration probability

* Avoid None type object generating

* fix bug in cold start

* remove unused comment

Fix bug in Curvefitting Assessor

* add different tuner config files for config_test

* change MetisTuner config test due to no lightgbm python module in integration test

* install smac package in azure-pipelines

* SMAC need swig to be installed

* Try to install swig from source code

* remove SMAC test because the dependency can not be installed

* use sudo to install the swig

* sleep 10s to make sure the port has been released

* remove tuner test for networkmorphism because it uses more than 30s to release the tcp port

* word "down" to "done"

* add config test for Curvefitting assessor

* change file name

* Fix data type not match bug

* Optimize MetisTunner

* pretty the code

* Follow the review comment

* add exploration probability

* update readme in ga_squad

* update readme

* fix typo

* Update README.md

* Update README.md

* Update README.md

* update readme

* update

* fix path

* update reference

* fix bug in config file

* update nni_arch_overview.png

* update

* update

* update

* update home page

* quickly fix bug in cold start

fix broken link for curfitting assessor doc

In trialkeeper, the output stream of trial process may be in cache, and multiphase need the output stream returns immediately. Flush when sdk print content.

Improve the performance of the curve fitting assessor 

* change version

trial_keeper use 50070 port to connect to webhdfs server, and PAI use a mapping method to map 50070 port to 5070 port to visit restful server, this method has some risk for PAI may not support this kind of mapping in later release.Now use Pylon path(/webhdfs/api/v1) instead of 50070 port in webhdfs client of trial_keeper, the path is transmitted in trainingService.
In this pr, we have these changes:

1. Change to use webhdfs path instead of 50070 port in hdfs client.
2. Change to use new hdfs package "PythonWebHDFS", which is build to support pylon by myself. You could test the new function from "sparksnail/nni:dev-pai" image to test pai trainingService.
3. Update some variables' name according to comments.

* move tuners readme under docs
* fix deadlink in builtin
* fix reference warning
* remove smac dependency

* add trialkeeper_stdout and trialkeeper_stderr
* fix nnictl set remote nniManagerIP

* Fix a race condition bug that does not store Trial Job cancel status correctly

Unit test for nnimanager

change Relative path to absolute path, unified docstring (include Hyperband), sdk_reference.rst

* fix some bugs in doc and log

* The learning rate focus more on validation sets accuracy than training sets accuracy.

* update Dockerfile and README

* Update README.md

Merge to branch v0.5

* random search when suggested parameter occurs in the history

* avoid infinite loop when seacrh space is limited

* follow the review comments

* add comments

* update readme in ga_squad

* update readme

* fix typo

* Update README.md

* Update README.md

* Update README.md

* update readme

* update

* fix path

* update reference

* fix bug in config file

* update nni_arch_overview.png

* update

* update

* update

* update home page

* update for reference