GitLab

SSH client has a max number of open channels for a connection, if we set the number of trialCurrency too big, our ssh client will exec command using ssh frequently, then we will meet the error of Error: (SSH) Channel open failure: open failed.
Refactor the code, set one connection has a max trial concurrency, when the number of trial reach the ssh connection restriction, will create a new ssh connection to exec trial commands.

* fix remote bug

* add document

* add document

* fix remote issue

* fix forEach

* update doc according to comments

* remove 'any more'

* set nniManager.log and dispatcher.log time format to local time

trial_keeper use 50070 port to connect to webhdfs server, and PAI use a mapping method to map 50070 port to 5070 port to visit restful server, this method has some risk for PAI may not support this kind of mapping in later release.Now use Pylon path(/webhdfs/api/v1) instead of 50070 port in webhdfs client of trial_keeper, the path is transmitted in trainingService.
In this pr, we have these changes:

1. Change to use webhdfs path instead of 50070 port in hdfs client.
2. Change to use new hdfs package "PythonWebHDFS", which is build to support pylon by myself. You could test the new function from "sparksnail/nni:dev-pai" image to test pai trainingService.
3. Update some variables' name according to comments.

* add trialkeeper_stdout and trialkeeper_stderr
* fix nnictl set remote nniManagerIP

* Fix a race condition bug that does not store Trial Job cancel status correctly

Unit test for nnimanager

* fix remote bug

* add document

* add document

* update

* update

* update

* update

* fix remote issue

* fix forEach

* update doc according to comments

* update

* update

* update

* remove 'any more'

* add base version for remote-log

* change launcher.py

* test

* basic version

* debug

* debug

* basic work version

* fix code

* update disable_log

* remove unused line

* add diable log in kubernetesTrainingService

* add detect frameworkcontroller

* fix comment

* update

* update

* fix kubernetesData

* debug

* debug

* debug

* fix comment

* fix conflict

* remove local temp files

* revert launcher.py

* update code by comments

* remove disableLog

* remove disable Log

* set timeout for cleanup

* fix code by comments

* update variable names

* add comments

* add delay function

* update

* update

* update by comments

* add  in remote script path

* rename variables

* update variable name

* add mkdir -p for subfolder

* Fix PAI webhdfs api endpoint

* Pull code (#22)

* Support distributed job for frameworkcontroller (#612)

support distributed job for frameworkcontroller

* Multiphase doc (#519)

* multiPhase doc

* updates

* updates

* Add time parser for 'nnictl update duration' (#632)

Current nnictl update duration only support seconds unit, add a parser for this command to support {s, m, h, d}

* fix experiment state bug (#629)

* update top README.md (#622)

* Update README.md

* update (#634)

* Integration tests refactoring (#625)

* Integration test refactoring (#21) (#616)

* Integration test refactoring (#21)

* Refactoring integration tests

* test metrics

* update azure pipeline

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* update trigger

* Integration test refactoring (#618)

* updates

* updates

* update pipeline (#619)

* update pipeline

* updates

* updates

* updates

* updates

* updates

* test pipeline (#623)

* test pipeline

* updates

* updates

* updates

* Update integration test (#624)

* Update integration test

* updates

* updates

* updates

* updates

* updates

* updates

* Revert "Pull code (#22)"

This reverts commit 62fc165ad7b2ba724eead3b99f010aa34491e2c7.

* Update nnimanager logs

* updates

* Update README.md

* Revert "Update README.md"

This reverts commit bc67061160e5d57305a6e7fb63d491d12d0e9002.

* updates

* updates

* Pull code (#22)

* Support distributed job for frameworkcontroller (#612)

support distributed job for frameworkcontroller

* Multiphase doc (#519)

* multiPhase doc

* updates

* updates

* Add time parser for 'nnictl update duration' (#632)

Current nnictl update duration only support seconds unit, add a parser for this command to support {s, m, h, d}

* fix experiment state bug (#629)

* update top README.md (#622)

* Update README.md

* update (#634)

* Integration tests refactoring (#625)

* Integration test refactoring (#21) (#616)

* Integration test refactoring (#21)

* Refactoring integration tests

* test metrics

* update azure pipeline

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* updates

* update trigger

* Integration test refactoring (#618)

* updates

* updates

* update pipeline (#619)

* update pipeline

* updates

* updates

* updates

* updates

* updates

* test pipeline (#623)

* test pipeline

* updates

* updates

* updates

* Update integration test (#624)

* Update integration test

* updates

* updates

* updates

* updates

* updates

* updates

* Revert "Pull code (#22)"

This reverts commit 62fc165ad7b2ba724eead3b99f010aa34491e2c7.

* Configurable nniManager log path

* Configure log level

* add --debug command line for nnictl

* updates

* add serviceAccountName

* add serviceAccountName for frameworkcontroller

support distributed job for frameworkcontroller

* update readme in ga_squad

* update readme

* fix typo

* Update README.md

* Update README.md

* Update README.md

* update readme

* update

* fix path

* update reference

* fix bug in config file

* update nni_arch_overview.png

* update

* update

* update

* add metis tuner code

* 1. fix bug about import 2.update other sdk file

* add auto-gbdt-example and remove unused code

* add metis_tuner into README

* update the README

* update README | remove unused variable

* fix typo

* add sklearn into requirments

* Update src/sdk/pynni/nni/metis_tuner/metis_tuner.py

add default value in __init__
Co-Authored-By: xuehui1991 <xuehui@microsoft.com>

* Update docs/HowToChooseTuner.md
Co-Authored-By: xuehui1991 <xuehui@microsoft.com>

* Update docs/HowToChooseTuner.md
Co-Authored-By: xuehui1991 <xuehui@microsoft.com>

* fix typo | add more comments

trial job could not be stopped in remote machine when experiment is stopped, because awit/async does not work normally in forEach, refer https://codeburst.io/javascript-async-await-with-foreach-b6ba62bbf404.

* Add UT code coverage report

* updates

* updates

* updates

* updates

* updates

* updates

* integration test python code coverage report

* Removed unused log code, refactor to rename some class name in nni sdk and trial_tools

* Fix the regression bug that loca/remote mode doesnt work

* fix bug

* add docs

* add ut

* add ut

* add to ci

* update doc

* update doc

* update ut

* add ut to ci

* add ut to ci

* add ut to ci

* add ut to ci

* add ut to ci

* add ut to ci

* add ut to ci

* add ut to ci

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* revert

* refactor

* refactor

* s

* merge

Remote TrainingService, passwd is not required when users set sshKeyPath.

Add frameworkcontroller training service based on kubeflow training service.
Refactor code structure, add kubernetes training service as father class, and set kubeflow training service and frameworkcontroller training service as child class.

* Fix IPC unit test

* updates

* Dockerfile improvement

* Fix UT broken for kubeflow training service

* Update nnictl.py

Fix the issue that nnictl --version via pip installation doesn't work

* Update kubeflow training service document (#494)

* Remove kubectl related document, add messages for kubeconfig
* Add design section for kubeflow training service
* Move the image files for PAI training service doc into img folder.

* Update KubeflowMode.md (#498)

Update KubeflowMode.md, small terms change

* [V0.4.1 bug fix] Cannot run kubeflow training service due to trial_keeper change (#503)

* Update kubeflow training service document

* fix bug a that kubeflow trial job cannot run

* upgrade version number (#499)

* [V0.4.1 bug fix] Support read K8S config from KUBECONFIG environment variable (#507)

* Add KUBCONFIG env variable support

* In main.ts, throw cached error to make sure nnictl can show the error in stderr

* Add curve fitting assessor

* Update HowToChooseTuner.md

* Update HowToChooseTuner.md

* Update HowToChooseTuner.md

* Update README.md

* Update README.md

* Update README.md

* Update HowToChooseTuner.md

* Update HowToChooseTuner.md

* Update HowToChooseTuner.md

* Update HowToChooseTuner.md

* Update curvefitting_assessor.py

* Update config_schema.py

* Add some comments and modifications

* Remove unnecessary .json file

* Remove unnecessary .lock file

* Revert "Remove unnecessary .lock file"

This reverts commit cdfaacb29114b3dee9c797d3e9b46ee18d7d34cc.

* Revert "Revert "Remove unnecessary .lock file""

This reverts commit 7182a5fb31a02b01684429eabb3347952bf7ce2a.

* Revert "Revert "Revert "Remove unnecessary .lock file"""

This reverts commit 0f010e2b508e9f7b34c809647ba09e4e132876d8.

* Revert "Remove unnecessary .json file"

This reverts commit c6f7b47c199dd0db7ccb850d4f2ac1fd97b0caf8.

* Revert "Add some comments and modifications"

This reverts commit f78f055df9a4eec5b433a9241ce93d8ba78e3500.

* Add some modifications by comments

* suppoort minimize mode

* Update README.md

* Update README.md

* Update modelfactory.py

* minor changes and fix typo

* minor chages

* update README.md

* Small refactor: remove useless INFO log, and pring valid PAI token error message

* [PAI training service] codeDir files upload improvement

* Create full local temp folder

* Organize the folder structure for experiment and trial files

* Support download log files

* updates

* modify status experiment_running to running

* fix hyperband doc

* fix hyperband doc

* fix hyperband doc

rest api of kubernetes does not use base64 to encode chars, now use base64 to encode username and then create secret.

[Kubeflow training service] Use Kubernete API server to replace kubectl dependency

* Quick fix nnictl config logic (#289)

* fix nnictl bug

* fix install.sh

* add desc for Dockerfile.build.base

* update document for Dockerfile

* update

* refactor port detect

* update

* refactor NNICTLDOC.md

* add document for pai and nnictl

* add default value for port

* add exception handling in trial_keeper.py

* fix port bug

* fix resume

* fix nnictl resume and fix nnictl stop

* fix document

* update

* refactor nnictl

* update

* update doc

* update

* update nnictl

* fix comment

* revert dockerfile

* update

* update

* update

* fix nnictl error hit

* fix comments

* fix bash-completion

* fix paramiko install

* quick fix resume logic

* update

* quick fix nnictl

* PR merge to 0.3 (#297)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* update doc

* fix links

* fix links

* fix links

* merge

* fix links and doc errors

* merge

* merge

* merge

* merge

* Update README.md (#288)

added License badge

* merge

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* fix doc mistakes and broken links. (#271)

* refactor doc

* update with Mao's suggestions

* Set theme jekyll-theme-dinky

* updated the "Contribute" part (merged Gems' wiki in, updated ReadMe)

* fix link

* Update README.md

* Fix misspelling in examples/trials/ga_squad/README.md

* revise the installation cmd to v0.2

* revise to install v0.2

* remove enas readme (#292)

* Fix datastore performance issue (#301)

* Fix nnictl in v0.3 (#299)

Fix old version of config file
fix sklearn requirements
Fix resume log logic

* add basic tuner and trial for network morphism

* Complete basic receive_trial_result() and generate_parameters(). Use onnx  as the intermediate representation ( But it cannot convert to pytorch model )

* add tensorflow cifar10 for network morphism

* add unit test for tuner and its function

* use temporary torch_model

* fix request bug and program can communicate nni

* add basic pickle support for graph and train successful in pytorch

* Update unittest for networkmorphism_tuner

* Network Morphism add multi-gpu trial training support

* Format code with black tool

* change intermediate representation from pickle file to json we defined

* successfully pass the unittest for test_graph_json_transform

* add README for network morphism and it works fine in both Pytorch and Keras.

* separate the original Readme.md in network-morphism into two parts (tuner and trial)

* change the openpai image path

* beautify the file structure of network_morphism and add a fashion_mnist keras example

* pretty the source and add some docstring for funtion in order to pass the pylint.

* remove unused module import and add some docstring

* add some details for the application scenario Network Morphism Tuner

* follow the advice and modify the doc file

* add the config file for each task in the examples trial of network morphism

* change default python interpreter from python to python3