TrainingService is a module related to platform management and job schedule in NNI. TrainingService is designed to be easily implemented, we define an abstract class TrainingService as the parent class of all kinds of TrainignService, users just need to inherit the parent class and complete their own clild class if they want to implement customized TrainingService.
The brief system architecture of NNI is shown in the picture. NNIManager is the core management module of system, in charge of calling TrainingService to manage trial jobs and the communication between different modules. Dispatcher is a message processing center responsible for message dispatch. TrainingService is a module to manage trial jobs, it communicates with nniManager module, and has different instance according to different training platform. For the time being, NNI supports local platfrom, [remote platfrom](RemoteMachineMode.md), [PAI platfrom](PAIMode.md), [kubeflow platform](KubeflowMode.md) and [FrameworkController platfrom](FrameworkController.md).
In this document, we introduce the brief design of TrainingService. If users want to add a new TrainingService instance, they just need to complete a child class to implement TrainingService, don't need to understand the code detail of NNIManager, Dispatcher or other modules.
@@ -44,9 +44,9 @@ NNI's folder structure is shown below:
...
@@ -44,9 +44,9 @@ NNI's folder structure is shown below:
| |-nni_trial_tool
| |-nni_trial_tool
`nni/src/`folder stores the most source code of NNI. The code in this folder is related to NNIManager, TrainingService, SDK, WebUI and other modules. Users could find the abstract class of TrainingService in `nni/src/nni_manager/common/trainingService.ts` file, and they should put their own implemented TrainingService in `nni/src/nni_manager/training_service` folder. If users have implemented their own TrainingService code, they should also supplement the unit test of the code, and place them in`nni/src/nni_manager/training_service/test`folder.
public abstract listTrialJobs(): Promise<TrialJobDetail[]>;
public abstract listTrialJobs(): Promise<TrialJobDetail[]>;
...
@@ -64,10 +64,10 @@ NNI's folder structure is shown below:
...
@@ -64,10 +64,10 @@ NNI's folder structure is shown below:
}
}
The parent class of TrainingService has a few abstract functions, users need to inherit the parent class and implement all of these abstract functions.
@@ -93,13 +93,13 @@ ClusterMetadata is the data related to platform details, for examples, the Clust
...
@@ -93,13 +93,13 @@ ClusterMetadata is the data related to platform details, for examples, the Clust
}
}
The metadata includes the host address, the username or other configuration related to the platform. Users need to define their own metadata format, and set the metadata instance in this function. This function is called before the experiment is started to set the configuration of remote machines.
This function will return the metadata value according to the values, it could be left empty if users don't need to use it.
这个方法返回 metadata 的内容，如果不需要使用这个方法，可将方法内容留空。
**submitTrialJob(form: JobApplicationForm)**
**submitTrialJob(form: JobApplicationForm)**
SubmitTrialJob is a function to submit new trial jobs, users should generate a job instance in TrialJobDetail type. TrialJobDetail is defined as follow:
@@ -116,39 +116,39 @@ SubmitTrialJob is a function to submit new trial jobs, users should generate a j
...
@@ -116,39 +116,39 @@ SubmitTrialJob is a function to submit new trial jobs, users should generate a j
}
}
According to different kinds of implementation, users could put the job detail into a job queue, and keep fetching the job from the queue and start preparing and running them. Or they could finish preparing and running process in this function, and return job detail after the submit work.
If this function is called, the trial started by the platform should be canceled. Different kind of platform has diffenent methods to calcel a running job, this function should be implemented according to specific platform.
This function is called to update the trial job's status, trial job's status should be detected according to different platform, and be updated to `RUNNING`, `SUCCEED`, `FAILED`etc.
NNI will hold an EventEmitter to get job metrics, if there is new job metrics detected, the EventEmitter will be triggered. Users should start the EventEmitter in this function.
NNI 会启动一个 EventEmitter 来处理作业的指标数据，如果有检测到有新的数据，EventEmitter就会被触发，来执行相应的事件。 用户需要在这个方法中开始 EventEmitter。
The run() function is a main loop function in TrainingService, users could set a while loop to execute their logic code, and finish executing them when the experiment is stopped.
This function is called to clean up the environment when a experiment is stopped. Users should do the platform-related cleaning operation in this function.
当实验结束后，这个方法用来清除实验环境。 用户需要在这个方法中实现与平台相关的清除操作。
## TrialKeeper tool
## TrialKeeper 工具
NNI offers a TrialKeeper tool to help maintaining trial jobs. Users can find the source code in `nni/tools/nni_trial_tool`. If users want to run trial jobs in cloud platform, this tool will be a fine choice to help keeping trial running in the platform. The running architecture of TrialKeeper is show as follow:
When users submit a trial job to cloud platform, they should wrap their trial command into TrialKeeper, and start a TrialKeeper process in cloud platform. Notice that TrialKeeper use restful server to communicate with TrainingService, users should start a restful server in local machine to receive metrics sent from TrialKeeper. The source code about restful server could be found in`nni/src/nni_manager/training_service/common/clusterJobRestServer.ts`.
**准备 Tuner**: NNI 支持多种流行的自动机器学习算法，包括：Random Search（随机搜索），Tree of Parzen Estimators (TPE)，Evolution（进化算法）等等。 也可以实现自己的 Tuner（参考[这里](Customize_Tuner.md)）。下面使用了 NNI 内置的 Tuner：
