Skip to content

GitLab

Unverified Commit 88ef6c04 authored by SparkSnail's avatar SparkSnail Committed by GitHub
Browse files

Merge pull request #197 from microsoft/master 

merge master
parents 5f3c5ffd 555334de
Hide whitespace changes
Inline Side-by-side
Showing with 951 additions and 61 deletions
examples/notebooks/retrieve_nni_info_with_python.ipynb 0 → 100644
View file @ 88ef6c04
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python wrapper for nni restful APIs\n",
"\n",
"nni provides nnicli module as a python wrapper for its restful APIs, which can be used to retrieve nni experiment and trial job information in your python code. This notebook shows how to use nnicli module.\n",
"\n",
"Following are the functions available in nnicli module:\n",
"\n",
"#### start_nni(config_file)\n",
"Starts nni experiment with specified configuration file\n",
"\n",
"#### stop_nni()\n",
"Stop nni experiment.\n",
"\n",
"#### set_endpoint(endpoint)\n",
"Set nni endpoint for nnicli, the endpoint is showed while nni experiment is started successfully using nnictl command or start_nni function\n",
"\n",
"#### version()\n",
"Returns nni version\n",
"\n",
"#### get_experiment_profile()\n",
"Returns experiment profile.\n",
"\n",
"#### get_experiment_status()\n",
"Returns nni experiment status.\n",
"\n",
"#### get_job_metrics(trial_job_id)\n",
"Returns specified trial job metrics, including final results and intermediate results.\n",
"\n",
"#### get_job_statistics()\n",
"Returns trial job statistics information\n",
"\n",
"#### get_trial_job(trial_job_id)\n",
"Returns information of a specified trial job.\n",
"\n",
"#### list_trial_jobs()\n",
"Returns information of all trial jobs of current experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Start nni experiment using specified configuration file\n",
"Let's use a configruation file in nni examples directory to start an experiment."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"authorName: default\r\n",
"experimentName: example_mnist\r\n",
"trialConcurrency: 1\r\n",
"maxExecDuration: 1h\r\n",
"maxTrialNum: 10\r\n",
"#choice: local, remote, pai\r\n",
"trainingServicePlatform: local\r\n",
"searchSpacePath: search_space.json\r\n",
"#choice: true, false\r\n",
"useAnnotation: false\r\n",
"tuner:\r\n",
" #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner\r\n",
" #SMAC (SMAC should be installed through nnictl)\r\n",
" builtinTunerName: TPE\r\n",
" classArgs:\r\n",
" #choice: maximize, minimize\r\n",
" optimize_mode: maximize\r\n",
"trial:\r\n",
" command: python3 mnist.py\r\n",
" codeDir: .\r\n",
" gpuNum: 0\r\n"
]
}
],
"source": [
"! cat ../trials/mnist/config.yml"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: expand searchSpacePath: search_space.json to /mnt/d/Repos/nni/examples/trials/mnist/search_space.json\n",
"INFO: expand codeDir: . to /mnt/d/Repos/nni/examples/trials/mnist/.\n",
"INFO: Starting restful server...\n",
"INFO: Successfully started Restful server!\n",
"INFO: Setting local config...\n",
"INFO: Successfully set local config!\n",
"INFO: Starting experiment...\n",
"INFO: Successfully started experiment!\n",
"-----------------------------------------------------------------------\n",
"The experiment id is PlUIfDTR\n",
"The Web UI urls are: http://172.18.17.1:8080 http://10.172.121.40:8080 http://10.0.75.1:8080 http://127.0.0.1:8080\n",
"-----------------------------------------------------------------------\n",
"\n",
"You can use these commands to get more information about the experiment\n",
"-----------------------------------------------------------------------\n",
"commands description\n",
"1. nnictl experiment show show the information of experiments\n",
"2. nnictl trial ls list all of trial jobs\n",
"3. nnictl top monitor the status of running experiments\n",
"4. nnictl log stderr show stderr log content\n",
"5. nnictl log stdout show stdout log content\n",
"6. nnictl stop stop an experiment\n",
"7. nnictl trial kill kill a trial job by id\n",
"8. nnictl --help get help information about nnictl\n",
"-----------------------------------------------------------------------\n",
"\n"
]
}
],
"source": [
"import nnicli as nc\n",
"nc.start_nni(config_file='../trials/mnist/config.yml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Connect nnicli module to started nni experiment\n",
"Call set_endpoint to connect nnicli moduele to the rest server of started nni experiment. Local mode training serviced is used in this notebook, but nnicli module can connect to any started nni experiment. The endpoint can be found in the output of start_nni function."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"nc.set_endpoint('http://127.0.0.1:8080')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Retrieve nni experiment and trial job information"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'errors': [], 'status': 'RUNNING'}"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_status()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'trialJobNumber': 4, 'trialJobStatus': 'SUCCEEDED'},\n",
" {'trialJobNumber': 1, 'trialJobStatus': 'RUNNING'}]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_job_statistics()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'execDuration': 1117,\n",
" 'id': 'PlUIfDTR',\n",
" 'logDir': '/home/chicm/nni/experiments/PlUIfDTR',\n",
" 'maxSequenceId': 3,\n",
" 'params': {'authorName': 'default',\n",
" 'clusterMetaData': [{'key': 'codeDir',\n",
" 'value': '/mnt/d/Repos/nni/examples/trials/mnist/.'},\n",
" {'key': 'command', 'value': 'python3 mnist.py'}],\n",
" 'experimentName': 'example_mnist',\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"hidden_size\": {\"_value\": [124, 512, 1024], \"_type\": \"choice\"}, \"batch_size\": {\"_value\": [1, 4, 8, 16, 32], \"_type\": \"choice\"}, \"conv_size\": {\"_value\": [2, 3, 5, 7], \"_type\": \"choice\"}, \"dropout_rate\": {\"_value\": [0.5, 0.9], \"_type\": \"uniform\"}, \"learning_rate\": {\"_value\": [0.0001, 0.001, 0.01, 0.1], \"_type\": \"choice\"}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'trialConcurrency': 1,\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'checkpointDir': '/home/chicm/nni/experiments/PlUIfDTR/checkpoint',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'className': 'TPE'},\n",
" 'versionCheck': True},\n",
" 'revision': 116,\n",
" 'startTime': 1564484985839}"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define an utility function to format json string returned by nnicli module."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"def show_json(res):\n",
" print(json.dumps(res, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"params\": {\n",
" \"searchSpace\": \"{\\\"hidden_size\\\": {\\\"_value\\\": [124, 512, 1024], \\\"_type\\\": \\\"choice\\\"}, \\\"batch_size\\\": {\\\"_value\\\": [1, 4, 8, 16, 32], \\\"_type\\\": \\\"choice\\\"}, \\\"conv_size\\\": {\\\"_value\\\": [2, 3, 5, 7], \\\"_type\\\": \\\"choice\\\"}, \\\"dropout_rate\\\": {\\\"_value\\\": [0.5, 0.9], \\\"_type\\\": \\\"uniform\\\"}, \\\"learning_rate\\\": {\\\"_value\\\": [0.0001, 0.001, 0.01, 0.1], \\\"_type\\\": \\\"choice\\\"}}\",\n",
" \"clusterMetaData\": [\n",
" {\n",
" \"key\": \"codeDir\",\n",
" \"value\": \"/mnt/d/Repos/nni/examples/trials/mnist/.\"\n",
" },\n",
" {\n",
" \"key\": \"command\",\n",
" \"value\": \"python3 mnist.py\"\n",
" }\n",
" ],\n",
" \"tuner\": {\n",
" \"classArgs\": {\n",
" \"optimize_mode\": \"maximize\"\n",
" },\n",
" \"builtinTunerName\": \"TPE\",\n",
" \"checkpointDir\": \"/home/chicm/nni/experiments/PlUIfDTR/checkpoint\",\n",
" \"className\": \"TPE\"\n",
" },\n",
" \"maxTrialNum\": 10,\n",
" \"maxExecDuration\": 3600,\n",
" \"experimentName\": \"example_mnist\",\n",
" \"authorName\": \"default\",\n",
" \"trialConcurrency\": 1,\n",
" \"trainingServicePlatform\": \"local\",\n",
" \"versionCheck\": true\n",
" },\n",
" \"execDuration\": 1192,\n",
" \"revision\": 124,\n",
" \"logDir\": \"/home/chicm/nni/experiments/PlUIfDTR\",\n",
" \"maxSequenceId\": 3,\n",
" \"id\": \"PlUIfDTR\",\n",
" \"startTime\": 1564484985839\n",
"}\n"
]
}
],
"source": [
"show_json(nc.get_experiment_profile())"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\n",
" {\n",
" \"startTime\": 1564484995992,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":0,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":1024,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.8055724367106529}}\"\n",
" ],\n",
" \"id\": \"BW0NR\",\n",
" \"endTime\": 1564485259753,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 0,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"0\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"BW0NR\",\n",
" \"timestamp\": 1564485258774,\n",
" \"data\": \"0.9078999757766724\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/BW0NR\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485271947,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":1,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":4,\\\"conv_size\\\":5,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.01,\\\"dropout_rate\\\":0.5547528540531742}}\"\n",
" ],\n",
" \"id\": \"x0P5w\",\n",
" \"endTime\": 1564485642784,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 1,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"1\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"x0P5w\",\n",
" \"timestamp\": 1564485642072,\n",
" \"data\": \"0.10100000351667404\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/x0P5w\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485652151,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":2,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.5584485925416655}}\"\n",
" ],\n",
" \"id\": \"V9jSG\",\n",
" \"endTime\": 1564485917057,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 2,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"2\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"V9jSG\",\n",
" \"timestamp\": 1564485916403,\n",
" \"data\": \"0.928600013256073\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/V9jSG\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485927295,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":3,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":7,\\\"hidden_size\\\":124,\\\"learning_rate\\\":0.001,\\\"dropout_rate\\\":0.6281630602835235}}\"\n",
" ],\n",
" \"id\": \"CDlRX\",\n",
" \"status\": \"RUNNING\",\n",
" \"sequenceId\": 3,\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/CDlRX\"\n",
" }\n",
"]\n"
]
}
],
"source": [
"show_json(nc.list_trial_jobs())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing nni experiment result\n",
"\n",
"With the retrieved trial job information, we can do some analysis by visualizing the metric data, below is a simple example."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3QAAAF4CAYAAAAczbvpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl8TNfj//F3MiQoGoQIsdTSVBFSitoiERIkoiFFPnSh0ZbSaq1dkBa1VKuWVqtaSz6WplJLpPiqvSUUReujVKkiIhJbQiWZ5PeHn/mYT0gGE8nl9Xw8PB6Ze889c85xM5P3vefe65CdnZ0tAAAAAIDhOBZ0AwAAAAAAd4ZABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgCAXcycOVOjRo2yqeyQIUM0ffr0m6777rvv9OKLL9pUT8+ePRUTE2NzG6OiovTUU0/J29tbly5dkre3t06dOmXz9rfy8ccfa8SIEXddz91q3bq14uPjC7oZAIB7qEhBNwAAUPh4e3tbfr5y5YqcnJxkMpkkSZGRkercuXOObQYMGGCX93766af19NNP26WuG129elUTJ05UTEyMateuLUnas2eP3d+nsPj444+VmJioCRMmFHRTAAD5iEAHAMjhxqDj5+ensWPHqnnz5rcsn5mZqSJFCvdXSlJSktLT0y1hrqAYYawAAMbBlEsAwG37+OOP9frrr+uNN96Qt7e3VqxYYTXtMCsrS4MGDVKLFi3UuHFj9e7dW0eOHLGp7ujoaPXu3dvy+ueff1ZoaKgaNWqkbt26ae/evVbl//rrL8v6AQMG6MKFCznqPHLkiIKCgiRdO/vYp08fZWZmytPTUydOnJB0bRro2LFj9eKLL8rb21vdu3fX33//banjvffeU+vWrfXEE0+oa9eu2r17t039+emnn+Tn56dZs2apRYsWevfddyVJP/zwgzp37qzGjRurZ8+eOnTokGWbWbNmqWXLlnriiScUGBhomUb5v1NVr9f9vzZs2KA5c+Zo5cqV8vb2VmhoqGVs/fz85O3trbZt22rVqlU29QEAUHgR6AAAd2TdunUKCgrSrl271LFjxxzr27RpozVr1ujHH39U7dq1NXTo0Nt+j5SUFL300kvq06eP4uPj1atXL/Xr188qtC1btkyTJk3Sli1blJ2drQ8++CBHPTVr1tTy5cslXTv7+NVXX930/VauXKnXXntNO3bskLu7uz755BPLOi8vL61YsUI7duxQQECAXnvtNaWnp9vUj9OnT+vy5cvasGGDRo8erX379undd9/V2LFjFR8fr65du6p///5KT0/X4cOHtWTJEn333XfavXu3Zs+erUqVKt3OsMnX11d9+/ZVcHCw9uzZo5iYGKWmpmrChAn66quvtGfPHi1atEienp63VS8AoPAh0AEA7sgTTzwhPz8/OTo6qlixYlbrHB0dFRoaqpIlS8rZ2VmvvvqqfvvtN12+fPm23mPDhg2qXbu2goKCVKRIEXXp0kUeHh7auHGjpUyXLl1Uq1YtlShRQoMGDdKqVauUnZ19R30KCAhQ/fr1VbRoUQUHB+vgwYNW7+Pi4qIiRYooIiJCqamp+uuvv2yq12Qy6dVXX5WTk5OKFSumb775RuHh4fLy8pLJZFK3bt0kSfv375fJZNLVq1f1xx9/KDMzU1WqVFGVKlXuqD//y8HBQYcPH9bVq1dVoUIF1apVyy71AgAKDpP4AQB3xN3d/ZbrzGazpkyZojVr1ujcuXNydLx2/PDcuXMqUaKEze9x5syZHGenKleurMTExJu2o3LlykpPT9f58+dVpkwZm9/nuvLly1t+Ll68uFUAnT17tpYuXaqkpCQ5ODjoypUrOnfunE31urq6ysnJyfL61KlTWrlypebOnWtZlpGRocTERDVq1EjDhw/XJ598oj///FMtW7bUyJEjrdp2J0qWLKkpU6bo66+/1siRI9WoUSONGDFCjzzyyF3VCwAoWJyhAwDcEQcHh1uuW7ZsmTZv3qx58+Zp165dWrt2rSTd9pmzChUq5HiswKlTp+Tm5mZ5nZCQYLXOyclJLi4ut/U+edm+fbvmzp2r6dOn6+eff9bOnTtVokQJm/vzv2NVsWJFDRgwQD///LPl3969ey1TV0NCQrR48WL98MMPMpvN+uijjyRJJUqU0JUrVyz1JCUl2fyekuTj46O5c+dq69atqlq1qs2PmQAAFF4EOgCA3aWlpVmC1ZUrVzR16tQ7qsfX11eHDx9WXFycMjMztXLlSh0/flxt2rSxlFm+fLmOHDmiy5cva/r06erQoUOuYfNOpKWlyWQyqUyZMsrIyND06dOtgtXteuaZZ7Rw4ULt27dP2dnZSktL0/r163X58mUdOXJE27dvV3p6upydnVWsWDFLfx577DFt2rRJFy5c0JkzZ7RgwYJbvoerq6tOnjxpCZ1nzpzR+vXrdeXKFRUtWlQlSpSwnDkFABgXn+QAALsLDQ1VhQoV1KpVKwUFBVk91+52lC1bVp999plmz56tpk2bau7cuZo1a5YefvhhS5mQkBANHTpULVu2lNls1ltvvWWvblj4+PioefPmat++vfz8/FSyZMm7mgLZsGFDjRkzRmPGjNGTTz6pgIAArVixQpKUnp6uyZMnq2nTpmrZsqUuXLigwYMHS7o2rjVr1pSvr69efPFFderU6Zbv0bFjR2VkZKhJkybq1q2bsrKyNGfOHLVs2VJNmzbV7t27OUMHAPcBh+w7vXIcAIB8sGTJEq1Zs+aWd6IEAAD/xRk6AEChcvjwYXl4eBR0MwAAMATucgkAKDReeuklnTp1yur5bwAA4NaYcgkAAAAABsWUSwAAAAAwKAIdAAAAABgUgQ4AAAAADKpQ3RTl3Lk0ZWVxSR8AAACAB4ujo4PKlHnotrcrVIEuKyubQAcAAAAANmLKJQAAAAAYFIEOAAAAAAyKQAcAAAAABpVnoJs4caL8/Pzk6empQ4cO3bSM2WxWZGSk/P391a5dO0VHR9u9oQAAAAAePEeOHFaHDm3VrJm3OnRoqz///CNHmcTERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZkqQNG35Qu3Y+8vBw1ejRb1vVmdu6wiTPQNe2bVv9+9//VuXKlW9ZZuXKlTp+/LjWrl2rJUuWaPr06Tpx4oRdGwoAAADgwTN06GD16ROh7dv3qE+fCA0Z8nqOMqNHj1SDBt7atGmbli9frfHj39PJk9fyyCeffKjatT21adM2bdy4Tfv2/aJVq1ZIkqpVq66PPpquAQMG5agzt3WFSZ6BrnHjxnJ3d8+1TFxcnMLCwuTo6KiyZcvK399fq1evtlsjAQAAADx4kpKStG/fXoWGhkmSQkPDtG/fXp09e9aq3G+//So/P39Jkqurq+rVq6/ly7+TJDk4OCg1NVVZWVm6evWq0tMzVLFiJUlSjRo1Vb++l0ymnDf/z21dYWKXa+gSEhJUqVIly2t3d3edPn3aHlUDAAAAeECdOnVC7u7uMplMkiSTyaSKFSvq1Cnr2YBeXg313XdLlZ2drb/+OqadO+N14sRxSdIbbwzTn3/+oXr1aqtevdry9W2rpk2b3fO+5JdCFTfLlStZ0E0AAAAAUEiUKfOQTCZHlS9fyrLMZHJUmTIPWS2bOXOaBg8erHbtWqlq1ary9/dXqVIlVL58KcXELFSjRt7avHmjLl26pA4dOmjTpjXq1q2bZfuHHnJWdnaGVZ22rCsM7BLo3N3dderUKXl5eUnKecbOVsnJqfn6YPEjRw7r1Vdf1rlzKSpTpqxmzvxcNWrUsiqTmJiooUNf019//aXMzAy9/voQhYX1sKxfvjxGH300SdnZ2XJwcFB09ApVqFBBkyaN19y5X8rN7dr01CZNmmrixI8s7/vmm6/pwoULSk+/qpCQUA0b9la+9RMAAAC4HxQvXkYnTpzU6dPnZTKZZDabdfLkKRUvXkZJSZduKFlMH3/8meVVz55d1axZKyUlXdLUqZ9o6tSZSk5Ok+Sotm0DFRe3Rj4+AZbyaWlXdfly+v/Umfc6e3J0dLijE1x2mXIZGBio6OhoZWVlKSUlRevWrVNAQEDeG95jd3tB5S+/7NbkyR/om2+Wa/PmeK1cuUalS5e2bBsW1lMbNvyoDRt+tIQ5SYqMHKXg4BBt2PCj1qzZqMWL/63du3/O/w4DAADY4G7vIihdO+jt49NMrVs3lY9PM505c0bStbuhDx/+hp580ktNmjRQVNQ8yzZTpkxUq1ZN5OPzlPz9W2v9+nX521EYTvny5VWvXn3FxFy7i35MTLTq1/eSq6urVbmUlGTLnSu3bNmk//zngOW6u6pVq1n2rfT0dG3evFGPPfb4PexF/soz0I0dO1atW7fW6dOn9cILL6hTp06SpIiICO3fv1+SFBISIg8PD7Vv317PPPOMBgwYoCpVquRvy2+TPS6onDVrpvr3HyQ3NzdJUunSD6tYsWJ5vreDg4MuXrwoSbpy5YocHBzk6lrebn2D8d3tF+mkSeP1+OM15OvbQr6+LTR8+BtWdXfp0lG+vi3UokVjTZo03rLu8uXLioh4Xk2aNFDz5o20du33+dtRAEChlJ8Hvb/9domOHv1T8fG/KC7uB02e/IGOH/9LkuTt3Uhr1mzUpk3bNHXqTPXr94KuXLly7zoOQ5g8earmzPlczZp5a86czzV58lRJ187C/fLLbknSnj271KJFYzVv3kgTJ47TggVLVKJECUnS++9P0Pbt2+Tj00x+fi1Us2ZN9e79vCRp+/ZtatDgMc2aNVPz53+tBg0es4S/3NYVJg7Z2dn5N8fxNuXnlMu9e/fo1Vdf0pYtOyzLWrZ8Up9+OlteXg0tywYM6Kdy5VwVGTlOx4//pYCANgoNDdP48ZPl59dS7dq117ZtPyktLU2dOgVr8OChcnBw0KRJ4/Xvf8+Xi0sZVahQQcOGvaUnn2wqSfr77+Pq1au7zp1L0YUL5zVq1Pvq27dfvvQTxhQaGqSePXspLKyHoqMXa9GiKMXExFqVefnlPqpd21NvvjlcZ8+eVbt2rRUbu1aVK3to0qTxSktLU2TkuBx1P/tsT/n4tFHfvi8pNTVVrVs31ZdfztMTTzTWhx9O0KlTJ/XRR9P1559/KDg4UPHxv6hkSa5nBYAHRVJSkp566gn9/vsxy5Q2T8/q2r59j9VZkFatmmjatM/k7d1IktS7d3c99VRL9e8/UC+/3FetW7dReHjvHPWHh3dTz569FBzcRZI0YsSb8vCoqldffc2qXHZ2tmrVqqItW+JVqdKtH5cF3K8KdMrl/SQycrySks7I17eF3n57mFq1amO5VanZbNaBA78pOnq5li+P0w8//J+++WaRJOm55/rq55/3a9OmbRow4DU991xPpaQkS5Lmz/9aYWE9tG/f79qxY6++/HKWdu3aWWB9ROFij7PHucntDPHy5TF69tkXJEk1atRSw4beWr/+/+zWNwBA4WePuwgeOvS7/vrrqDp3DlTbtq0s9xuQpJMnT8jD478ztypXrpKjbklasmShqld/hDAH3KZCdZfL/FSpkocSEhJkNpstR59Onz6tSpU8rMq5urrqs8++tLzu2bOrPD19JUkeHh4KCgqRs7OznJ2dFRjYSXv27FL37uGWaZiS1KaNnypV8tDBg/9R8+YtNXv2LO3cuU+S5OZWUS1b+mjbtp/UqNGT96DnKOxy+yK98cjo9S/Shg2f0PHjf2nnznhVqVLVsn7ZsqXauHF9jjPEY8dOUK9e3fX1119azhBXrVpNknTixAl5ePy3jsqVPXTy5Ml70W0AgMFERo7Xu++OkK9vC3l4eNzyoHdGRrq6dw9V5coe6t493Ka6f/ppqyZOHKfo6OX52QUUkNIuxeVc9IGJHbm6mpGpi+ftO634gRnZGy+oDAvrkesFlaVLP6wiRYpYLqicM2eBpGtnTtatW6tnnumpzMxMbdmy0TJ9ICHhlNzdr93Zc//+ffr7779Us2ZtSVK1atW0fv3/qXv3cKWmXlJ8/E8KDOxwD3uP+0FuX6TPPddXgwcPVdGiRbVx43o991xPbd26U2XLlrOcIX711deUmHhaXbp0VMOG3hxQAABIyv+D3pUre+jEib8tUzVPnvzb6mDizp3x6t8/QvPnL1KtWrXvQY9xrzkXLaI3vttU0M0oFD562sfudT5QUy7v9oLKp5/uJlfX8mrZ8kn5+bWQp2cd/etfz0qSxo2LVOvWTdWmTXO9+eZAzZz5heWs3bRpn2nevK/Upk1zBQb6qXPnp9W2bfsCGAEURjd+kUrK84t048afFBX1jVJTL8nT8zFJkpubm4oWLSrJ+gyxJM2ePctyhPTGM8TStS/g69NlpGvTYipXZqoLADxI7HEXwdDQMG3cuF7Z2dnKyMjQli0bVbdufUlS585Pa8GCucrKytLZs2f1/ferFBwcIuna3139+r2gOXPmW93TAIDtHpibogCFWZcuHfWvfz1ruSnKwoUL9N13q6zK/O/Z44EDX9ZPP+1SiRIlcpwh7tYtWJs375Cbm5t8fJqpf/9BljPEHTv6a/To99W2bXtNmjRep08nWG6KEhQUoB07flHJkoXzwZm2uttnTi5aFKVZs2bK0dFRWVlm9er1nCIiXslzu6SkJL322is6efKkMjMz1KJFK40fP1lFijwwkyEAGNThw4c0cOBLOn/+vFxcXDRjxheqVau2evbsquHD31bDhk/ohx/W6q23hslkMqls2XL64IMPVb/+tWcQZ2VlafTot7V+/f/J0dFRbdq0VWTkODk6OspsNmvEiCHauPEHSdLAgYMt12+3b++jv/8+rooV//v84pkzv9Djj9e994OAfFO+fCnO0P1/Hz3tc8vn2d3pTVEIdEAhcLdfpK+++pL27ftFjo4mOTkV1bBhb8nf/9qzIPfu3aORI4fq8uXLyszMUJcuXTVkyAhJUlpamgYNekX79++VyWTSqFHvq0OHTgU2DvZyt3cNvXTpokqWLCUHBwelpl5S69bNtGDBEtWtWy/X7d55Z7hMpiKKjBynjIwMBQe31yuvDFRISGgBjQQAAAWPQPdf+RHoDHPYuFTpYirmXLSgm1Eo/HM1Q5cu/lPQzYAd1a79qFav3pBj+aJFSy0/t23bXvHxN5+qO2PG57esu0EDb8XF3fyZKQ899JDmzJl/m60t3K7fNfT6hfWhoWEaOXKozp49azV96LffftVLLw2QZH3X0P79B6pUqdKWcpcvX1FGRoYcHBzy3O5aAExVVlaWrl69qvT0DKujzgCAB0N+zhS57o8/Dqtt25Z6/vkXLY8tio5erBkzPtGhQwc1duwE9e370r3pMAqUYQJdMeeiCh/274JuRqGwcNK/dEkEOuBm7HXX0NWr4zRu3BgdO3ZUb7892jL9J7ft3nhjmPr06a169Wrr8uXL6tu3n5o2bXYPew/gQfXww8Xk5MSBb0lKT8/QhQsF+3fS9Qe1X58pMmTI6zlmilx/UPv8+YstMz6aN2+pypU9FBTUWT16/Mtqpkjz5q1Ut249Sdeutx8y5LUcs2rq1fPSF198rWnTPrpnfUXBM0ygAwqz0g87y9nJqaCbUShcTU/XxQtXC7oZecrtrqGSFBjYUYGBHXXixN967rlw+fsHqFat2rlut2LFMj3+eF0tXbpSqamX1KNHV61cucxyN1wAyC9OTkU1ZcqUgm5GofDmm29KBXjgO79nikjStGkfqV27QKWlpSotLc2yvE6dxyVJjo4P1H0PH3gEOsAOnJ2c9PzXrxV0MwqFuS98IqngAp09br99Iw+PKvL2bqS1a1erVq3auW43Z87nmjr12hSZ0qUfVmBgJ23duplABwAPkPyeKfLrr/u1YcMP+u67VZoyZeK97RwKJeI7gPuKPW6/fejQ75ZyycnJ+vHHzZajnrltV7VqNa1ff+16xfT0dG3evFGPPfZ4PvYWAGBUkZHjlZR0Rr6+LfT228NuOlNky5Yd2rZtt6Kjl+iPPw4rIyNDQ4YM0uTJUy2BEeAMHYD7zuTJUzVw4EuaMmWi5a6hkqzuGrpnzy6ru4be+MzJBQu+1saN61WkSFFlZ2erb99+8vVtK0m5bvf++xM0dOhg+fg0k9lsVosWrdS79/MFMgYAgIKRnzNFOnfuomPHjio8vJsk6cKFC8rOzlZq6iVNmTItfzuGQotAB+C+c7d3DX3//Qm3rDu37R55pIa+/Xb5bbYWAHA/uXGmSFhYj1xnitz4fNn//OeA5sxZIOnaTJFHH/WU9N+ZIp06BcvDo4oOHjxmqWPSpPFKS0uz3OUSDyYCHQAAAGBH+TlTJDcxMdGKjHxXFy6c1+rVcZo27WN9880yeXo+lq/9RcEi0AEodFxKOaloMeeCbkahkPHPVZ2/lF7QzQAA3Ib8nClyo2HD3rJ6HRoaZrmuGw8OAh2AQqdoMWfFPftCQTejUOg4/2uJQAcAAG6BQAcAAADcoIyLs4oU5fmykpSZka5z5wv/82UfZAQ6AAAA4AZFijppc+yYgm5GodA6aIwK8vmyyBvPoQMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUEVsKXT06FGNGDFC58+fl4uLiyZOnKjq1atblUlOTtbIkSOVkJCgzMxMNW3aVO+8846KFLHpLQAAAAAAt8mmM3SjR49WeHi41qxZo/DwcI0aNSpHmVmzZqlmzZpauXKlVqxYod9++01r1661e4MBAAAAANfkGeiSk5N14MABBQUFSZKCgoJ04MABpaSkWJVzcHBQWlqasrKylJ6eroyMDLm5ueVPqwEAAAAAeQe6hIQEubm5yWQySZJMJpMqVKighIQEq3L9+/fX0aNH1bJlS8u/Ro0a5U+rAQAAAAC2XUNni9WrV8vT01Pz5s1TWlqaIiIitHr1agUGBtpcR7lyJe3VnPte+fKlCroJwC2xf9oX4wkAt4fPTftiPO3L3uOZZ6Bzd3dXYmKizGazTCaTzGazzpw5I3d3d6tyUVFRGj9+vBwdHVWqVCn5+fkpPj7+tgJdcnKqsrKyb7qOHclaUtKlgm4CbsD+ae1u90/G0xq/7wDywuemNb6H7IvxtK9bjaejo8MdneDKc8pluXLlVKdOHcXGxkqSYmNjVadOHZUtW9aqnIeHhzZv3ixJSk9P17Zt21S7du3bbhAAAAAAwDY23eVyzJgxioqKUkBAgKKiohQZGSlJioiI0P79+yVJb731lnbt2qXg4GB16dJF1atX1zPPPJN/LQcAAACAB5xN19DVrFlT0dHROZbPnj3b8nPVqlX19ddf269lAAAAAIBc2XSGDgAAAABQ+BDoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIOy6cHiAIAH05Ejh/Xqqy/r3LkUlSlTVjNnfq4aNWpZlRkwoJ8OHPjN8vrAgV81b94iBQZ21JQpE7Vs2VI5OppUtGhRvfXWKPn5+UuSunbtrJSUZEmS2Zypgwf/ow0bflLduvV05Mhhvfnma7pw4YLS068qJCRUw4a9de86DgCAQRDoAAC3NHToYPXpE6GwsB6Kjl6sIUNeV0xMrFWZmTO/sPz866/71bVrkHx920qSvL0b6ZVXBqpEiRL69df96tKlo/bvP6TixYtr6dIVlu3i4mI1YcL7qlu3niQpMnKUgoND1LfvS0pNTVXr1k3l799eTzzR+B70GgAA42DKJQDgppKSkrRv316FhoZJkkJDw7Rv316dPXv2ltssXDhfXbs+I2dnZ0mSn5+/SpQoIUmqW7eesrOzde5cSo7tFi1aoJ49e1teOzg46OLFi5KkK1euyMHBQa6u5e3WNwAA7hcEOgDATZ06dULu7u4ymUySJJPJpIoVK+rUqRM3LZ+enq6YmGirYHajJUsWqnr1R1SpUmWr5YmJidq8eaPCwnpYlo0dO0HLlsXIy8tTjRvXU//+g1S1ajU79QwAgPsHgQ4AYBfffx+rypWrqH59rxzrfvppqyZOHKfPP/8qx7pvvlkkX19/ubq6WpbNn/+1wsJ6aN++37Vjx159+eUs7dq1M1/bDwCAERHoAAA3VamShxISEmQ2myVJZrNZp0+fVqVKHjctv3DhAoWH98qxfOfOePXvH6F58xaqVq3aOdYvXhyVY7vZs2epe/dwSZKbW0W1bOmjbdt+utsuAQBw3yHQAQBuqnz58qpXr75iYqIlSTEx0apf38vqTNp1p06dVHz8NnXt+ozV8j17dqlfvxc0Z858eXk1zLHdjh3xunjxotq2bW+1vFq1alq//v8kSamplxQf/5Pq1Kljr64BAHDfINABAG5p8uSpmjPnczVr5q05cz7X5MlTJUk9e3bVL7/stpRbsmSh2rcPlItLGavthw9/Q//8c0VDhrwuX98W8vVtYfWIg8WLo/TMMz0t1+ldN23aZ5o37yu1adNcgYF+6tz56RyhDwAA8NgCAEAuatd+VKtXb8ixfNGipVavBw8eetPt167dlGv9H300/abLGzTwVlzcOhtbCQDAg4szdAAAAABgUJyhA4D72MOli8vJmY96SUq/mqkLF68UdDMAALArvuUB4D7m5FxE49/+tqCbUSi8Na5bQTcBAAC7Y8olAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMZtZ4OAAAgAElEQVQi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADMqmQHf06FF1795dAQEB6t69u44dO3bTcnFxcQoODlZQUJCCg4N19uxZe7YVAAAAAHCDIrYUGj16tMLDwxUSEqLly5dr1KhRmj9/vlWZ/fv3a8aMGZo3b57Kly+vS5cuycnJKV8aDQAAAACw4QxdcnKyDhw4oKCgIElSUFCQDhw4oJSUFKtyc+fOVZ8+fVS+fHlJUqlSpeTs7JwPTQYAAAAASDYEuoSEBLm5uclkMkmSTCaTKlSooISEBKtyR44c0d9//61//etfevrpp/Xpp58qOzs7f1oNAAAAALBtyqUtzGazfv/9d3399ddKT0/Xiy++qEqVKqlLly4211GuXEl7Nee+V758qYJuAnBL7J/2xXjaD2MJPBj4XbcvxtO+7D2eeQY6d3d3JSYmymw2y2QyyWw268yZM3J3d7cqV6lSJQUGBsrJyUlOTk5q27at9u3bd1uBLjk5VVlZNz+rx45kLSnpUkE3ATdg/7R2t/sn42ntbsaTsbTGZyfuV/yuW+N7yL4YT/u61Xg6Ojrc0QmuPKdclitXTnXq1FFsbKwkKTY2VnXq1FHZsmWtygUFBWnr1q3Kzs5WRkaGtm/frscee+y2GwQAAAAAsI1Njy0YM2aMoqKiFBAQoKioKEVGRkqSIiIitH//fklSp06dVK5cOXXs2FFdunRRrVq11K1bt/xrOQAAAAA84Gy6hq5mzZqKjo7OsXz27NmWnx0dHTVy5EiNHDnSfq0DAAAAANySTWfoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQNgW6o0ePqnv37goICFD37t117NixW5b9888/1aBBA02cONFebQQAAAAA3IRNgW706NEKDw/XmjVrFB4erlGjRt20nNls1ujRo+Xv72/XRgIAAAAAcsoz0CUnJ+vAgQMKCgqSJAUFBenAgQNKSUnJUfaLL75QmzZtVL16dbs3FAAAAABgLc9Al5CQIDc3N5lMJkmSyWRShQoVlJCQYFXu4MGD2rp1q55//vl8aSgAAAAAwFoRe1SSkZGhd999Vx988IEl+N2JcuVK2qM5D4Ty5UsVdBOAW2L/tC/G034YS+DBwO+6fTGe9mXv8cwz0Lm7uysxMVFms1kmk0lms1lnzpyRu7u7pUxSUpKOHz+ufv36SZIuXryo7Oxspaam6v3337e5McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5RnoypUrpzp16ig2NlYhISGKjY1VnTp1VLZsWUuZSpUqKT4+3vJ6+vTpunz5soYPH37bDQIAAAAA2Mamu1yOGTNGUVFRCggIUFRUlCIjIyVJERER2r9/f742EAAAAABwczZdQ1ezZk1FR0fnWD579uyblh84cODdtQoAAAAAkCebztABAAAAAAofAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBFbGl0NGjRzVixAidP39eLi4umjhxoqpXr25VZubMmYqLi5Ojo6OKFi2qwYMHq1WrVvnRZgAAAACAbAx0o0ePVnh4uEJCQrR8+XKNGjVK8+fPtyrj5eWlPn36qHjx4jp48KB69eqlrVu3qlixYvnScAAAAAB40OU55TI5OVkHDhxQUFCQJCkoKEgHDhxQSkqKVblWrVqpePHikiRPT09lZ2fr/Pnz+dBkAAAAAIBkQ6BLSEiQm5ubTCaTJMlkMqlChQpKSEi45TbLli1T1apVVbFiRfu1FAAAAABgxaYpl7djx44d+uSTT/TVV1/d9rblypW0d3PuW+XLlyroJgC3xP5pX4yn/TCWwIOB33X7Yjzty97jmWegc3d3V2Jiosxms0wmk8xms86cOSN3d/ccZffs2aOhQ4fq008/VY0aNW67McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5Tnlsly5cqpTp45iY2MlSbGxsapTp47Kli1rVW7fvn0aPHiwpk2bprp16952QwAAAAAAt8em59CNGTNGUVFRCggIUFRUlCIjIyVJERER2r9/vyQpMjJS//zzj0aNGqWQkBCFhITo999/z7+WAwAAAMADzqZr6GrWrKno6Ogcy2fPnm35eenSpfZrFQAAAAAgTzadoQMAAAAAFD4EOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHS4I0eOHFaHDm3VrJm3OnRoqz///CNHmQ0bflC7dj7y8HDV6NFvW60zm80aPvwNPfmkl5o0aaCoqHmWdYsWRcnH5yn5+raQj08zzZ79mWXdlCkT1apVE/n4PCV//9Zav35d/nUSAAAAKOQIdLgjQ4cOVp8+Edq+fY/69InQkCGv5yhTrVp1ffTRdA0YMCjHum+/XaKjR/9UfPwviov7QZMnf6Djx/+SJAUFddbGjT9pw4YftWrV/+mzz2bot99+lSR5ezfSmjUbtWnTNk2dOlP9+r2gK1eu5G9nAcBObDkYltsBr8TERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZedYJADA2Ah1uW1JSkvbt26vQ0DBJUmhomPbt26uzZ89alatRo6bq1/eSyVQkRx3Ll8eod+/n5ejoKFdXV3Xo0EkrViyTJJUqVVoODg6SpMuXrygjI8Py2s/PXyVKlJAk1a1bT9nZ2Tp3LiXf+goA9mTLwbDcDniNHj1SDRp4a9OmbVq+fLXGj39PJ0+ekCR98smHql3bU5s2bdPGjdu0b98vWrVqRZ51Gll+BuQBA/rJ17eF5Z+b28NavTrOqu4//jisatXccsxCAYB7iUCH23bq1Am5u7vLZDJJkkwmkypWrKhTp07YXMfJkyfk4VHF8rpy5SpW269eHadWrZqoUaO6GjBgkB5/vG6OOpYsWajq1R9RpUqV76I3AHBv2HowLLcDXr/99qv8/PwlSa6urqpXr76WL/9OkuTg4KDU1FRlZWXp6tWrSk/PUMWKlfKs08jyMyDPnPmFNmz4URs2/Kjp02fJxcVFvr5tLfWazWYNGfKaOnTodG86CwC3QKBDoRQY2FFbtuzQtm27FR29RH/8cdhq/U8/bdXEieP0+edfFVALAeD22HowLLcDXl5eDfXdd0uVnZ2tv/46pp0743XixHFJ0htvDNOff/6hevVqq1692vL1baumTZvlWadR5XdAvtHChfPVteszcnZ2tiybNu0jtWsXqBo1auVXFwHAJgQ63LZKlTyUkJAgs9ks6dpRytOnT6tSJQ+b66hc2UMnTvxteX3y5N833d7Do4q8vRtp7drVlmU7d8arf/8IzZu3ULVq1b6LngCAsURGjldS0hn5+rbQ228PU6tWbSzT2lesWKbHH6+rX389rH37Dmrbth+1cqXxz8LdSn4H5OvS09MVExOtnj17W5b9+ut+bdjwg15+eUB+dQ8AbEagw20rX7686tWrr5iYaElSTEy06tf3kqurq811dO78tBYsmKusrCydPXtW33+/SsHBIZKkQ4d+t5RLTk7Wjz9uVp06j0uS9uzZpX79XtCcOfPl5dXQjr0CgPxl68Gw3A54ubq66rPPvtTGjT8pKuobpaZekqfnY5KkOXM+V9euz8jR0VGlSz+swMBO2rp1c551PshyC8jXff99rCpXrqL69b0kSRkZGRoyZJAmT55qCZMAUJAIdLgjkydP1Zw5n6tZM2/NmfO5Jk+eKknq2bOrfvlltyRp+/ZtatDgMc2aNVPz53+tBg0eszxmICysh6pVe0RNmzZUhw5+evPN4apWrbokacGCr9WqVRP5+rZQ167B6tu3n+W6heHD39A//1zRkCGvWy5UP3Dgt3s/AABwm2w9GJbbAa+UlGTLnSu3bNmk//zngGXKYdWq1Syfsenp6dq8eaMee+zxPOs0qvwOyNctXLhA4eG9LK8TE0/r2LGjCg/vpkaN6umLLz5TVNQ8vflmzjs6A8C9kPP2g4ANatd+VKtXb8ixfNGipZafmzV7Snv3Hrzp9iaTSZMnf3zTde+/P+GW77t27abbbCkAFB6TJ0/VwIEvacqUiXJxcdGMGV9IunYwbPjwt9Ww4RMKC+uhXbt+VtOm12Yh3HjAa8+eXXrrrWEymUwqW7acFixYYrnz7/vvT9DQoYPl49NMZrNZLVq0Uu/ez0tSrnUa1Y0BOSysR54BuVOnzkpJSdH336/SihXXpvGnpCSrdOmHVaRIEUtAnjNngWXbU6dOKj5+m9X12h4eVXTw4DHL60mTxistLU2RkePyt8MAcAsEOgAA7hFbDobldsCrbdv2io9vf9N1jzxSQ99+u/ym63Kr08jyMyBL1+6m3L59oFxcytzzvgGArQh0D6gyDzupiJNz3gUfAJnpV3XuQnpBNwMAcJvyMyBL0uDBQ/Nsw7Bhb9nQUgDIPwS6B1QRJ2ftmvRiQTejUGg07EtJBDoAeXu4tJOcnDkYJknpV6/qwsW7++ws83BxFXHiTxFJykzP1LkLVwq6GQAMiE9RAABs5OTsrI9GvlTQzSgU3vjgc93twbAiTkW099ONdmmP0TXo36agmwDAoLjLJQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYlE2B7ujRo+revbsCAgLUvXt3HTt2LEcZs9msyMhI+fv7q127doqOjrZ3WwEAAAAAN7Ap0I0ePVrh4eFas2aNwsPDNWrUqBxlVq5cqePHj2vt2rVasmSJpk+frhMnTti9wQAAAACAa/IMdMnJyTpw4ICCgoIkSUFBQTpw4IBSUlKsysXFxSksLEyOjo4qW7as/P39tXr16vxpNQAAAABARfIqkJCQIDc3N5lMJkmSyWRShQoVlJCQoLJly1qVq1SpkuW1u7u7Tp8+fVuNcXR0yHW9a5mHbqu++1leY2ULp9Ll7NCS+4M9xtO1ZNm8Cz0g7DGexV3ZP6+72/F82KWEnVpifPbYN0u7sG9eZ4/xLFqqmB1acn+42/EsXbq0nVpifPbYN52Lu9ihJfcHe4xnmRLOdmjJ/eFW43mn4+yQnZ2dnVuBX3/9VcOHD9eqVassyzp27KjJkyerbt26lmXBwcEaN26cvLy8JEmzZ89WYmKi3nnnnTtqGAAAAAAgd3lOuXR3d1diYqLMZrOkazc/OXPmjNzd3XOUO3XqlOV1QkKCKlasaOfmAgAAAACuyzPQlStXTnXq1FFsbKwkKTY2VnXq1LGabilJgYGBio6OVlZWllJSUrRu3ToFBATkT6sBAAAAAHlPuZSkI0eOaMSIEbp48aJKly6tiRMnqkaNGoqIiNCgQYNUv359mc1mvffee/rxxx8lSREREerevXu+dwAAAAAAHlQ2BToAAAAAQOFj03PoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6Gy0bt06Pf300woMDJS/v78mTJig9PT0PLeLj4/X1q1b70ELC8aFCxfk5eWlsWPHWpbFxMRo0KBBOcreanleMjIyNH36dAUEBKhTp07q3LmzBg0apD/++OOu2l6YpKena8KECfL391dgYKC6dOmidevWSbq2D4WGhlqVP3TokPz8/HLUM2PGDHl6eurQoUNWy48cOSJPT0/NnTvXavnJkyfVr18/BQcHKzg4WKGhoTm2vZ/87ziHhITo+++/l3RtnBs0aKCQkBDLv2HDhlnW3fh/4OnpqbS0NElS7969tWHDhnvfmXvEz89PgYGB6ty5s9q1a6dXXnlFu3fvtqyfOXOm/P395e/vr5kzZ9pc77Zt2xQaGqpOnTqpU6dOOnjwYH403xCmT5+e5/dJ79691bZtW6v98/r/g5+f3y1/b3Nbdz+53s8RI0aodevWCgkJUUBAgD788ENLGU9PTwUHB6tz584KDg7WDz/8YFk3ffp0TZw48aZ1x8TEyNPTU3FxcVbL7uT77EF14/j+72dtWFhYAbfu3rrx+yMiIkLHjx+XJI0YMUJRUVGSru1fjRs3VkhIiDp27Kh+/fopKSnJqh6z2axWrVrplVdeubcdKGQSExPVu3dvSdL58+fVunVr7du3z7J+1qxZGjhwoK5cuaJ69erp7NmzlnWhoaFWv8f79++Xj4+PpGv7rKenp/bu3WtZn9vnREErUtANMIKdO3fqvffe0+zZs+Xp6amrV69q+PDhioyM1Lhx43LddseOHbp8+bJatmx5j1p7b8XGxqpBgwZatWqVhg0bJicnJ7u/x8iRI/XPP/8oOjpapUuXVnZ2tjZt2qSjR4+qVq1adn+/gjBmzBhdvnxZq1atkrOzsw4dOqQXX3xRDz/8sM11/Pbbb/rll19UuXLlHOu+/fZbNWvWTEuXLtXzzz9vWR4ZGanWrVurV69ekq59MBYpcv9+LNxsnPv27SsXFxc5OjqqZs2aiomJKehmFjrTpk3To48+Kklau3at+vXrpzlz5ig9PV2rV69WbGysJCksLExNmjTRk08+mWt9iYmJevvtt/Xll1+qRo0a+ueff5SZmZnv/SisZsyYoT59+uT5+fnOO+/I19f3HrXKuPr166devXrp0qVLCgkJkbe3t9q2bStJWrx4sR566CFt2rRJr7/+unbu3GnTZ17lypX1ySefqH379vf1Z+S9wmftNbNnz77luubNm2vatGnKzs7WG2+8oRkzZigyMtKyfsuWLapQoYJ2796ts2fPytXV9V40udBxc3PTggULJEkuLi4aNWqURo4cqe+++05Hjx7Vv//9by1btkzFixeXl5eXduzYoY4dOyo1NVX//POP1QGvHTt2qEmTJpbXlStX1pQpUzR//vx73q/b9cCdofv00081fvx4y+tz586padOmGjRokOXIiGR9pGT69Ol65ZVX5OnpKUlydnbWmDFjFBcXp5MnT+rIkSPy8fHRyZMnJV37ch48eLB+//13LV68WMuWLVNISIi++OKLe9jTe2Pp0qXq37+/PD09rY525iUhIUGhoaGWI55r1qyxnJmaNWuW5QjWsWPHtG7dOo0bN06lS5eWJDk4OKhNmzZq165dvvTpXjt58qS+//57jRkzRs7OzpKkRx99VC+//LJmzJhhUx3p6el67733NGbMmBzrMjMztXLlSr333nu6evWq1ZGr06dPy83NzfLazc1N5cqVu7sOFVK3GudXXnnF5nGG1L59e/Xo0UNz5sxRXFycunTpomLFiqlYsWLq0qWL5Xf60qVLGjRokAIDA9W7d28NHTrUcmRz4cKFCgkJUY0aNSRJxYoVU8mSJSVJrVq1UnJysqRrR6/79esnSUpOTlbr1q3vdXfz3fU/0Hr06KGQkBC1a9fOcuaiRYsWGjBggE31rFixQqGhoWrXrp3Vd9mDrFSpUqpfv76OHj2aY13Tpk11+fJlXbx40bIsMTFRERERCgwMVL9+/XTlyhXLunr16umRRx7Rt99+e0/aXhisXbvWMpNh5syZlu/lW31f3+rvq8uXLxdgLwo3W86gOzg46Mknn1RCQoLV8qVLl6pHjx7y9/fXsmXL8rOZhcbevXvVu3dvhYaGKjQ0VBs3btSJEyfUtGlTSxl/f3/VqVNHH374oUaMGKGRI0da/q5p0qSJ4uPjJUm7du1S48aNVb16dR0+fFjStUB3Y13t27fX+fPntWXLlnvYyzvzwAW6639wXD8aHBsbKz8/P5UoUeKW2/z+++9q2LCh1TIXFxdVqVJFhw4dUs2aNTV48GANHjxYW7du1cqVK/X+++/L09NTPXr0UJcuXbR8+XLLHyb3i4MH/1979x5TZf0HcPx9BghMEMXCJBHyBqHjYigHZxuB0yMUcEjLrHCRKIph0ooNEHA1WDlNKi6jDLUtGnIOHMfqlEXguMoERcd+GDZMUmSMWBgSxyO/P8545iPgJTPh8H1t/PPceJ4vj8/39vl8/R+9vb0olUoiIyPRaDT3fN6WLVtISkoiJCSE7u5uUlNTycvLo7S0FBsbG+nYlpYWXF1d72umaqI5f/48c+fOZfr06bLtPj4+UhjahQsXZKFWt4f6ZGVlERYWxpw5c0Zcv6KiAldXV1xdXVGr1bK/0+bNm3nvvfd47bXX2Lt3r6yzZ27uVM7DFert5Sw6eqPz9vamra2NK1eu4OzsLG2fPXu21OjIzs5m6tSp6PV6srKyaGhokI5ra2vjr7/+IioqioiICDIzM6WQQ39/f+rq6jAYDHR0dNDR0YHBYKC2tlZW0ZqLtLQ0wDRzpNPpOH78ODqdjtzcXGxtbYmJiZGO/eCDD2Tv53DHF0wdXq1WS2FhIXl5eZM6hHXY1atXaWxsxNPTc8S+48ePo1QqcXR0lLadO3eOffv28d1330kDYbfatWsXubm5DAwMPPR7f9S6u7vZvXs3OTk56HQ6afa4t7d3zPr6ftpX7e3tqNVq1q9fT0lJyX/zUBPU4OAgJ06cICQkRNrW09NDXV0da9euJTIyclLMdv7555+kpaWxb98+tFoteXl5pKam0tfXN+LY3bt3U1xczJNPPikrN39/f06ePAmYQn+XL1+On58fJ0+exGg0curUKVk9o1AoSEhI4OOPP2ZoaOjhP+QDmHQdOmdnZxYsWEBlZSUAJSUlI/KT/omIiAjmzZtHXFwc+/btk0abzVlxcTHh4eEoFApWr15Nc3MzV69eveM5ra2t7NixgwMHDuDn5weYRlw8PT1xc3MD4MUXXxzz/La2Nik34ta8vYnsXj4S8+fPR6fTST+ffPKJtK+pqYlz586xcePGUc/VaDSo1WrA9J7q9Xr+/vtvAMLCwigvL2fjxo1cv36dTZs2SeFz5uZO5axQKICR5bxjx47/6vYmlHt5Z+vr61m3bh0Ajo6Oshl1o9FIY2Mj2dnZFBUVceXKFSmCISAggJqaGs6cOYOPjw9eXl6cOXOGmpoalErlw3mgcaavr4/Y2Fjeffdd2WBiSkqK7P28dTZ9uKwfe+wxAgMDpUbLZJSfn094eDjbtm1j8+bNrFixQtq3YcMGgoKCSE5O5u2335adt3LlSqZNm4ZCocDLy0vKbRrm7u7OsmXLpPAuczZcLw/Por/88suAaUB2rPr6XttXixcvprKykpKSEvbv3092djY1NTUP+YkmnpqaGsLDwwkICKCnp4e1a9dK+44dO8Zzzz2HnZ0dzzzzDEajkaampkd4tw9fU1MTHR0dxMTEEB4eTkxMDAqFYtRw/draWuzs7Pj1119l+cm+vr50dHTQ3d1NQ0MDy5cvl2btWlpasLe3x8XFRXatwMBArK2tpXz78WrSdegA1Go1paWltLAYJCIAAAefSURBVLa20tfXh5+fHxYWFty8eVM6ZrjBC6aP+OnTp2XX6O3t5dKlSyxcuBAwjaD88ssv2Nvby0ZNzdXg4CBlZWVoNBqCgoIICQnBYDDcdZRo1qxZODg4SFPed+Pp6cnFixelsJgFCxag0+l4/fXXuXbt2gM/x3iwaNEifvvtN3p7e2XbT58+LYX53klDQwMXLlwgODiYoKAgOjs7efPNN6mqqqK7u5uqqio+++wzgoKCePXVV7l+/Trff/+9dP6MGTMICQkhNTWVbdu2mW2H7k7l7Ovr+4juamI6e/YsCxcuZPbs2Vy+fFnafuXKFWbPnn3X852dnQkMDMTe3p4pU6agUqk4e/YsAEqlktraWmpra1EqlSiVSurq6qirqyMgIOChPdN4YTAYeOutt1Cr1axZs+ZR386EtGXLFnQ6HVqtlk2bNsn2ffPNN5SXl7Nz504SEhJkdf1wKDaAhYUFRqNxxLV37tzJoUOHRp0VEEZvX93Ozs4Oe3t7AFxcXFi1apVsoSXBZMWKFeh0OiorK1EoFGRlZUn7NBoN1dXVBAUFERQURE9Pzz1HSU1UQ0NDuLu7ywa1KisrmTFjhuy4np4eMjIyyM/PZ8mSJbIBcBsbG7y9vfn555/p7+/HyckJT09PWlpaRuTP3eqdd94hKytrXOd6T8oO3erVq2loaKCgoAC1Wo1CocDV1VVqUHR1dck6HHFxceTm5tLa2gqYOnvp6emoVCopxO2jjz5i8eLFFBQUkJaWRmdnJ2D6cJnjh/+nn37iqaee4sSJE5SXl1NeXs6XX35519CJ6dOnc+jQIY4dO0ZBQQFgCt9qaWmRRkNvvYabmxvBwcGkpKTIytGcYvLnzJmDSqUiPT1dalycP3+evLy8e5oh2rJlC1VVVdLf4YknnuDgwYOsXLmS0tJS1qxZQ0VFhbQ/IyND+vBXVFRIv9NoNNLa2jpq2KY5GKucDx8+PGKkXhjbjz/+SGFhIdHR0ahUKkpLSxkYGGBgYIDS0lJpFFmpVEoDPH/88Ye0aivA888/T319PYODgwwNDVFVVYWHhwdgSkK3sLCgpKSEgIAAAgIC0Gq1WFpaysI7zcnUqVOlAarU1FTmzZvHG2+8cV/XGP5u9vT0UFlZaZbhqf+m6OhoZs6cSWFh4X2d5+Liwpo1azh8+PBDurPxwcfHh5aWFtrb2wE4evQoAB4eHmPW1zB6++p2XV1d0ix/b28v1dXV0r9/YSQ7Ozv27NlDYWEhXV1dNDc309fXJ6v3y8rK0Ov1srxPc+Pr68vFixepq6uTtjU3N4+IGNmzZw8vvfQSHh4eJCcnU1ZWJrXvwZRH98UXX7B06VIALC0tmTt3LkVFRWN+N4dz7W4Pwx5PJuVSTba2tgQHB6PVaqWFPNavX098fDwhISG4ubnh5eUlHe/v709KSgqJiYkMDAxgMBgIDg4mISEBMDVwTp48ydGjR7G2tiYuLo6EhASOHDkiJauGh4cTGhpqNnl0Go2GF154QbbN19eXmzdvcvnyZSorK2ULGERGRjJ37lzAlKh+8OBBYmNj6e/vJy4ujvT0dGJiYrC1tSUwMBArKytsbW0ByMzMJCcnh3Xr1mFpacm0adNwcnIym7IEUx7N/v37CQkJwcrKCmtra5KTk2UJvP+EVqslMTFRti04OJi0tDQ6Ojqor6/nww8/xNLSEqPRyJIlS9i5c+eDPs64dWs5KxQKrl69SlFREU8//TT19fVSDt0wJyenO65CNlnEx8czZcoUrl+/zvz588nPz8fb2xswNeBCQ0MBU0jv8Ajn9u3bSUpKQqVS8fjjj8tG6pcuXcqzzz5LREQEFhYWeHp6snXrVml/QEAAp06dwsnJCTCNqo420m8uoqOjiYqKwmAw8Pvvv7No0SLpPfT39ycpKQkw5dAdOHBAOi8+Pl5avXHGjBlERkbS19fH1q1b72l235zcuHFDNsN2NwqFgsTERHbt2sWGDRvu63dt377d7PO+Zs6cyfvvv09sbCw2NjasXr0aMA3K3qm+Hq19dbsffviBwsJCqd6JiIhg1apV/9mzjTfDnd47vcMeHh6oVCo+//xzBgcHCQ0NlXWWZ82ahaenJ3q9XkqxMDcODg7k5OSwd+9eMjIyMBgMuLi4sHv3bumYb7/9lvb2dum/K3FwcCA1NZWkpCQ0Gg1TpkzB39+f7Oxs2YJTy5Yto7q6eswZOjDl0I7nslUMjfcsP2FSuHbtmpR3qNFoKC4uvu+RU0G4HwaDgdTUVDo7O8nLy7uvxqBw/z799FP6+/tHDDAIwoPq6upi7dq1VFdXyxbpEP5d7u7uNDY2MjQ0JOrrf8GNGzfw9/dHr9fj6OiIWq0mJSXljp0KQRjLpJyhE8afr776Cr1ej9FoxMHBwWwWPBHGLysrKzIzMx/1bQiC8ACOHDnC119/TWJioujM/UdEff3guru7eeWVVwgLC+PSpUtERUXh5+cnOnPCPyZm6ARBEARBEARBECaoSbkoiiAIgiAIgiAIgjkQHTpBEARBEARBEIQJSnToBEEQBEEQBEEQJijRoRMEQRAEQRAEQZigRIdOEARBEARBEARhghIdOkEQBEEQBEEQhAnq/9qDRoCeJ1AFAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1080x432 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"sns.set(style=\"whitegrid\")\n",
"\n",
"jobs = nc.list_trial_jobs()\n",
"job_ids = [x['id'] for x in jobs]\n",
"final_metrics = [float(x['finalMetricData'][0]['data']) for x in jobs]\n",
"\n",
"data = {'job id': job_ids, 'final metrics': final_metrics}\n",
"sns.set(rc={'figure.figsize':(15, 6)})\n",
"\n",
"plt.title('Trial job final results')\n",
"ax = sns.barplot(x='job id', y='final metrics', data=data) \n",
"\n",
"for i,p in enumerate(ax.patches):\n",
" ax.annotate('{:.4f}'.format(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()),\n",
" ha='center', va='center', fontsize=11, color='black', rotation=0, xytext=(0, 5),\n",
" textcoords='offset points') "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stop nni experiment"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: Stoping experiment PlUIfDTR\n",
"INFO: Stop experiment success.\n"
]
}
],
"source": [
"nc.stop_nni()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/trials/auto-feature-engineering/README.md 0 → 100644
View file @ 88ef6c04
**Automatic Feature Engineering in nni**
===
Now we have an [example](https://github.com/SpongebBob/tabular_automl_NNI), which could automaticlly do feature engineering in nni.
These code come from our contributors. And thanks our lovely contributors!
And welcome more and more people to join us!
examples/tuners/random_nas_tuner/random_nas_tuner.py
View file @ 88ef6c04
...@@ -7,7 +7,9 @@ def random_archi_generator(nas_ss, random_state): ...@@ -7,7 +7,9 @@ def random_archi_generator(nas_ss, random_state):
''' '''
chosen_archi = {} chosen_archi = {}
print("zql: nas search space: ", nas_ss) print("zql: nas search space: ", nas_ss)
for block_name, block in nas_ss.items(): for block_name, block_value in nas_ss.items():
assert block_value['_type'] == "mutable_layer", "Random NAS Tuner only receives NAS search space whose _type is 'mutable_layer'"
block = block_value['_value']
tmp_block = {} tmp_block = {}
for layer_name, layer in block.items(): for layer_name, layer in block.items():
tmp_layer = {} tmp_layer = {}
......
setup.py
View file @ 88ef6c04
...@@ -35,9 +35,10 @@ setup( ...@@ -35,9 +35,10 @@ setup(
license = 'MIT', license = 'MIT',
url = 'https://github.com/Microsoft/nni', url = 'https://github.com/Microsoft/nni',
packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('tools'), packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('src/sdk/pycli') + find_packages('tools'),
package_dir = { package_dir = {
'nni': 'src/sdk/pynni/nni', 'nni': 'src/sdk/pynni/nni',
'nnicli': 'src/sdk/pycli/nnicli',
'nni_annotation': 'tools/nni_annotation', 'nni_annotation': 'tools/nni_annotation',
'nni_cmd': 'tools/nni_cmd', 'nni_cmd': 'tools/nni_cmd',
'nni_trial_tool':'tools/nni_trial_tool', 'nni_trial_tool':'tools/nni_trial_tool',
......
src/nni_manager/rest_server/restValidationSchemas.ts
View file @ 88ef6c04
...@@ -51,10 +51,12 @@ export namespace ValidationSchemas { ...@@ -51,10 +51,12 @@ export namespace ValidationSchemas {
command: joi.string().min(1), command: joi.string().min(1),
virtualCluster: joi.string(), virtualCluster: joi.string(),
shmMB: joi.number(), shmMB: joi.number(),
authFile: joi.string(),
nasMode: joi.string().valid('classic_mode', 'enas_mode', 'oneshot_mode'), nasMode: joi.string().valid('classic_mode', 'enas_mode', 'oneshot_mode'),
worker: joi.object({ worker: joi.object({
replicas: joi.number().min(1).required(), replicas: joi.number().min(1).required(),
image: joi.string().min(1), image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(), outputDir: joi.string(),
cpuNum: joi.number().min(1), cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100), memoryMB: joi.number().min(100),
...@@ -64,6 +66,7 @@ export namespace ValidationSchemas { ...@@ -64,6 +66,7 @@ export namespace ValidationSchemas {
ps: joi.object({ ps: joi.object({
replicas: joi.number().min(1).required(), replicas: joi.number().min(1).required(),
image: joi.string().min(1), image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(), outputDir: joi.string(),
cpuNum: joi.number().min(1), cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100), memoryMB: joi.number().min(100),
...@@ -73,6 +76,7 @@ export namespace ValidationSchemas { ...@@ -73,6 +76,7 @@ export namespace ValidationSchemas {
master: joi.object({ master: joi.object({
replicas: joi.number().min(1).required(), replicas: joi.number().min(1).required(),
image: joi.string().min(1), image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(), outputDir: joi.string(),
cpuNum: joi.number().min(1), cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100), memoryMB: joi.number().min(100),
...@@ -83,6 +87,7 @@ export namespace ValidationSchemas { ...@@ -83,6 +87,7 @@ export namespace ValidationSchemas {
name: joi.string().min(1), name: joi.string().min(1),
taskNum: joi.number().min(1).required(), taskNum: joi.number().min(1).required(),
image: joi.string().min(1), image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(), outputDir: joi.string(),
cpuNum: joi.number().min(1), cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100), memoryMB: joi.number().min(100),
......
src/nni_manager/training_service/kubernetes/frameworkcontroller/frameworkcontrollerConfig.ts
View file @ 88ef6c04
...@@ -43,8 +43,8 @@ export class FrameworkControllerTrialConfigTemplate extends KubernetesTrialConfi ...@@ -43,8 +43,8 @@ export class FrameworkControllerTrialConfigTemplate extends KubernetesTrialConfi
public readonly taskNum: number; public readonly taskNum: number;
constructor(taskNum: number, command : string, gpuNum : number, constructor(taskNum: number, command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string, cpuNum: number, memoryMB: number, image: string,
frameworkAttemptCompletionPolicy: FrameworkAttemptCompletionPolicy) { frameworkAttemptCompletionPolicy: FrameworkAttemptCompletionPolicy, privateRegistryFilePath?: string | undefined) {
super(command, gpuNum, cpuNum, memoryMB, image); super(command, gpuNum, cpuNum, memoryMB, image, privateRegistryFilePath);
this.frameworkAttemptCompletionPolicy = frameworkAttemptCompletionPolicy; this.frameworkAttemptCompletionPolicy = frameworkAttemptCompletionPolicy;
this.name = name; this.name = name;
this.taskNum = taskNum; this.taskNum = taskNum;
......
src/nni_manager/training_service/kubernetes/frameworkcontroller/frameworkcontrollerTrainingService.ts
View file @ 88ef6c04
...@@ -305,7 +305,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -305,7 +305,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
} }
// Generate frameworkcontroller job resource config object // Generate frameworkcontroller job resource config object
const frameworkcontrollerJobConfig: any = const frameworkcontrollerJobConfig: any =
this.generateFrameworkControllerJobConfig(trialJobId, trialWorkingFolder, frameworkcontrollerJobName, podResources); await this.generateFrameworkControllerJobConfig(trialJobId, trialWorkingFolder, frameworkcontrollerJobName, podResources);
return Promise.resolve(frameworkcontrollerJobConfig); return Promise.resolve(frameworkcontrollerJobConfig);
} }
...@@ -329,8 +329,8 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -329,8 +329,8 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
* @param frameworkcontrollerJobName job name * @param frameworkcontrollerJobName job name
* @param podResources pod template * @param podResources pod template
*/ */
private generateFrameworkControllerJobConfig(trialJobId: string, trialWorkingFolder: string, private async generateFrameworkControllerJobConfig(trialJobId: string, trialWorkingFolder: string,
frameworkcontrollerJobName : string, podResources : any) : any { frameworkcontrollerJobName : string, podResources : any) : Promise<any> {
if (this.fcClusterConfig === undefined) { if (this.fcClusterConfig === undefined) {
throw new Error('frameworkcontroller Cluster config is not initialized'); throw new Error('frameworkcontroller Cluster config is not initialized');
} }
...@@ -345,12 +345,14 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -345,12 +345,14 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
if (containerPort === undefined) { if (containerPort === undefined) {
throw new Error('Container port is not initialized'); throw new Error('Container port is not initialized');
} }
const taskRole: any = this.generateTaskRoleConfig( const taskRole: any = this.generateTaskRoleConfig(
trialWorkingFolder, trialWorkingFolder,
this.fcTrialConfig.taskRoles[index].image, this.fcTrialConfig.taskRoles[index].image,
`run_${this.fcTrialConfig.taskRoles[index].name}.sh`, `run_${this.fcTrialConfig.taskRoles[index].name}.sh`,
podResources[index], podResources[index],
containerPort containerPort,
await this.createRegistrySecret(this.fcTrialConfig.taskRoles[index].privateRegistryAuthPath)
); );
taskRoles.push({ taskRoles.push({
name: this.fcTrialConfig.taskRoles[index].name, name: this.fcTrialConfig.taskRoles[index].name,
...@@ -363,7 +365,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -363,7 +365,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
}); });
} }
return { return Promise.resolve({
apiVersion: `frameworkcontroller.microsoft.com/v1`, apiVersion: `frameworkcontroller.microsoft.com/v1`,
kind: 'Framework', kind: 'Framework',
metadata: { metadata: {
...@@ -379,11 +381,11 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -379,11 +381,11 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
executionType: 'Start', executionType: 'Start',
taskRoles: taskRoles taskRoles: taskRoles
} }
}; });
} }
private generateTaskRoleConfig(trialWorkingFolder: string, replicaImage: string, runScriptFile: string, private generateTaskRoleConfig(trialWorkingFolder: string, replicaImage: string, runScriptFile: string,
podResources: any, containerPort: number): any { podResources: any, containerPort: number, privateRegistrySecretName: string | undefined): any {
if (this.fcClusterConfig === undefined) { if (this.fcClusterConfig === undefined) {
throw new Error('frameworkcontroller Cluster config is not initialized'); throw new Error('frameworkcontroller Cluster config is not initialized');
} }
...@@ -451,13 +453,22 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple ...@@ -451,13 +453,22 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
mountPath: '/mnt/frameworkbarrier' mountPath: '/mnt/frameworkbarrier'
}] }]
}]; }];
const spec: any = {
containers: containers, let spec: any = {
initContainers: initContainers, containers: containers,
restartPolicy: 'OnFailure', initContainers: initContainers,
volumes: volumeSpecMap.get('nniVolumes'), restartPolicy: 'OnFailure',
hostNetwork: false volumes: volumeSpecMap.get('nniVolumes'),
hostNetwork: false
}; };
if(privateRegistrySecretName) {
spec.imagePullSecrets = [
{
name: privateRegistrySecretName
}
]
}
if (this.fcClusterConfig.serviceAccountName !== undefined) { if (this.fcClusterConfig.serviceAccountName !== undefined) {
spec.serviceAccountName = this.fcClusterConfig.serviceAccountName; spec.serviceAccountName = this.fcClusterConfig.serviceAccountName;
} }
......
src/nni_manager/training_service/kubernetes/kubeflow/kubeflowConfig.ts
View file @ 88ef6c04
...@@ -135,8 +135,8 @@ export class KubeflowTrialConfig extends KubernetesTrialConfig { ...@@ -135,8 +135,8 @@ export class KubeflowTrialConfig extends KubernetesTrialConfig {
export class KubeflowTrialConfigTemplate extends KubernetesTrialConfigTemplate { export class KubeflowTrialConfigTemplate extends KubernetesTrialConfigTemplate {
public readonly replicas: number; public readonly replicas: number;
constructor(replicas: number, command : string, gpuNum : number, constructor(replicas: number, command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string) { cpuNum: number, memoryMB: number, image: string, privateRegistryAuthPath?: string) {
super(command, gpuNum, cpuNum, memoryMB, image); super(command, gpuNum, cpuNum, memoryMB, image, privateRegistryAuthPath);
this.replicas = replicas; this.replicas = replicas;
} }
} }
......
src/nni_manager/training_service/kubernetes/kubeflow/kubeflowTrainingService.ts
View file @ 88ef6c04
...@@ -347,7 +347,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -347,7 +347,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
} }
// Generate kubeflow job resource config object // Generate kubeflow job resource config object
const kubeflowJobConfig: any = this.generateKubeflowJobConfig(trialJobId, trialWorkingFolder, kubeflowJobName, workerPodResources, const kubeflowJobConfig: any = await this.generateKubeflowJobConfig(trialJobId, trialWorkingFolder, kubeflowJobName, workerPodResources,
nonWorkerResources); nonWorkerResources);
return Promise.resolve(kubeflowJobConfig); return Promise.resolve(kubeflowJobConfig);
...@@ -361,8 +361,8 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -361,8 +361,8 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
* @param workerPodResources worker pod template * @param workerPodResources worker pod template
* @param nonWorkerPodResources non-worker pod template, like ps or master * @param nonWorkerPodResources non-worker pod template, like ps or master
*/ */
private generateKubeflowJobConfig(trialJobId: string, trialWorkingFolder: string, kubeflowJobName : string, workerPodResources : any, private async generateKubeflowJobConfig(trialJobId: string, trialWorkingFolder: string, kubeflowJobName : string, workerPodResources : any,
nonWorkerPodResources?: any) : any { nonWorkerPodResources?: any) : Promise<any> {
if (this.kubeflowClusterConfig === undefined) { if (this.kubeflowClusterConfig === undefined) {
throw new Error('Kubeflow Cluster config is not initialized'); throw new Error('Kubeflow Cluster config is not initialized');
} }
...@@ -377,29 +377,32 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -377,29 +377,32 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
const replicaSpecsObj: any = {}; const replicaSpecsObj: any = {};
const replicaSpecsObjMap: Map<string, object> = new Map<string, object>(); const replicaSpecsObjMap: Map<string, object> = new Map<string, object>();
if (this.kubeflowTrialConfig.operatorType === 'tf-operator') { if (this.kubeflowTrialConfig.operatorType === 'tf-operator') {
const tensorflowTrialConfig: KubeflowTrialConfigTensorflow = <KubeflowTrialConfigTensorflow>this.kubeflowTrialConfig; const tensorflowTrialConfig: KubeflowTrialConfigTensorflow = <KubeflowTrialConfigTensorflow>this.kubeflowTrialConfig;
let privateRegistrySecretName = await this.createRegistrySecret(tensorflowTrialConfig.worker.privateRegistryAuthPath);
replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.worker.replicas, replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.worker.replicas,
tensorflowTrialConfig.worker.image, 'run_worker.sh', workerPodResources); tensorflowTrialConfig.worker.image, 'run_worker.sh', workerPodResources, privateRegistrySecretName);
if (tensorflowTrialConfig.ps !== undefined) { if (tensorflowTrialConfig.ps !== undefined) {
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(tensorflowTrialConfig.ps.privateRegistryAuthPath);
replicaSpecsObj.Ps = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.ps.replicas, replicaSpecsObj.Ps = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.ps.replicas,
tensorflowTrialConfig.ps.image, 'run_ps.sh', nonWorkerPodResources); tensorflowTrialConfig.ps.image, 'run_ps.sh', nonWorkerPodResources, privateRegistrySecretName);
} }
replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {tfReplicaSpecs: replicaSpecsObj}); replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {tfReplicaSpecs: replicaSpecsObj});
} else if (this.kubeflowTrialConfig.operatorType === 'pytorch-operator') { } else if (this.kubeflowTrialConfig.operatorType === 'pytorch-operator') {
const pytorchTrialConfig: KubeflowTrialConfigPytorch = <KubeflowTrialConfigPytorch>this.kubeflowTrialConfig; const pytorchTrialConfig: KubeflowTrialConfigPytorch = <KubeflowTrialConfigPytorch>this.kubeflowTrialConfig;
if (pytorchTrialConfig.worker !== undefined) { if (pytorchTrialConfig.worker !== undefined) {
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(pytorchTrialConfig.worker.privateRegistryAuthPath);
replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.worker.replicas, replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.worker.replicas,
pytorchTrialConfig.worker.image, 'run_worker.sh', workerPodResources); pytorchTrialConfig.worker.image, 'run_worker.sh', workerPodResources, privateRegistrySecretName);
} }
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(pytorchTrialConfig.master.privateRegistryAuthPath);
replicaSpecsObj.Master = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.master.replicas, replicaSpecsObj.Master = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.master.replicas,
pytorchTrialConfig.master.image, 'run_master.sh', nonWorkerPodResources); pytorchTrialConfig.master.image, 'run_master.sh', nonWorkerPodResources, privateRegistrySecretName);
replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {pytorchReplicaSpecs: replicaSpecsObj}); replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {pytorchReplicaSpecs: replicaSpecsObj});
} }
return { return Promise.resolve({
apiVersion: `kubeflow.org/${this.kubernetesCRDClient.apiVersion}`, apiVersion: `kubeflow.org/${this.kubernetesCRDClient.apiVersion}`,
kind: this.kubernetesCRDClient.jobKind, kind: this.kubernetesCRDClient.jobKind,
metadata: { metadata: {
...@@ -412,7 +415,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -412,7 +415,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
} }
}, },
spec: replicaSpecsObjMap.get(this.kubernetesCRDClient.jobKind) spec: replicaSpecsObjMap.get(this.kubernetesCRDClient.jobKind)
}; });
} }
/** /**
...@@ -424,7 +427,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -424,7 +427,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
* @param podResources pod resource config section * @param podResources pod resource config section
*/ */
private generateReplicaConfig(trialWorkingFolder: string, replicaNumber: number, replicaImage: string, runScriptFile: string, private generateReplicaConfig(trialWorkingFolder: string, replicaNumber: number, replicaImage: string, runScriptFile: string,
podResources: any): any { podResources: any, privateRegistrySecretName: string | undefined): any {
if (this.kubeflowClusterConfig === undefined) { if (this.kubeflowClusterConfig === undefined) {
throw new Error('Kubeflow Cluster config is not initialized'); throw new Error('Kubeflow Cluster config is not initialized');
} }
...@@ -436,7 +439,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -436,7 +439,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
if (this.kubernetesCRDClient === undefined) { if (this.kubernetesCRDClient === undefined) {
throw new Error('Kubeflow operator client is not initialized'); throw new Error('Kubeflow operator client is not initialized');
} }
// The config spec for volume field
const volumeSpecMap: Map<string, object> = new Map<string, object>(); const volumeSpecMap: Map<string, object> = new Map<string, object>();
if (this.kubeflowClusterConfig.storageType === 'azureStorage') { if (this.kubeflowClusterConfig.storageType === 'azureStorage') {
volumeSpecMap.set('nniVolumes', [ volumeSpecMap.set('nniVolumes', [
...@@ -459,7 +462,34 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -459,7 +462,34 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
} }
}]); }]);
} }
// The config spec for container field
const containersSpecMap: Map<string, object> = new Map<string, object>();
containersSpecMap.set('containers', [
{
// Kubeflow tensorflow operator requires that containers' name must be tensorflow
// TODO: change the name based on operator's type
name: this.kubernetesCRDClient.containerName,
image: replicaImage,
args: ['sh', `${path.join(trialWorkingFolder, runScriptFile)}`],
volumeMounts: [
{
name: 'nni-vol',
mountPath: this.CONTAINER_MOUNT_PATH
}],
resources: podResources
}
]);
let spec: any = {
containers: containersSpecMap.get('containers'),
restartPolicy: 'ExitCode',
volumes: volumeSpecMap.get('nniVolumes')
}
if (privateRegistrySecretName) {
spec.imagePullSecrets = [
{
name: privateRegistrySecretName
}]
}
return { return {
replicas: replicaNumber, replicas: replicaNumber,
template: { template: {
...@@ -467,26 +497,9 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber ...@@ -467,26 +497,9 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
// tslint:disable-next-line:no-null-keyword // tslint:disable-next-line:no-null-keyword
creationTimestamp: null creationTimestamp: null
}, },
spec: { spec: spec
containers: [
{
// Kubeflow tensorflow operator requires that containers' name must be tensorflow
// TODO: change the name based on operator's type
name: this.kubernetesCRDClient.containerName,
image: replicaImage,
args: ['sh', `${path.join(trialWorkingFolder, runScriptFile)}`],
volumeMounts: [
{
name: 'nni-vol',
mountPath: this.CONTAINER_MOUNT_PATH
}],
resources: podResources
}],
restartPolicy: 'ExitCode',
volumes: volumeSpecMap.get('nniVolumes')
}
} }
}; }
} }
} }
// tslint:enable: no-unsafe-any no-any // tslint:enable: no-unsafe-any no-any
......
src/nni_manager/training_service/kubernetes/kubernetesConfig.ts
View file @ 88ef6c04
...@@ -179,6 +179,9 @@ export class KubernetesTrialConfigTemplate { ...@@ -179,6 +179,9 @@ export class KubernetesTrialConfigTemplate {
// Docker image // Docker image
public readonly image: string; public readonly image: string;
// Private registry config file path to download docker iamge
public readonly privateRegistryAuthPath?: string;
// Trail command // Trail command
public readonly command : string; public readonly command : string;
...@@ -186,12 +189,13 @@ export class KubernetesTrialConfigTemplate { ...@@ -186,12 +189,13 @@ export class KubernetesTrialConfigTemplate {
public readonly gpuNum : number; public readonly gpuNum : number;
constructor(command : string, gpuNum : number, constructor(command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string) { cpuNum: number, memoryMB: number, image: string, privateRegistryAuthPath?: string) {
this.command = command; this.command = command;
this.gpuNum = gpuNum; this.gpuNum = gpuNum;
this.cpuNum = cpuNum; this.cpuNum = cpuNum;
this.memoryMB = memoryMB; this.memoryMB = memoryMB;
this.image = image; this.image = image;
this.privateRegistryAuthPath = privateRegistryAuthPath;
} }
} }
......
src/nni_manager/training_service/kubernetes/kubernetesTrainingService.ts
View file @ 88ef6c04
...@@ -38,6 +38,8 @@ import { KubernetesClusterConfig } from './kubernetesConfig'; ...@@ -38,6 +38,8 @@ import { KubernetesClusterConfig } from './kubernetesConfig';
import { kubernetesScriptFormat, KubernetesTrialJobDetail } from './kubernetesData'; import { kubernetesScriptFormat, KubernetesTrialJobDetail } from './kubernetesData';
import { KubernetesJobRestServer } from './kubernetesJobRestServer'; import { KubernetesJobRestServer } from './kubernetesJobRestServer';
var fs = require('fs');
/** /**
* Training Service implementation for Kubernetes * Training Service implementation for Kubernetes
*/ */
...@@ -327,5 +329,34 @@ abstract class KubernetesTrainingService { ...@@ -327,5 +329,34 @@ abstract class KubernetesTrainingService {
return Promise.resolve(); return Promise.resolve();
} }
protected async createRegistrySecret(filePath: string | undefined): Promise<string | undefined> {
if(filePath === undefined || filePath === '') {
return undefined;
}
let body = fs.readFileSync(filePath).toString('base64');
let registrySecretName = String.Format('nni-secret-{0}', uniqueString(8)
.toLowerCase());
await this.genericK8sClient.createSecret(
{
apiVersion: 'v1',
kind: 'Secret',
metadata: {
name: registrySecretName,
namespace: 'default',
labels: {
app: this.NNI_KUBERNETES_TRIAL_LABEL,
expId: getExperimentId()
}
},
type: 'kubernetes.io/dockerconfigjson',
data: {
'.dockerconfigjson': body
}
}
);
return registrySecretName;
}
} }
export { KubernetesTrainingService }; export { KubernetesTrainingService };
src/nni_manager/training_service/pai/paiConfig.ts
View file @ 88ef6c04
...@@ -71,6 +71,8 @@ export class PAIJobConfig { ...@@ -71,6 +71,8 @@ export class PAIJobConfig {
public readonly image: string; public readonly image: string;
// Code directory on HDFS // Code directory on HDFS
public readonly codeDir: string; public readonly codeDir: string;
//authentication file used for private Docker registry
public readonly authFile?: string;
// List of taskRole, one task role at least // List of taskRole, one task role at least
public taskRoles: PAITaskRole[]; public taskRoles: PAITaskRole[];
...@@ -87,12 +89,13 @@ export class PAIJobConfig { ...@@ -87,12 +89,13 @@ export class PAIJobConfig {
* @param taskRoles List of taskRole, one task role at least * @param taskRoles List of taskRole, one task role at least
*/ */
constructor(jobName: string, image : string, codeDir : string, constructor(jobName: string, image : string, codeDir : string,
taskRoles : PAITaskRole[], virtualCluster: string) { taskRoles : PAITaskRole[], virtualCluster: string, authFile?: string) {
this.jobName = jobName; this.jobName = jobName;
this.image = image; this.image = image;
this.codeDir = codeDir; this.codeDir = codeDir;
this.taskRoles = taskRoles; this.taskRoles = taskRoles;
this.virtualCluster = virtualCluster; this.virtualCluster = virtualCluster;
this.authFile = authFile;
} }
} }
...@@ -129,14 +132,17 @@ export class NNIPAITrialConfig extends TrialConfig { ...@@ -129,14 +132,17 @@ export class NNIPAITrialConfig extends TrialConfig {
public virtualCluster?: string; public virtualCluster?: string;
//Shared memory for one task in the task role //Shared memory for one task in the task role
public shmMB?: number; public shmMB?: number;
//authentication file used for private Docker registry
public authFile?: string;
constructor(command : string, codeDir : string, gpuNum : number, cpuNum: number, memoryMB: number, constructor(command : string, codeDir : string, gpuNum : number, cpuNum: number, memoryMB: number,
image: string, virtualCluster?: string, shmMB?: number) { image: string, virtualCluster?: string, shmMB?: number, authFile?: string) {
super(command, codeDir, gpuNum); super(command, codeDir, gpuNum);
this.cpuNum = cpuNum; this.cpuNum = cpuNum;
this.memoryMB = memoryMB; this.memoryMB = memoryMB;
this.image = image; this.image = image;
this.virtualCluster = virtualCluster; this.virtualCluster = virtualCluster;
this.shmMB = shmMB; this.shmMB = shmMB;
this.authFile = authFile;
} }
} }
src/nni_manager/training_service/pai/paiTrainingService.ts
View file @ 88ef6c04
...@@ -442,7 +442,7 @@ class PAITrainingService implements TrainingService { ...@@ -442,7 +442,7 @@ class PAITrainingService implements TrainingService {
// Task command // Task command
nniPaiTrialCommand, nniPaiTrialCommand,
// Task shared memory // Task shared memory
this.paiTrialConfig.shmMB this.paiTrialConfig.shmMB,
) )
]; ];
...@@ -456,7 +456,9 @@ class PAITrainingService implements TrainingService { ...@@ -456,7 +456,9 @@ class PAITrainingService implements TrainingService {
// PAI Task roles // PAI Task roles
paiTaskRoles, paiTaskRoles,
// Add Virutal Cluster // Add Virutal Cluster
this.paiTrialConfig.virtualCluster === undefined ? 'default' : this.paiTrialConfig.virtualCluster.toString() this.paiTrialConfig.virtualCluster === undefined ? 'default' : this.paiTrialConfig.virtualCluster.toString(),
//Task auth File
this.paiTrialConfig.authFile
); );
// Step 2. Upload code files in codeDir onto HDFS // Step 2. Upload code files in codeDir onto HDFS
......
src/sdk/pycli/nnicli/__init__.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
from .nni_client import *
src/sdk/pycli/nnicli/nni_client.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
""" A python wrapper for nni rest api
Example:
import nnicli as nc
nc.start_nni('../../../../examples/trials/mnist/config.yml')
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_experiment_status())
print(nc.get_job_statistics())
print(nc.list_trial_jobs())
nc.stop_nni()
"""
import sys
import os
import subprocess
import requests
__all__ = [
'start_nni',
'stop_nni',
'set_endpoint',
'version',
'get_experiment_status',
'get_experiment_profile',
'get_trial_job',
'list_trial_jobs',
'get_job_statistics',
'get_job_metrics',
'export_data'
]
EXPERIMENT_PATH = 'experiment'
VERSION_PATH = 'version'
STATUS_PATH = 'check-status'
JOB_STATISTICS_PATH = 'job-statistics'
TRIAL_JOBS_PATH = 'trial-jobs'
METRICS_PATH = 'metric-data'
EXPORT_DATA_PATH = 'export-data'
API_ROOT_PATH = 'api/v1/nni'
_api_endpoint = None
def set_endpoint(endpoint):
"""set endpoint of nni rest server for nnicli, for example:
http://localhost:8080
"""
global _api_endpoint
_api_endpoint = endpoint
def _check_endpoint():
if _api_endpoint is None:
raise AssertionError("Please call set_endpoint to specify nni endpoint")
def _nni_rest_get(api_path, response_type='json'):
_check_endpoint()
uri = '{}/{}/{}'.format(_api_endpoint, API_ROOT_PATH, api_path)
res = requests.get(uri)
if _http_succeed(res.status_code):
if response_type == 'json':
return res.json()
elif response_type == 'text':
return res.text
else:
raise AssertionError('Incorrect response_type')
else:
return None
def _http_succeed(status_code):
return status_code // 100 == 2
def _create_process(cmd):
if sys.platform == 'win32':
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
else:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while process.poll() is None:
output = process.stdout.readline()
if output:
print(output.decode('utf-8').strip())
return process.returncode
def start_nni(config_file):
"""start nni experiment with specified configuration file"""
cmd = 'nnictl create --config {}'.format(config_file).split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to start nni.')
def stop_nni():
"""stop nni experiment"""
cmd = 'nnictl stop'.split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to stop nni.')
def version():
"""return version of nni"""
return _nni_rest_get(VERSION_PATH, 'text')
def get_experiment_status():
"""return experiment status as a dict"""
return _nni_rest_get(STATUS_PATH)
def get_experiment_profile():
"""return experiment profile as a dict"""
return _nni_rest_get(EXPERIMENT_PATH)
def get_trial_job(trial_job_id):
"""return trial job information as a dict"""
assert trial_job_id is not None
return _nni_rest_get(os.path.join(TRIAL_JOBS_PATH, trial_job_id))
def list_trial_jobs():
"""return information for all trial jobs as a list"""
return _nni_rest_get(TRIAL_JOBS_PATH)
def get_job_statistics():
"""return trial job statistics information as a dict"""
return _nni_rest_get(JOB_STATISTICS_PATH)
def get_job_metrics(trial_job_id=None):
"""return trial job metrics"""
api_path = METRICS_PATH if trial_job_id is None else os.path.join(METRICS_PATH, trial_job_id)
return _nni_rest_get(api_path)
def export_data():
"""return exported information for all trial jobs"""
return _nni_rest_get(EXPORT_DATA_PATH)
src/sdk/pycli/setup.py 0 → 100644
View file @ 88ef6c04
import setuptools
setuptools.setup(
name = 'nnicli',
version = '999.0.0-developing',
packages = setuptools.find_packages(),
python_requires = '>=3.5',
install_requires = [
'requests'
],
author = 'Microsoft NNI Team',
author_email = 'nni@microsoft.com',
description = 'nnicli for Neural Network Intelligence project',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
)
src/sdk/pynni/nni/hyperopt_tuner/hyperopt_tuner.py
View file @ 88ef6c04
...@@ -190,13 +190,19 @@ class HyperoptTuner(Tuner): ...@@ -190,13 +190,19 @@ class HyperoptTuner(Tuner):
HyperoptTuner is a tuner which using hyperopt algorithm. HyperoptTuner is a tuner which using hyperopt algorithm.
""" """
def __init__(self, algorithm_name, optimize_mode='minimize'): def __init__(self, algorithm_name, optimize_mode='minimize',
parallel_optimize=False, constant_liar_type='min'):
""" """
Parameters Parameters
---------- ----------
algorithm_name : str algorithm_name : str
algorithm_name includes "tpe", "random_search" and anneal". algorithm_name includes "tpe", "random_search" and anneal".
optimize_mode : str optimize_mode : str
parallel_optimize : bool
More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
constant_liar_type : str
constant_liar_type including "min", "max" and "mean"
More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
""" """
self.algorithm_name = algorithm_name self.algorithm_name = algorithm_name
self.optimize_mode = OptimizeMode(optimize_mode) self.optimize_mode = OptimizeMode(optimize_mode)
...@@ -205,6 +211,13 @@ class HyperoptTuner(Tuner): ...@@ -205,6 +211,13 @@ class HyperoptTuner(Tuner):
self.rval = None self.rval = None
self.supplement_data_num = 0 self.supplement_data_num = 0
self.parallel = parallel_optimize
if self.parallel:
self.CL_rval = None
self.constant_liar_type = constant_liar_type
self.running_data = []
self.optimal_y = None
def _choose_tuner(self, algorithm_name): def _choose_tuner(self, algorithm_name):
""" """
Parameters Parameters
...@@ -266,6 +279,10 @@ class HyperoptTuner(Tuner): ...@@ -266,6 +279,10 @@ class HyperoptTuner(Tuner):
# but it can cause deplicate parameter rarely # but it can cause deplicate parameter rarely
total_params = self.get_suggestion(random_search=True) total_params = self.get_suggestion(random_search=True)
self.total_data[parameter_id] = total_params self.total_data[parameter_id] = total_params
if self.parallel:
self.running_data.append(parameter_id)
params = split_index(total_params) params = split_index(total_params)
return params return params
...@@ -287,10 +304,39 @@ class HyperoptTuner(Tuner): ...@@ -287,10 +304,39 @@ class HyperoptTuner(Tuner):
raise RuntimeError('Received parameter_id not in total_data.') raise RuntimeError('Received parameter_id not in total_data.')
params = self.total_data[parameter_id] params = self.total_data[parameter_id]
# code for parallel
if self.parallel:
constant_liar = kwargs.get('constant_liar', False)
if constant_liar:
rval = self.CL_rval
else:
rval = self.rval
self.running_data.remove(parameter_id)
# update the reward of optimal_y
if self.optimal_y is None:
if self.constant_liar_type == 'mean':
self.optimal_y = [reward, 1]
else:
self.optimal_y = reward
else:
if self.constant_liar_type == 'mean':
_sum = self.optimal_y[0] + reward
_number = self.optimal_y[1] + 1
self.optimal_y = [_sum, _number]
elif self.constant_liar_type == 'min':
self.optimal_y = min(self.optimal_y, reward)
elif self.constant_liar_type == 'max':
self.optimal_y = max(self.optimal_y, reward)
logger.debug("Update optimal_y with reward, optimal_y = %s", self.optimal_y)
else:
rval = self.rval
if self.optimize_mode is OptimizeMode.Maximize: if self.optimize_mode is OptimizeMode.Maximize:
reward = -reward reward = -reward
rval = self.rval
domain = rval.domain domain = rval.domain
trials = rval.trials trials = rval.trials
...@@ -375,13 +421,26 @@ class HyperoptTuner(Tuner): ...@@ -375,13 +421,26 @@ class HyperoptTuner(Tuner):
total_params : dict total_params : dict
parameter suggestion parameter suggestion
""" """
if self.parallel and len(self.total_data)>20 and len(self.running_data) and self.optimal_y is not None:
self.CL_rval = copy.deepcopy(self.rval)
if self.constant_liar_type == 'mean':
_constant_liar_y = self.optimal_y[0] / self.optimal_y[1]
else:
_constant_liar_y = self.optimal_y
for _parameter_id in self.running_data:
self.receive_trial_result(parameter_id=_parameter_id, parameters=None, value=_constant_liar_y, constant_liar=True)
rval = self.CL_rval
rval = self.rval random_state = np.random.randint(2**31 - 1)
else:
rval = self.rval
random_state = rval.rstate.randint(2**31 - 1)
trials = rval.trials trials = rval.trials
algorithm = rval.algo algorithm = rval.algo
new_ids = rval.trials.new_trial_ids(1) new_ids = rval.trials.new_trial_ids(1)
rval.trials.refresh() rval.trials.refresh()
random_state = rval.rstate.randint(2**31 - 1)
if random_search: if random_search:
new_trials = hp.rand.suggest(new_ids, rval.domain, trials, new_trials = hp.rand.suggest(new_ids, rval.domain, trials,
random_state) random_state)
......
test/cli_test.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge,
# to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import sys
import time
import traceback
from utils import GREEN, RED, CLEAR, setup_experiment
def test_nni_cli():
import nnicli as nc
config_file = 'config_test/examples/mnist.test.yml'
try:
# Sleep here to make sure previous stopped exp has enough time to exit to avoid port conflict
time.sleep(6)
print(GREEN + 'Testing nnicli:' + config_file + CLEAR)
nc.start_nni(config_file)
time.sleep(3)
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_job_statistics())
print(nc.get_experiment_status())
nc.list_trial_jobs()
print(GREEN + 'Test nnicli {}: TEST PASS'.format(config_file) + CLEAR)
except Exception as error:
print(RED + 'Test nnicli {}: TEST FAIL'.format(config_file) + CLEAR)
print('%r' % error)
traceback.print_exc()
raise error
finally:
nc.stop_nni()
if __name__ == '__main__':
installed = (sys.argv[-1] != '--preinstall')
setup_experiment(installed)
test_nni_cli()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment