Skip to content

GitLab

Unverified Commit 88ef6c04 authored by SparkSnail's avatar SparkSnail Committed by GitHub
Browse files

Merge pull request #197 from microsoft/master 

merge master
parents 5f3c5ffd 555334de
Hide whitespace changes
Inline Side-by-side
Showing with 951 additions and 61 deletions
examples/notebooks/retrieve_nni_info_with_python.ipynb 0 → 100644
View file @ 88ef6c04
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python wrapper for nni restful APIs\n",
"\n",
"nni provides nnicli module as a python wrapper for its restful APIs, which can be used to retrieve nni experiment and trial job information in your python code. This notebook shows how to use nnicli module.\n",
"\n",
"Following are the functions available in nnicli module:\n",
"\n",
"#### start_nni(config_file)\n",
"Starts nni experiment with specified configuration file\n",
"\n",
"#### stop_nni()\n",
"Stop nni experiment.\n",
"\n",
"#### set_endpoint(endpoint)\n",
"Set nni endpoint for nnicli, the endpoint is showed while nni experiment is started successfully using nnictl command or start_nni function\n",
"\n",
"#### version()\n",
"Returns nni version\n",
"\n",
"#### get_experiment_profile()\n",
"Returns experiment profile.\n",
"\n",
"#### get_experiment_status()\n",
"Returns nni experiment status.\n",
"\n",
"#### get_job_metrics(trial_job_id)\n",
"Returns specified trial job metrics, including final results and intermediate results.\n",
"\n",
"#### get_job_statistics()\n",
"Returns trial job statistics information\n",
"\n",
"#### get_trial_job(trial_job_id)\n",
"Returns information of a specified trial job.\n",
"\n",
"#### list_trial_jobs()\n",
"Returns information of all trial jobs of current experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Start nni experiment using specified configuration file\n",
"Let's use a configruation file in nni examples directory to start an experiment."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"authorName: default\r\n",
"experimentName: example_mnist\r\n",
"trialConcurrency: 1\r\n",
"maxExecDuration: 1h\r\n",
"maxTrialNum: 10\r\n",
"#choice: local, remote, pai\r\n",
"trainingServicePlatform: local\r\n",
"searchSpacePath: search_space.json\r\n",
"#choice: true, false\r\n",
"useAnnotation: false\r\n",
"tuner:\r\n",
" #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner\r\n",
" #SMAC (SMAC should be installed through nnictl)\r\n",
" builtinTunerName: TPE\r\n",
" classArgs:\r\n",
" #choice: maximize, minimize\r\n",
" optimize_mode: maximize\r\n",
"trial:\r\n",
" command: python3 mnist.py\r\n",
" codeDir: .\r\n",
" gpuNum: 0\r\n"
]
}
],
"source": [
"! cat ../trials/mnist/config.yml"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: expand searchSpacePath: search_space.json to /mnt/d/Repos/nni/examples/trials/mnist/search_space.json\n",
"INFO: expand codeDir: . to /mnt/d/Repos/nni/examples/trials/mnist/.\n",
"INFO: Starting restful server...\n",
"INFO: Successfully started Restful server!\n",
"INFO: Setting local config...\n",
"INFO: Successfully set local config!\n",
"INFO: Starting experiment...\n",
"INFO: Successfully started experiment!\n",
"-----------------------------------------------------------------------\n",
"The experiment id is PlUIfDTR\n",
"The Web UI urls are: http://172.18.17.1:8080 http://10.172.121.40:8080 http://10.0.75.1:8080 http://127.0.0.1:8080\n",
"-----------------------------------------------------------------------\n",
"\n",
"You can use these commands to get more information about the experiment\n",
"-----------------------------------------------------------------------\n",
"commands description\n",
"1. nnictl experiment show show the information of experiments\n",
"2. nnictl trial ls list all of trial jobs\n",
"3. nnictl top monitor the status of running experiments\n",
"4. nnictl log stderr show stderr log content\n",
"5. nnictl log stdout show stdout log content\n",
"6. nnictl stop stop an experiment\n",
"7. nnictl trial kill kill a trial job by id\n",
"8. nnictl --help get help information about nnictl\n",
"-----------------------------------------------------------------------\n",
"\n"
]
}
],
"source": [
"import nnicli as nc\n",
"nc.start_nni(config_file='../trials/mnist/config.yml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Connect nnicli module to started nni experiment\n",
"Call set_endpoint to connect nnicli moduele to the rest server of started nni experiment. Local mode training serviced is used in this notebook, but nnicli module can connect to any started nni experiment. The endpoint can be found in the output of start_nni function."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"nc.set_endpoint('http://127.0.0.1:8080')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Retrieve nni experiment and trial job information"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'errors': [], 'status': 'RUNNING'}"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_status()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'trialJobNumber': 4, 'trialJobStatus': 'SUCCEEDED'},\n",
" {'trialJobNumber': 1, 'trialJobStatus': 'RUNNING'}]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_job_statistics()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'execDuration': 1117,\n",
" 'id': 'PlUIfDTR',\n",
" 'logDir': '/home/chicm/nni/experiments/PlUIfDTR',\n",
" 'maxSequenceId': 3,\n",
" 'params': {'authorName': 'default',\n",
" 'clusterMetaData': [{'key': 'codeDir',\n",
" 'value': '/mnt/d/Repos/nni/examples/trials/mnist/.'},\n",
" {'key': 'command', 'value': 'python3 mnist.py'}],\n",
" 'experimentName': 'example_mnist',\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"hidden_size\": {\"_value\": [124, 512, 1024], \"_type\": \"choice\"}, \"batch_size\": {\"_value\": [1, 4, 8, 16, 32], \"_type\": \"choice\"}, \"conv_size\": {\"_value\": [2, 3, 5, 7], \"_type\": \"choice\"}, \"dropout_rate\": {\"_value\": [0.5, 0.9], \"_type\": \"uniform\"}, \"learning_rate\": {\"_value\": [0.0001, 0.001, 0.01, 0.1], \"_type\": \"choice\"}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'trialConcurrency': 1,\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'checkpointDir': '/home/chicm/nni/experiments/PlUIfDTR/checkpoint',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'className': 'TPE'},\n",
" 'versionCheck': True},\n",
" 'revision': 116,\n",
" 'startTime': 1564484985839}"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define an utility function to format json string returned by nnicli module."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"def show_json(res):\n",
" print(json.dumps(res, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"params\": {\n",
" \"searchSpace\": \"{\\\"hidden_size\\\": {\\\"_value\\\": [124, 512, 1024], \\\"_type\\\": \\\"choice\\\"}, \\\"batch_size\\\": {\\\"_value\\\": [1, 4, 8, 16, 32], \\\"_type\\\": \\\"choice\\\"}, \\\"conv_size\\\": {\\\"_value\\\": [2, 3, 5, 7], \\\"_type\\\": \\\"choice\\\"}, \\\"dropout_rate\\\": {\\\"_value\\\": [0.5, 0.9], \\\"_type\\\": \\\"uniform\\\"}, \\\"learning_rate\\\": {\\\"_value\\\": [0.0001, 0.001, 0.01, 0.1], \\\"_type\\\": \\\"choice\\\"}}\",\n",
" \"clusterMetaData\": [\n",
" {\n",
" \"key\": \"codeDir\",\n",
" \"value\": \"/mnt/d/Repos/nni/examples/trials/mnist/.\"\n",
" },\n",
" {\n",
" \"key\": \"command\",\n",
" \"value\": \"python3 mnist.py\"\n",
" }\n",
" ],\n",
" \"tuner\": {\n",
" \"classArgs\": {\n",
" \"optimize_mode\": \"maximize\"\n",
" },\n",
" \"builtinTunerName\": \"TPE\",\n",
" \"checkpointDir\": \"/home/chicm/nni/experiments/PlUIfDTR/checkpoint\",\n",
" \"className\": \"TPE\"\n",
" },\n",
" \"maxTrialNum\": 10,\n",
" \"maxExecDuration\": 3600,\n",
" \"experimentName\": \"example_mnist\",\n",
" \"authorName\": \"default\",\n",
" \"trialConcurrency\": 1,\n",
" \"trainingServicePlatform\": \"local\",\n",
" \"versionCheck\": true\n",
" },\n",
" \"execDuration\": 1192,\n",
" \"revision\": 124,\n",
" \"logDir\": \"/home/chicm/nni/experiments/PlUIfDTR\",\n",
" \"maxSequenceId\": 3,\n",
" \"id\": \"PlUIfDTR\",\n",
" \"startTime\": 1564484985839\n",
"}\n"
]
}
],
"source": [
"show_json(nc.get_experiment_profile())"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\n",
" {\n",
" \"startTime\": 1564484995992,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":0,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":1024,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.8055724367106529}}\"\n",
" ],\n",
" \"id\": \"BW0NR\",\n",
" \"endTime\": 1564485259753,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 0,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"0\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"BW0NR\",\n",
" \"timestamp\": 1564485258774,\n",
" \"data\": \"0.9078999757766724\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/BW0NR\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485271947,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":1,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":4,\\\"conv_size\\\":5,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.01,\\\"dropout_rate\\\":0.5547528540531742}}\"\n",
" ],\n",
" \"id\": \"x0P5w\",\n",
" \"endTime\": 1564485642784,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 1,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"1\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"x0P5w\",\n",
" \"timestamp\": 1564485642072,\n",
" \"data\": \"0.10100000351667404\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/x0P5w\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485652151,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":2,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.5584485925416655}}\"\n",
" ],\n",
" \"id\": \"V9jSG\",\n",
" \"endTime\": 1564485917057,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 2,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"2\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"V9jSG\",\n",
" \"timestamp\": 1564485916403,\n",
" \"data\": \"0.928600013256073\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/V9jSG\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485927295,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":3,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":7,\\\"hidden_size\\\":124,\\\"learning_rate\\\":0.001,\\\"dropout_rate\\\":0.6281630602835235}}\"\n",
" ],\n",
" \"id\": \"CDlRX\",\n",
" \"status\": \"RUNNING\",\n",
" \"sequenceId\": 3,\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/CDlRX\"\n",
" }\n",
"]\n"
]
}
],
"source": [
"show_json(nc.list_trial_jobs())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing nni experiment result\n",
"\n",
"With the retrieved trial job information, we can do some analysis by visualizing the metric data, below is a simple example."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3QAAAF4CAYAAAAczbvpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl8TNfj//F3MiQoGoQIsdTSVBFSitoiERIkoiFFPnSh0ZbSaq1dkBa1VKuWVqtaSz6WplJLpPiqvSUUReujVKkiIhJbQiWZ5PeHn/mYT0gGE8nl9Xw8PB6Ze889c85xM5P3vefe65CdnZ0tAAAAAIDhOBZ0AwAAAAAAd4ZABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgCAXcycOVOjRo2yqeyQIUM0ffr0m6777rvv9OKLL9pUT8+ePRUTE2NzG6OiovTUU0/J29tbly5dkre3t06dOmXz9rfy8ccfa8SIEXddz91q3bq14uPjC7oZAIB7qEhBNwAAUPh4e3tbfr5y5YqcnJxkMpkkSZGRkercuXOObQYMGGCX93766af19NNP26WuG129elUTJ05UTEyMateuLUnas2eP3d+nsPj444+VmJioCRMmFHRTAAD5iEAHAMjhxqDj5+ensWPHqnnz5rcsn5mZqSJFCvdXSlJSktLT0y1hrqAYYawAAMbBlEsAwG37+OOP9frrr+uNN96Qt7e3VqxYYTXtMCsrS4MGDVKLFi3UuHFj9e7dW0eOHLGp7ujoaPXu3dvy+ueff1ZoaKgaNWqkbt26ae/evVbl//rrL8v6AQMG6MKFCznqPHLkiIKCgiRdO/vYp08fZWZmytPTUydOnJB0bRro2LFj9eKLL8rb21vdu3fX33//banjvffeU+vWrfXEE0+oa9eu2r17t039+emnn+Tn56dZs2apRYsWevfddyVJP/zwgzp37qzGjRurZ8+eOnTokGWbWbNmqWXLlnriiScUGBhomUb5v1NVr9f9vzZs2KA5c+Zo5cqV8vb2VmhoqGVs/fz85O3trbZt22rVqlU29QEAUHgR6AAAd2TdunUKCgrSrl271LFjxxzr27RpozVr1ujHH39U7dq1NXTo0Nt+j5SUFL300kvq06eP4uPj1atXL/Xr188qtC1btkyTJk3Sli1blJ2drQ8++CBHPTVr1tTy5cslXTv7+NVXX930/VauXKnXXntNO3bskLu7uz755BPLOi8vL61YsUI7duxQQECAXnvtNaWnp9vUj9OnT+vy5cvasGGDRo8erX379undd9/V2LFjFR8fr65du6p///5KT0/X4cOHtWTJEn333XfavXu3Zs+erUqVKt3OsMnX11d9+/ZVcHCw9uzZo5iYGKWmpmrChAn66quvtGfPHi1atEienp63VS8AoPAh0AEA7sgTTzwhPz8/OTo6qlixYlbrHB0dFRoaqpIlS8rZ2VmvvvqqfvvtN12+fPm23mPDhg2qXbu2goKCVKRIEXXp0kUeHh7auHGjpUyXLl1Uq1YtlShRQoMGDdKqVauUnZ19R30KCAhQ/fr1VbRoUQUHB+vgwYNW7+Pi4qIiRYooIiJCqamp+uuvv2yq12Qy6dVXX5WTk5OKFSumb775RuHh4fLy8pLJZFK3bt0kSfv375fJZNLVq1f1xx9/KDMzU1WqVFGVKlXuqD//y8HBQYcPH9bVq1dVoUIF1apVyy71AgAKDpP4AQB3xN3d/ZbrzGazpkyZojVr1ujcuXNydLx2/PDcuXMqUaKEze9x5syZHGenKleurMTExJu2o3LlykpPT9f58+dVpkwZm9/nuvLly1t+Ll68uFUAnT17tpYuXaqkpCQ5ODjoypUrOnfunE31urq6ysnJyfL61KlTWrlypebOnWtZlpGRocTERDVq1EjDhw/XJ598oj///FMtW7bUyJEjrdp2J0qWLKkpU6bo66+/1siRI9WoUSONGDFCjzzyyF3VCwAoWJyhAwDcEQcHh1uuW7ZsmTZv3qx58+Zp165dWrt2rSTd9pmzChUq5HiswKlTp+Tm5mZ5nZCQYLXOyclJLi4ut/U+edm+fbvmzp2r6dOn6+eff9bOnTtVokQJm/vzv2NVsWJFDRgwQD///LPl3969ey1TV0NCQrR48WL98MMPMpvN+uijjyRJJUqU0JUrVyz1JCUl2fyekuTj46O5c+dq69atqlq1qs2PmQAAFF4EOgCA3aWlpVmC1ZUrVzR16tQ7qsfX11eHDx9WXFycMjMztXLlSh0/flxt2rSxlFm+fLmOHDmiy5cva/r06erQoUOuYfNOpKWlyWQyqUyZMsrIyND06dOtgtXteuaZZ7Rw4ULt27dP2dnZSktL0/r163X58mUdOXJE27dvV3p6upydnVWsWDFLfx577DFt2rRJFy5c0JkzZ7RgwYJbvoerq6tOnjxpCZ1nzpzR+vXrdeXKFRUtWlQlSpSwnDkFABgXn+QAALsLDQ1VhQoV1KpVKwUFBVk91+52lC1bVp999plmz56tpk2bau7cuZo1a5YefvhhS5mQkBANHTpULVu2lNls1ltvvWWvblj4+PioefPmat++vfz8/FSyZMm7mgLZsGFDjRkzRmPGjNGTTz6pgIAArVixQpKUnp6uyZMnq2nTpmrZsqUuXLigwYMHS7o2rjVr1pSvr69efPFFderU6Zbv0bFjR2VkZKhJkybq1q2bsrKyNGfOHLVs2VJNmzbV7t27OUMHAPcBh+w7vXIcAIB8sGTJEq1Zs+aWd6IEAAD/xRk6AEChcvjwYXl4eBR0MwAAMATucgkAKDReeuklnTp1yur5bwAA4NaYcgkAAAAABsWUSwAAAAAwKAIdAAAAABgUgQ4AAAAADKpQ3RTl3Lk0ZWVxSR8AAACAB4ujo4PKlHnotrcrVIEuKyubQAcAAAAANmLKJQAAAAAYFIEOAAAAAAyKQAcAAAAABpVnoJs4caL8/Pzk6empQ4cO3bSM2WxWZGSk/P391a5dO0VHR9u9oQAAAAAePEeOHFaHDm3VrJm3OnRoqz///CNHmcTERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZkqQNG35Qu3Y+8vBw1ejRb1vVmdu6wiTPQNe2bVv9+9//VuXKlW9ZZuXKlTp+/LjWrl2rJUuWaPr06Tpx4oRdGwoAAADgwTN06GD16ROh7dv3qE+fCA0Z8nqOMqNHj1SDBt7atGmbli9frfHj39PJk9fyyCeffKjatT21adM2bdy4Tfv2/aJVq1ZIkqpVq66PPpquAQMG5agzt3WFSZ6BrnHjxnJ3d8+1TFxcnMLCwuTo6KiyZcvK399fq1evtlsjAQAAADx4kpKStG/fXoWGhkmSQkPDtG/fXp09e9aq3G+//So/P39Jkqurq+rVq6/ly7+TJDk4OCg1NVVZWVm6evWq0tMzVLFiJUlSjRo1Vb++l0ymnDf/z21dYWKXa+gSEhJUqVIly2t3d3edPn3aHlUDAAAAeECdOnVC7u7uMplMkiSTyaSKFSvq1Cnr2YBeXg313XdLlZ2drb/+OqadO+N14sRxSdIbbwzTn3/+oXr1aqtevdry9W2rpk2b3fO+5JdCFTfLlStZ0E0AAAAAUEiUKfOQTCZHlS9fyrLMZHJUmTIPWS2bOXOaBg8erHbtWqlq1ary9/dXqVIlVL58KcXELFSjRt7avHmjLl26pA4dOmjTpjXq1q2bZfuHHnJWdnaGVZ22rCsM7BLo3N3dderUKXl5eUnKecbOVsnJqfn6YPEjRw7r1Vdf1rlzKSpTpqxmzvxcNWrUsiqTmJiooUNf019//aXMzAy9/voQhYX1sKxfvjxGH300SdnZ2XJwcFB09ApVqFBBkyaN19y5X8rN7dr01CZNmmrixI8s7/vmm6/pwoULSk+/qpCQUA0b9la+9RMAAAC4HxQvXkYnTpzU6dPnZTKZZDabdfLkKRUvXkZJSZduKFlMH3/8meVVz55d1axZKyUlXdLUqZ9o6tSZSk5Ok+Sotm0DFRe3Rj4+AZbyaWlXdfly+v/Umfc6e3J0dLijE1x2mXIZGBio6OhoZWVlKSUlRevWrVNAQEDeG95jd3tB5S+/7NbkyR/om2+Wa/PmeK1cuUalS5e2bBsW1lMbNvyoDRt+tIQ5SYqMHKXg4BBt2PCj1qzZqMWL/63du3/O/w4DAADY4G7vIihdO+jt49NMrVs3lY9PM505c0bStbuhDx/+hp580ktNmjRQVNQ8yzZTpkxUq1ZN5OPzlPz9W2v9+nX521EYTvny5VWvXn3FxFy7i35MTLTq1/eSq6urVbmUlGTLnSu3bNmk//zngOW6u6pVq1n2rfT0dG3evFGPPfb4PexF/soz0I0dO1atW7fW6dOn9cILL6hTp06SpIiICO3fv1+SFBISIg8PD7Vv317PPPOMBgwYoCpVquRvy2+TPS6onDVrpvr3HyQ3NzdJUunSD6tYsWJ5vreDg4MuXrwoSbpy5YocHBzk6lrebn2D8d3tF+mkSeP1+OM15OvbQr6+LTR8+BtWdXfp0lG+vi3UokVjTZo03rLu8uXLioh4Xk2aNFDz5o20du33+dtRAEChlJ8Hvb/9domOHv1T8fG/KC7uB02e/IGOH/9LkuTt3Uhr1mzUpk3bNHXqTPXr94KuXLly7zoOQ5g8earmzPlczZp5a86czzV58lRJ187C/fLLbknSnj271KJFYzVv3kgTJ47TggVLVKJECUnS++9P0Pbt2+Tj00x+fi1Us2ZN9e79vCRp+/ZtatDgMc2aNVPz53+tBg0es4S/3NYVJg7Z2dn5N8fxNuXnlMu9e/fo1Vdf0pYtOyzLWrZ8Up9+OlteXg0tywYM6Kdy5VwVGTlOx4//pYCANgoNDdP48ZPl59dS7dq117ZtPyktLU2dOgVr8OChcnBw0KRJ4/Xvf8+Xi0sZVahQQcOGvaUnn2wqSfr77+Pq1au7zp1L0YUL5zVq1Pvq27dfvvQTxhQaGqSePXspLKyHoqMXa9GiKMXExFqVefnlPqpd21NvvjlcZ8+eVbt2rRUbu1aVK3to0qTxSktLU2TkuBx1P/tsT/n4tFHfvi8pNTVVrVs31ZdfztMTTzTWhx9O0KlTJ/XRR9P1559/KDg4UPHxv6hkSa5nBYAHRVJSkp566gn9/vsxy5Q2T8/q2r59j9VZkFatmmjatM/k7d1IktS7d3c99VRL9e8/UC+/3FetW7dReHjvHPWHh3dTz569FBzcRZI0YsSb8vCoqldffc2qXHZ2tmrVqqItW+JVqdKtH5cF3K8KdMrl/SQycrySks7I17eF3n57mFq1amO5VanZbNaBA78pOnq5li+P0w8//J+++WaRJOm55/rq55/3a9OmbRow4DU991xPpaQkS5Lmz/9aYWE9tG/f79qxY6++/HKWdu3aWWB9ROFij7PHucntDPHy5TF69tkXJEk1atRSw4beWr/+/+zWNwBA4WePuwgeOvS7/vrrqDp3DlTbtq0s9xuQpJMnT8jD478ztypXrpKjbklasmShqld/hDAH3KZCdZfL/FSpkocSEhJkNpstR59Onz6tSpU8rMq5urrqs8++tLzu2bOrPD19JUkeHh4KCgqRs7OznJ2dFRjYSXv27FL37uGWaZiS1KaNnypV8tDBg/9R8+YtNXv2LO3cuU+S5OZWUS1b+mjbtp/UqNGT96DnKOxy+yK98cjo9S/Shg2f0PHjf2nnznhVqVLVsn7ZsqXauHF9jjPEY8dOUK9e3fX1119azhBXrVpNknTixAl5ePy3jsqVPXTy5Ml70W0AgMFERo7Xu++OkK9vC3l4eNzyoHdGRrq6dw9V5coe6t493Ka6f/ppqyZOHKfo6OX52QUUkNIuxeVc9IGJHbm6mpGpi+ftO634gRnZGy+oDAvrkesFlaVLP6wiRYpYLqicM2eBpGtnTtatW6tnnumpzMxMbdmy0TJ9ICHhlNzdr93Zc//+ffr7779Us2ZtSVK1atW0fv3/qXv3cKWmXlJ8/E8KDOxwD3uP+0FuX6TPPddXgwcPVdGiRbVx43o991xPbd26U2XLlrOcIX711deUmHhaXbp0VMOG3hxQAABIyv+D3pUre+jEib8tUzVPnvzb6mDizp3x6t8/QvPnL1KtWrXvQY9xrzkXLaI3vttU0M0oFD562sfudT5QUy7v9oLKp5/uJlfX8mrZ8kn5+bWQp2cd/etfz0qSxo2LVOvWTdWmTXO9+eZAzZz5heWs3bRpn2nevK/Upk1zBQb6qXPnp9W2bfsCGAEURjd+kUrK84t048afFBX1jVJTL8nT8zFJkpubm4oWLSrJ+gyxJM2ePctyhPTGM8TStS/g69NlpGvTYipXZqoLADxI7HEXwdDQMG3cuF7Z2dnKyMjQli0bVbdufUlS585Pa8GCucrKytLZs2f1/ferFBwcIuna3139+r2gOXPmW93TAIDtHpibogCFWZcuHfWvfz1ruSnKwoUL9N13q6zK/O/Z44EDX9ZPP+1SiRIlcpwh7tYtWJs375Cbm5t8fJqpf/9BljPEHTv6a/To99W2bXtNmjRep08nWG6KEhQUoB07flHJkoXzwZm2uttnTi5aFKVZs2bK0dFRWVlm9er1nCIiXslzu6SkJL322is6efKkMjMz1KJFK40fP1lFijwwkyEAGNThw4c0cOBLOn/+vFxcXDRjxheqVau2evbsquHD31bDhk/ohx/W6q23hslkMqls2XL64IMPVb/+tWcQZ2VlafTot7V+/f/J0dFRbdq0VWTkODk6OspsNmvEiCHauPEHSdLAgYMt12+3b++jv/8+rooV//v84pkzv9Djj9e994OAfFO+fCnO0P1/Hz3tc8vn2d3pTVEIdEAhcLdfpK+++pL27ftFjo4mOTkV1bBhb8nf/9qzIPfu3aORI4fq8uXLyszMUJcuXTVkyAhJUlpamgYNekX79++VyWTSqFHvq0OHTgU2DvZyt3cNvXTpokqWLCUHBwelpl5S69bNtGDBEtWtWy/X7d55Z7hMpiKKjBynjIwMBQe31yuvDFRISGgBjQQAAAWPQPdf+RHoDHPYuFTpYirmXLSgm1Eo/HM1Q5cu/lPQzYAd1a79qFav3pBj+aJFSy0/t23bXvHxN5+qO2PG57esu0EDb8XF3fyZKQ899JDmzJl/m60t3K7fNfT6hfWhoWEaOXKozp49azV96LffftVLLw2QZH3X0P79B6pUqdKWcpcvX1FGRoYcHBzy3O5aAExVVlaWrl69qvT0DKujzgCAB0N+zhS57o8/Dqtt25Z6/vkXLY8tio5erBkzPtGhQwc1duwE9e370r3pMAqUYQJdMeeiCh/274JuRqGwcNK/dEkEOuBm7HXX0NWr4zRu3BgdO3ZUb7892jL9J7ft3nhjmPr06a169Wrr8uXL6tu3n5o2bXYPew/gQfXww8Xk5MSBb0lKT8/QhQsF+3fS9Qe1X58pMmTI6zlmilx/UPv8+YstMz6aN2+pypU9FBTUWT16/Mtqpkjz5q1Ut249Sdeutx8y5LUcs2rq1fPSF198rWnTPrpnfUXBM0ygAwqz0g87y9nJqaCbUShcTU/XxQtXC7oZecrtrqGSFBjYUYGBHXXixN967rlw+fsHqFat2rlut2LFMj3+eF0tXbpSqamX1KNHV61cucxyN1wAyC9OTkU1ZcqUgm5GofDmm29KBXjgO79nikjStGkfqV27QKWlpSotLc2yvE6dxyVJjo4P1H0PH3gEOsAOnJ2c9PzXrxV0MwqFuS98IqngAp09br99Iw+PKvL2bqS1a1erVq3auW43Z87nmjr12hSZ0qUfVmBgJ23duplABwAPkPyeKfLrr/u1YcMP+u67VZoyZeK97RwKJeI7gPuKPW6/fejQ75ZyycnJ+vHHzZajnrltV7VqNa1ff+16xfT0dG3evFGPPfZ4PvYWAGBUkZHjlZR0Rr6+LfT228NuOlNky5Yd2rZtt6Kjl+iPPw4rIyNDQ4YM0uTJUy2BEeAMHYD7zuTJUzVw4EuaMmWi5a6hkqzuGrpnzy6ru4be+MzJBQu+1saN61WkSFFlZ2erb99+8vVtK0m5bvf++xM0dOhg+fg0k9lsVosWrdS79/MFMgYAgIKRnzNFOnfuomPHjio8vJsk6cKFC8rOzlZq6iVNmTItfzuGQotAB+C+c7d3DX3//Qm3rDu37R55pIa+/Xb5bbYWAHA/uXGmSFhYj1xnitz4fNn//OeA5sxZIOnaTJFHH/WU9N+ZIp06BcvDo4oOHjxmqWPSpPFKS0uz3OUSDyYCHQAAAGBH+TlTJDcxMdGKjHxXFy6c1+rVcZo27WN9880yeXo+lq/9RcEi0AEodFxKOaloMeeCbkahkPHPVZ2/lF7QzQAA3Ib8nClyo2HD3rJ6HRoaZrmuGw8OAh2AQqdoMWfFPftCQTejUOg4/2uJQAcAAG6BQAcAAADcoIyLs4oU5fmykpSZka5z5wv/82UfZAQ6AAAA4AZFijppc+yYgm5GodA6aIwK8vmyyBvPoQMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUEVsKXT06FGNGDFC58+fl4uLiyZOnKjq1atblUlOTtbIkSOVkJCgzMxMNW3aVO+8846KFLHpLQAAAAAAt8mmM3SjR49WeHi41qxZo/DwcI0aNSpHmVmzZqlmzZpauXKlVqxYod9++01r1661e4MBAAAAANfkGeiSk5N14MABBQUFSZKCgoJ04MABpaSkWJVzcHBQWlqasrKylJ6eroyMDLm5ueVPqwEAAAAAeQe6hIQEubm5yWQySZJMJpMqVKighIQEq3L9+/fX0aNH1bJlS8u/Ro0a5U+rAQAAAAC2XUNni9WrV8vT01Pz5s1TWlqaIiIitHr1agUGBtpcR7lyJe3VnPte+fKlCroJwC2xf9oX4wkAt4fPTftiPO3L3uOZZ6Bzd3dXYmKizGazTCaTzGazzpw5I3d3d6tyUVFRGj9+vBwdHVWqVCn5+fkpPj7+tgJdcnKqsrKyb7qOHclaUtKlgm4CbsD+ae1u90/G0xq/7wDywuemNb6H7IvxtK9bjaejo8MdneDKc8pluXLlVKdOHcXGxkqSYmNjVadOHZUtW9aqnIeHhzZv3ixJSk9P17Zt21S7du3bbhAAAAAAwDY23eVyzJgxioqKUkBAgKKiohQZGSlJioiI0P79+yVJb731lnbt2qXg4GB16dJF1atX1zPPPJN/LQcAAACAB5xN19DVrFlT0dHROZbPnj3b8nPVqlX19ddf269lAAAAAIBc2XSGDgAAAABQ+BDoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIOy6cHiAIAH05Ejh/Xqqy/r3LkUlSlTVjNnfq4aNWpZlRkwoJ8OHPjN8vrAgV81b94iBQZ21JQpE7Vs2VI5OppUtGhRvfXWKPn5+UuSunbtrJSUZEmS2Zypgwf/ow0bflLduvV05Mhhvfnma7pw4YLS068qJCRUw4a9de86DgCAQRDoAAC3NHToYPXpE6GwsB6Kjl6sIUNeV0xMrFWZmTO/sPz866/71bVrkHx920qSvL0b6ZVXBqpEiRL69df96tKlo/bvP6TixYtr6dIVlu3i4mI1YcL7qlu3niQpMnKUgoND1LfvS0pNTVXr1k3l799eTzzR+B70GgAA42DKJQDgppKSkrRv316FhoZJkkJDw7Rv316dPXv2ltssXDhfXbs+I2dnZ0mSn5+/SpQoIUmqW7eesrOzde5cSo7tFi1aoJ49e1teOzg46OLFi5KkK1euyMHBQa6u5e3WNwAA7hcEOgDATZ06dULu7u4ymUySJJPJpIoVK+rUqRM3LZ+enq6YmGirYHajJUsWqnr1R1SpUmWr5YmJidq8eaPCwnpYlo0dO0HLlsXIy8tTjRvXU//+g1S1ajU79QwAgPsHgQ4AYBfffx+rypWrqH59rxzrfvppqyZOHKfPP/8qx7pvvlkkX19/ubq6WpbNn/+1wsJ6aN++37Vjx159+eUs7dq1M1/bDwCAERHoAAA3VamShxISEmQ2myVJZrNZp0+fVqVKHjctv3DhAoWH98qxfOfOePXvH6F58xaqVq3aOdYvXhyVY7vZs2epe/dwSZKbW0W1bOmjbdt+utsuAQBw3yHQAQBuqnz58qpXr75iYqIlSTEx0apf38vqTNp1p06dVHz8NnXt+ozV8j17dqlfvxc0Z858eXk1zLHdjh3xunjxotq2bW+1vFq1alq//v8kSamplxQf/5Pq1Kljr64BAHDfINABAG5p8uSpmjPnczVr5q05cz7X5MlTJUk9e3bVL7/stpRbsmSh2rcPlItLGavthw9/Q//8c0VDhrwuX98W8vVtYfWIg8WLo/TMMz0t1+ldN23aZ5o37yu1adNcgYF+6tz56RyhDwAA8NgCAEAuatd+VKtXb8ixfNGipVavBw8eetPt167dlGv9H300/abLGzTwVlzcOhtbCQDAg4szdAAAAABgUJyhA4D72MOli8vJmY96SUq/mqkLF68UdDMAALArvuUB4D7m5FxE49/+tqCbUSi8Na5bQTcBAAC7Y8olAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMZtZ4OAAAgAElEQVQi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADMqmQHf06FF1795dAQEB6t69u44dO3bTcnFxcQoODlZQUJCCg4N19uxZe7YVAAAAAHCDIrYUGj16tMLDwxUSEqLly5dr1KhRmj9/vlWZ/fv3a8aMGZo3b57Kly+vS5cuycnJKV8aDQAAAACw4QxdcnKyDhw4oKCgIElSUFCQDhw4oJSUFKtyc+fOVZ8+fVS+fHlJUqlSpeTs7JwPTQYAAAAASDYEuoSEBLm5uclkMkmSTCaTKlSooISEBKtyR44c0d9//61//etfevrpp/Xpp58qOzs7f1oNAAAAALBtyqUtzGazfv/9d3399ddKT0/Xiy++qEqVKqlLly4211GuXEl7Nee+V758qYJuAnBL7J/2xXjaD2MJPBj4XbcvxtO+7D2eeQY6d3d3JSYmymw2y2QyyWw268yZM3J3d7cqV6lSJQUGBsrJyUlOTk5q27at9u3bd1uBLjk5VVlZNz+rx45kLSnpUkE3ATdg/7R2t/sn42ntbsaTsbTGZyfuV/yuW+N7yL4YT/u61Xg6Ojrc0QmuPKdclitXTnXq1FFsbKwkKTY2VnXq1FHZsmWtygUFBWnr1q3Kzs5WRkaGtm/frscee+y2GwQAAAAAsI1Njy0YM2aMoqKiFBAQoKioKEVGRkqSIiIitH//fklSp06dVK5cOXXs2FFdunRRrVq11K1bt/xrOQAAAAA84Gy6hq5mzZqKjo7OsXz27NmWnx0dHTVy5EiNHDnSfq0DAAAAANySTWfoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQNgW6o0ePqnv37goICFD37t117NixW5b9888/1aBBA02cONFebQQAAAAA3IRNgW706NEKDw/XmjVrFB4erlGjRt20nNls1ujRo+Xv72/XRgIAAAAAcsoz0CUnJ+vAgQMKCgqSJAUFBenAgQNKSUnJUfaLL75QmzZtVL16dbs3FAAAAABgLc9Al5CQIDc3N5lMJkmSyWRShQoVlJCQYFXu4MGD2rp1q55//vl8aSgAAAAAwFoRe1SSkZGhd999Vx988IEl+N2JcuVK2qM5D4Ty5UsVdBOAW2L/tC/G034YS+DBwO+6fTGe9mXv8cwz0Lm7uysxMVFms1kmk0lms1lnzpyRu7u7pUxSUpKOHz+ufv36SZIuXryo7Oxspaam6v3337e5McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5RnoypUrpzp16ig2NlYhISGKjY1VnTp1VLZsWUuZSpUqKT4+3vJ6+vTpunz5soYPH37bDQIAAAAA2Mamu1yOGTNGUVFRCggIUFRUlCIjIyVJERER2r9/f742EAAAAABwczZdQ1ezZk1FR0fnWD579uyblh84cODdtQoAAAAAkCebztABAAAAAAofAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBFbGl0NGjRzVixAidP39eLi4umjhxoqpXr25VZubMmYqLi5Ojo6OKFi2qwYMHq1WrVvnRZgAAAACAbAx0o0ePVnh4uEJCQrR8+XKNGjVK8+fPtyrj5eWlPn36qHjx4jp48KB69eqlrVu3qlixYvnScAAAAAB40OU55TI5OVkHDhxQUFCQJCkoKEgHDhxQSkqKVblWrVqpePHikiRPT09lZ2fr/Pnz+dBkAAAAAIBkQ6BLSEiQm5ubTCaTJMlkMqlChQpKSEi45TbLli1T1apVVbFiRfu1FAAAAABgxaYpl7djx44d+uSTT/TVV1/d9rblypW0d3PuW+XLlyroJgC3xP5pX4yn/TCWwIOB33X7Yjzty97jmWegc3d3V2Jiosxms0wmk8xms86cOSN3d/ccZffs2aOhQ4fq008/VY0aNW67McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5Tnlsly5cqpTp45iY2MlSbGxsapTp47Kli1rVW7fvn0aPHiwpk2bprp16952QwAAAAAAt8em59CNGTNGUVFRCggIUFRUlCIjIyVJERER2r9/vyQpMjJS//zzj0aNGqWQkBCFhITo999/z7+WAwAAAMADzqZr6GrWrKno6Ogcy2fPnm35eenSpfZrFQAAAAAgTzadoQMAAAAAFD4EOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHS4I0eOHFaHDm3VrJm3OnRoqz///CNHmQ0bflC7dj7y8HDV6NFvW60zm80aPvwNPfmkl5o0aaCoqHmWdYsWRcnH5yn5+raQj08zzZ79mWXdlCkT1apVE/n4PCV//9Zav35d/nUSAAAAKOQIdLgjQ4cOVp8+Edq+fY/69InQkCGv5yhTrVp1ffTRdA0YMCjHum+/XaKjR/9UfPwviov7QZMnf6Djx/+SJAUFddbGjT9pw4YftWrV/+mzz2bot99+lSR5ezfSmjUbtWnTNk2dOlP9+r2gK1eu5G9nAcBObDkYltsBr8TERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZedYJADA2Ah1uW1JSkvbt26vQ0DBJUmhomPbt26uzZ89alatRo6bq1/eSyVQkRx3Ll8eod+/n5ejoKFdXV3Xo0EkrViyTJJUqVVoODg6SpMuXrygjI8Py2s/PXyVKlJAk1a1bT9nZ2Tp3LiXf+goA9mTLwbDcDniNHj1SDRp4a9OmbVq+fLXGj39PJ0+ekCR98smHql3bU5s2bdPGjdu0b98vWrVqRZ51Gll+BuQBA/rJ17eF5Z+b28NavTrOqu4//jisatXccsxCAYB7iUCH23bq1Am5u7vLZDJJkkwmkypWrKhTp07YXMfJkyfk4VHF8rpy5SpW269eHadWrZqoUaO6GjBgkB5/vG6OOpYsWajq1R9RpUqV76I3AHBv2HowLLcDXr/99qv8/PwlSa6urqpXr76WL/9OkuTg4KDU1FRlZWXp6tWrSk/PUMWKlfKs08jyMyDPnPmFNmz4URs2/Kjp02fJxcVFvr5tLfWazWYNGfKaOnTodG86CwC3QKBDoRQY2FFbtuzQtm27FR29RH/8cdhq/U8/bdXEieP0+edfFVALAeD22HowLLcDXl5eDfXdd0uVnZ2tv/46pp0743XixHFJ0htvDNOff/6hevVqq1692vL1baumTZvlWadR5XdAvtHChfPVteszcnZ2tiybNu0jtWsXqBo1auVXFwHAJgQ63LZKlTyUkJAgs9ks6dpRytOnT6tSJQ+b66hc2UMnTvxteX3y5N833d7Do4q8vRtp7drVlmU7d8arf/8IzZu3ULVq1b6LngCAsURGjldS0hn5+rbQ228PU6tWbSzT2lesWKbHH6+rX389rH37Dmrbth+1cqXxz8LdSn4H5OvS09MVExOtnj17W5b9+ut+bdjwg15+eUB+dQ8AbEagw20rX7686tWrr5iYaElSTEy06tf3kqurq811dO78tBYsmKusrCydPXtW33+/SsHBIZKkQ4d+t5RLTk7Wjz9uVp06j0uS9uzZpX79XtCcOfPl5dXQjr0CgPxl68Gw3A54ubq66rPPvtTGjT8pKuobpaZekqfnY5KkOXM+V9euz8jR0VGlSz+swMBO2rp1c551PshyC8jXff99rCpXrqL69b0kSRkZGRoyZJAmT55qCZMAUJAIdLgjkydP1Zw5n6tZM2/NmfO5Jk+eKknq2bOrfvlltyRp+/ZtatDgMc2aNVPz53+tBg0eszxmICysh6pVe0RNmzZUhw5+evPN4apWrbokacGCr9WqVRP5+rZQ167B6tu3n+W6heHD39A//1zRkCGvWy5UP3Dgt3s/AABwm2w9GJbbAa+UlGTLnSu3bNmk//zngGXKYdWq1Syfsenp6dq8eaMee+zxPOs0qvwOyNctXLhA4eG9LK8TE0/r2LGjCg/vpkaN6umLLz5TVNQ8vflmzjs6A8C9kPP2g4ANatd+VKtXb8ixfNGipZafmzV7Snv3Hrzp9iaTSZMnf3zTde+/P+GW77t27abbbCkAFB6TJ0/VwIEvacqUiXJxcdGMGV9IunYwbPjwt9Ww4RMKC+uhXbt+VtOm12Yh3HjAa8+eXXrrrWEymUwqW7acFixYYrnz7/vvT9DQoYPl49NMZrNZLVq0Uu/ez0tSrnUa1Y0BOSysR54BuVOnzkpJSdH336/SihXXpvGnpCSrdOmHVaRIEUtAnjNngWXbU6dOKj5+m9X12h4eVXTw4DHL60mTxistLU2RkePyt8MAcAsEOgAA7hFbDobldsCrbdv2io9vf9N1jzxSQ99+u/ym63Kr08jyMyBL1+6m3L59oFxcytzzvgGArQh0D6gyDzupiJNz3gUfAJnpV3XuQnpBNwMAcJvyMyBL0uDBQ/Nsw7Bhb9nQUgDIPwS6B1QRJ2ftmvRiQTejUGg07EtJBDoAeXu4tJOcnDkYJknpV6/qwsW7++ws83BxFXHiTxFJykzP1LkLVwq6GQAMiE9RAABs5OTsrI9GvlTQzSgU3vjgc93twbAiTkW099ONdmmP0TXo36agmwDAoLjLJQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYlE2B7ujRo+revbsCAgLUvXt3HTt2LEcZs9msyMhI+fv7q127doqOjrZ3WwEAAAAAN7Ap0I0ePVrh4eFas2aNwsPDNWrUqBxlVq5cqePHj2vt2rVasmSJpk+frhMnTti9wQAAAACAa/IMdMnJyTpw4ICCgoIkSUFBQTpw4IBSUlKsysXFxSksLEyOjo4qW7as/P39tXr16vxpNQAAAABARfIqkJCQIDc3N5lMJkmSyWRShQoVlJCQoLJly1qVq1SpkuW1u7u7Tp8+fVuNcXR0yHW9a5mHbqu++1leY2ULp9Ll7NCS+4M9xtO1ZNm8Cz0g7DGexV3ZP6+72/F82KWEnVpifPbYN0u7sG9eZ4/xLFqqmB1acn+42/EsXbq0nVpifPbYN52Lu9ihJfcHe4xnmRLOdmjJ/eFW43mn4+yQnZ2dnVuBX3/9VcOHD9eqVassyzp27KjJkyerbt26lmXBwcEaN26cvLy8JEmzZ89WYmKi3nnnnTtqGAAAAAAgd3lOuXR3d1diYqLMZrOkazc/OXPmjNzd3XOUO3XqlOV1QkKCKlasaOfmAgAAAACuyzPQlStXTnXq1FFsbKwkKTY2VnXq1LGabilJgYGBio6OVlZWllJSUrRu3ToFBATkT6sBAAAAAHlPuZSkI0eOaMSIEbp48aJKly6tiRMnqkaNGoqIiNCgQYNUv359mc1mvffee/rxxx8lSREREerevXu+dwAAAAAAHlQ2BToAAAAAQOFj03PoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6Gy0bt06Pf300woMDJS/v78mTJig9PT0PLeLj4/X1q1b70ELC8aFCxfk5eWlsWPHWpbFxMRo0KBBOcreanleMjIyNH36dAUEBKhTp07q3LmzBg0apD/++OOu2l6YpKena8KECfL391dgYKC6dOmidevWSbq2D4WGhlqVP3TokPz8/HLUM2PGDHl6eurQoUNWy48cOSJPT0/NnTvXavnJkyfVr18/BQcHKzg4WKGhoTm2vZ/87ziHhITo+++/l3RtnBs0aKCQkBDLv2HDhlnW3fh/4OnpqbS0NElS7969tWHDhnvfmXvEz89PgYGB6ty5s9q1a6dXXnlFu3fvtqyfOXOm/P395e/vr5kzZ9pc77Zt2xQaGqpOnTqpU6dOOnjwYH403xCmT5+e5/dJ79691bZtW6v98/r/g5+f3y1/b3Nbdz+53s8RI0aodevWCgkJUUBAgD788ENLGU9PTwUHB6tz584KDg7WDz/8YFk3ffp0TZw48aZ1x8TEyNPTU3FxcVbL7uT77EF14/j+72dtWFhYAbfu3rrx+yMiIkLHjx+XJI0YMUJRUVGSru1fjRs3VkhIiDp27Kh+/fopKSnJqh6z2axWrVrplVdeubcdKGQSExPVu3dvSdL58+fVunVr7du3z7J+1qxZGjhwoK5cuaJ69erp7NmzlnWhoaFWv8f79++Xj4+PpGv7rKenp/bu3WtZn9vnREErUtANMIKdO3fqvffe0+zZs+Xp6amrV69q+PDhioyM1Lhx43LddseOHbp8+bJatmx5j1p7b8XGxqpBgwZatWqVhg0bJicnJ7u/x8iRI/XPP/8oOjpapUuXVnZ2tjZt2qSjR4+qVq1adn+/gjBmzBhdvnxZq1atkrOzsw4dOqQXX3xRDz/8sM11/Pbbb/rll19UuXLlHOu+/fZbNWvWTEuXLtXzzz9vWR4ZGanWrVurV69ekq59MBYpcv9+LNxsnPv27SsXFxc5OjqqZs2aiomJKehmFjrTpk3To48+Kklau3at+vXrpzlz5ig9PV2rV69WbGysJCksLExNmjTRk08+mWt9iYmJevvtt/Xll1+qRo0a+ueff5SZmZnv/SisZsyYoT59+uT5+fnOO+/I19f3HrXKuPr166devXrp0qVLCgkJkbe3t9q2bStJWrx4sR566CFt2rRJr7/+unbu3GnTZ17lypX1ySefqH379vf1Z+S9wmftNbNnz77luubNm2vatGnKzs7WG2+8oRkzZigyMtKyfsuWLapQoYJ2796ts2fPytXV9V40udBxc3PTggULJEkuLi4aNWqURo4cqe+++05Hjx7Vv//9by1btkzFixeXl5eXduzYoY4dOyo1NVX//POP1QGvHTt2qEmTJpbXlStX1pQpUzR//vx73q/b9cCdofv00081fvx4y+tz586padOmGjRokOXIiGR9pGT69Ol65ZVX5OnpKUlydnbWmDFjFBcXp5MnT+rIkSPy8fHRyZMnJV37ch48eLB+//13LV68WMuWLVNISIi++OKLe9jTe2Pp0qXq37+/PD09rY525iUhIUGhoaGWI55r1qyxnJmaNWuW5QjWsWPHtG7dOo0bN06lS5eWJDk4OKhNmzZq165dvvTpXjt58qS+//57jRkzRs7OzpKkRx99VC+//LJmzJhhUx3p6el67733NGbMmBzrMjMztXLlSr333nu6evWq1ZGr06dPy83NzfLazc1N5cqVu7sOFVK3GudXXnnF5nGG1L59e/Xo0UNz5sxRXFycunTpomLFiqlYsWLq0qWL5Xf60qVLGjRokAIDA9W7d28NHTrUcmRz4cKFCgkJUY0aNSRJxYoVU8mSJSVJrVq1UnJysqRrR6/79esnSUpOTlbr1q3vdXfz3fU/0Hr06KGQkBC1a9fOcuaiRYsWGjBggE31rFixQqGhoWrXrp3Vd9mDrFSpUqpfv76OHj2aY13Tpk11+fJlXbx40bIsMTFRERERCgwMVL9+/XTlyhXLunr16umRRx7Rt99+e0/aXhisXbvWMpNh5syZlu/lW31f3+rvq8uXLxdgLwo3W86gOzg46Mknn1RCQoLV8qVLl6pHjx7y9/fXsmXL8rOZhcbevXvVu3dvhYaGKjQ0VBs3btSJEyfUtGlTSxl/f3/VqVNHH374oUaMGKGRI0da/q5p0qSJ4uPjJUm7du1S48aNVb16dR0+fFjStUB3Y13t27fX+fPntWXLlnvYyzvzwAW6639wXD8aHBsbKz8/P5UoUeKW2/z+++9q2LCh1TIXFxdVqVJFhw4dUs2aNTV48GANHjxYW7du1cqVK/X+++/L09NTPXr0UJcuXbR8+XLLHyb3i4MH/1979x5TZf0HcPx9BghMEMXCJBHyBqHjYigHZxuB0yMUcEjLrHCRKIph0ooNEHA1WDlNKi6jDLUtGnIOHMfqlEXguMoERcd+GDZMUmSMWBgSxyO/P8545iPgJTPh8H1t/PPceJ4vj8/39vl8/R+9vb0olUoiIyPRaDT3fN6WLVtISkoiJCSE7u5uUlNTycvLo7S0FBsbG+nYlpYWXF1d72umaqI5f/48c+fOZfr06bLtPj4+UhjahQsXZKFWt4f6ZGVlERYWxpw5c0Zcv6KiAldXV1xdXVGr1bK/0+bNm3nvvfd47bXX2Lt3r6yzZ27uVM7DFert5Sw6eqPz9vamra2NK1eu4OzsLG2fPXu21OjIzs5m6tSp6PV6srKyaGhokI5ra2vjr7/+IioqioiICDIzM6WQQ39/f+rq6jAYDHR0dNDR0YHBYKC2tlZW0ZqLtLQ0wDRzpNPpOH78ODqdjtzcXGxtbYmJiZGO/eCDD2Tv53DHF0wdXq1WS2FhIXl5eZM6hHXY1atXaWxsxNPTc8S+48ePo1QqcXR0lLadO3eOffv28d1330kDYbfatWsXubm5DAwMPPR7f9S6u7vZvXs3OTk56HQ6afa4t7d3zPr6ftpX7e3tqNVq1q9fT0lJyX/zUBPU4OAgJ06cICQkRNrW09NDXV0da9euJTIyclLMdv7555+kpaWxb98+tFoteXl5pKam0tfXN+LY3bt3U1xczJNPPikrN39/f06ePAmYQn+XL1+On58fJ0+exGg0curUKVk9o1AoSEhI4OOPP2ZoaOjhP+QDmHQdOmdnZxYsWEBlZSUAJSUlI/KT/omIiAjmzZtHXFwc+/btk0abzVlxcTHh4eEoFApWr15Nc3MzV69eveM5ra2t7NixgwMHDuDn5weYRlw8PT1xc3MD4MUXXxzz/La2Nik34ta8vYnsXj4S8+fPR6fTST+ffPKJtK+pqYlz586xcePGUc/VaDSo1WrA9J7q9Xr+/vtvAMLCwigvL2fjxo1cv36dTZs2SeFz5uZO5axQKICR5bxjx47/6vYmlHt5Z+vr61m3bh0Ajo6Oshl1o9FIY2Mj2dnZFBUVceXKFSmCISAggJqaGs6cOYOPjw9eXl6cOXOGmpoalErlw3mgcaavr4/Y2Fjeffdd2WBiSkqK7P28dTZ9uKwfe+wxAgMDpUbLZJSfn094eDjbtm1j8+bNrFixQtq3YcMGgoKCSE5O5u2335adt3LlSqZNm4ZCocDLy0vKbRrm7u7OsmXLpPAuczZcLw/Por/88suAaUB2rPr6XttXixcvprKykpKSEvbv3092djY1NTUP+YkmnpqaGsLDwwkICKCnp4e1a9dK+44dO8Zzzz2HnZ0dzzzzDEajkaampkd4tw9fU1MTHR0dxMTEEB4eTkxMDAqFYtRw/draWuzs7Pj1119l+cm+vr50dHTQ3d1NQ0MDy5cvl2btWlpasLe3x8XFRXatwMBArK2tpXz78WrSdegA1Go1paWltLAYJCIAAAefSURBVLa20tfXh5+fHxYWFty8eVM6ZrjBC6aP+OnTp2XX6O3t5dKlSyxcuBAwjaD88ssv2Nvby0ZNzdXg4CBlZWVoNBqCgoIICQnBYDDcdZRo1qxZODg4SFPed+Pp6cnFixelsJgFCxag0+l4/fXXuXbt2gM/x3iwaNEifvvtN3p7e2XbT58+LYX53klDQwMXLlwgODiYoKAgOjs7efPNN6mqqqK7u5uqqio+++wzgoKCePXVV7l+/Trff/+9dP6MGTMICQkhNTWVbdu2mW2H7k7l7Ovr+4juamI6e/YsCxcuZPbs2Vy+fFnafuXKFWbPnn3X852dnQkMDMTe3p4pU6agUqk4e/YsAEqlktraWmpra1EqlSiVSurq6qirqyMgIOChPdN4YTAYeOutt1Cr1axZs+ZR386EtGXLFnQ6HVqtlk2bNsn2ffPNN5SXl7Nz504SEhJkdf1wKDaAhYUFRqNxxLV37tzJoUOHRp0VEEZvX93Ozs4Oe3t7AFxcXFi1apVsoSXBZMWKFeh0OiorK1EoFGRlZUn7NBoN1dXVBAUFERQURE9Pzz1HSU1UQ0NDuLu7ywa1KisrmTFjhuy4np4eMjIyyM/PZ8mSJbIBcBsbG7y9vfn555/p7+/HyckJT09PWlpaRuTP3eqdd94hKytrXOd6T8oO3erVq2loaKCgoAC1Wo1CocDV1VVqUHR1dck6HHFxceTm5tLa2gqYOnvp6emoVCopxO2jjz5i8eLFFBQUkJaWRmdnJ2D6cJnjh/+nn37iqaee4sSJE5SXl1NeXs6XX35519CJ6dOnc+jQIY4dO0ZBQQFgCt9qaWmRRkNvvYabmxvBwcGkpKTIytGcYvLnzJmDSqUiPT1dalycP3+evLy8e5oh2rJlC1VVVdLf4YknnuDgwYOsXLmS0tJS1qxZQ0VFhbQ/IyND+vBXVFRIv9NoNNLa2jpq2KY5GKucDx8+PGKkXhjbjz/+SGFhIdHR0ahUKkpLSxkYGGBgYIDS0lJpFFmpVEoDPH/88Ye0aivA888/T319PYODgwwNDVFVVYWHhwdgSkK3sLCgpKSEgIAAAgIC0Gq1WFpaysI7zcnUqVOlAarU1FTmzZvHG2+8cV/XGP5u9vT0UFlZaZbhqf+m6OhoZs6cSWFh4X2d5+Liwpo1azh8+PBDurPxwcfHh5aWFtrb2wE4evQoAB4eHmPW1zB6++p2XV1d0ix/b28v1dXV0r9/YSQ7Ozv27NlDYWEhXV1dNDc309fXJ6v3y8rK0Ov1srxPc+Pr68vFixepq6uTtjU3N4+IGNmzZw8vvfQSHh4eJCcnU1ZWJrXvwZRH98UXX7B06VIALC0tmTt3LkVFRWN+N4dz7W4Pwx5PJuVSTba2tgQHB6PVaqWFPNavX098fDwhISG4ubnh5eUlHe/v709KSgqJiYkMDAxgMBgIDg4mISEBMDVwTp48ydGjR7G2tiYuLo6EhASOHDkiJauGh4cTGhpqNnl0Go2GF154QbbN19eXmzdvcvnyZSorK2ULGERGRjJ37lzAlKh+8OBBYmNj6e/vJy4ujvT0dGJiYrC1tSUwMBArKytsbW0ByMzMJCcnh3Xr1mFpacm0adNwcnIym7IEUx7N/v37CQkJwcrKCmtra5KTk2UJvP+EVqslMTFRti04OJi0tDQ6Ojqor6/nww8/xNLSEqPRyJIlS9i5c+eDPs64dWs5KxQKrl69SlFREU8//TT19fVSDt0wJyenO65CNlnEx8czZcoUrl+/zvz588nPz8fb2xswNeBCQ0MBU0jv8Ajn9u3bSUpKQqVS8fjjj8tG6pcuXcqzzz5LREQEFhYWeHp6snXrVml/QEAAp06dwsnJCTCNqo420m8uoqOjiYqKwmAw8Pvvv7No0SLpPfT39ycpKQkw5dAdOHBAOi8+Pl5avXHGjBlERkbS19fH1q1b72l235zcuHFDNsN2NwqFgsTERHbt2sWGDRvu63dt377d7PO+Zs6cyfvvv09sbCw2NjasXr0aMA3K3qm+Hq19dbsffviBwsJCqd6JiIhg1apV/9mzjTfDnd47vcMeHh6oVCo+//xzBgcHCQ0NlXWWZ82ahaenJ3q9XkqxMDcODg7k5OSwd+9eMjIyMBgMuLi4sHv3bumYb7/9lvb2dum/K3FwcCA1NZWkpCQ0Gg1TpkzB39+f7Oxs2YJTy5Yto7q6eswZOjDl0I7nslUMjfcsP2FSuHbtmpR3qNFoKC4uvu+RU0G4HwaDgdTUVDo7O8nLy7uvxqBw/z799FP6+/tHDDAIwoPq6upi7dq1VFdXyxbpEP5d7u7uNDY2MjQ0JOrrf8GNGzfw9/dHr9fj6OiIWq0mJSXljp0KQRjLpJyhE8afr776Cr1ej9FoxMHBwWwWPBHGLysrKzIzMx/1bQiC8ACOHDnC119/TWJioujM/UdEff3guru7eeWVVwgLC+PSpUtERUXh5+cnOnPCPyZm6ARBEARBEARBECaoSbkoiiAIgiAIgiAIgjkQHTpBEARBEARBEIQJSnToBEEQBEEQBEEQJijRoRMEQRAEQRAEQZigRIdOEARBEARBEARhghIdOkEQBEEQBEEQhAnq/9qDRoCeJ1AFAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1080x432 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"sns.set(style=\"whitegrid\")\n",
"\n",
"jobs = nc.list_trial_jobs()\n",
"job_ids = [x['id'] for x in jobs]\n",
"final_metrics = [float(x['finalMetricData'][0]['data']) for x in jobs]\n",
"\n",
"data = {'job id': job_ids, 'final metrics': final_metrics}\n",
"sns.set(rc={'figure.figsize':(15, 6)})\n",
"\n",
"plt.title('Trial job final results')\n",
"ax = sns.barplot(x='job id', y='final metrics', data=data) \n",
"\n",
"for i,p in enumerate(ax.patches):\n",
" ax.annotate('{:.4f}'.format(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()),\n",
" ha='center', va='center', fontsize=11, color='black', rotation=0, xytext=(0, 5),\n",
" textcoords='offset points') "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stop nni experiment"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: Stoping experiment PlUIfDTR\n",
"INFO: Stop experiment success.\n"
]
}
],
"source": [
"nc.stop_nni()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/trials/auto-feature-engineering/README.md 0 → 100644
View file @ 88ef6c04
**Automatic Feature Engineering in nni**
===
Now we have an [example](https://github.com/SpongebBob/tabular_automl_NNI), which could automaticlly do feature engineering in nni.
These code come from our contributors. And thanks our lovely contributors!
And welcome more and more people to join us!
examples/tuners/random_nas_tuner/random_nas_tuner.py
View file @ 88ef6c04
......@@ -7,7 +7,9 @@ def random_archi_generator(nas_ss, random_state):
'''
chosen_archi = {}
print("zql: nas search space: ", nas_ss)
for block_name, block in nas_ss.items():
for block_name, block_value in nas_ss.items():
assert block_value['_type'] == "mutable_layer", "Random NAS Tuner only receives NAS search space whose _type is 'mutable_layer'"
block = block_value['_value']
tmp_block = {}
for layer_name, layer in block.items():
tmp_layer = {}
......
setup.py
View file @ 88ef6c04
......@@ -35,9 +35,10 @@ setup(
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('tools'),
packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('src/sdk/pycli') + find_packages('tools'),
package_dir = {
'nni': 'src/sdk/pynni/nni',
'nnicli': 'src/sdk/pycli/nnicli',
'nni_annotation': 'tools/nni_annotation',
'nni_cmd': 'tools/nni_cmd',
'nni_trial_tool':'tools/nni_trial_tool',
......
src/nni_manager/rest_server/restValidationSchemas.ts
View file @ 88ef6c04
......@@ -51,10 +51,12 @@ export namespace ValidationSchemas {
command: joi.string().min(1),
virtualCluster: joi.string(),
shmMB: joi.number(),
authFile: joi.string(),
nasMode: joi.string().valid('classic_mode', 'enas_mode', 'oneshot_mode'),
worker: joi.object({
replicas: joi.number().min(1).required(),
image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(),
cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100),
......@@ -64,6 +66,7 @@ export namespace ValidationSchemas {
ps: joi.object({
replicas: joi.number().min(1).required(),
image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(),
cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100),
......@@ -73,6 +76,7 @@ export namespace ValidationSchemas {
master: joi.object({
replicas: joi.number().min(1).required(),
image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(),
cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100),
......@@ -83,6 +87,7 @@ export namespace ValidationSchemas {
name: joi.string().min(1),
taskNum: joi.number().min(1).required(),
image: joi.string().min(1),
privateRegistryAuthPath: joi.string().min(1),
outputDir: joi.string(),
cpuNum: joi.number().min(1),
memoryMB: joi.number().min(100),
......
src/nni_manager/training_service/kubernetes/frameworkcontroller/frameworkcontrollerConfig.ts
View file @ 88ef6c04
......@@ -43,8 +43,8 @@ export class FrameworkControllerTrialConfigTemplate extends KubernetesTrialConfi
public readonly taskNum: number;
constructor(taskNum: number, command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string,
frameworkAttemptCompletionPolicy: FrameworkAttemptCompletionPolicy) {
super(command, gpuNum, cpuNum, memoryMB, image);
frameworkAttemptCompletionPolicy: FrameworkAttemptCompletionPolicy, privateRegistryFilePath?: string | undefined) {
super(command, gpuNum, cpuNum, memoryMB, image, privateRegistryFilePath);
this.frameworkAttemptCompletionPolicy = frameworkAttemptCompletionPolicy;
this.name = name;
this.taskNum = taskNum;
......
src/nni_manager/training_service/kubernetes/frameworkcontroller/frameworkcontrollerTrainingService.ts
View file @ 88ef6c04
......@@ -305,7 +305,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
}
// Generate frameworkcontroller job resource config object
const frameworkcontrollerJobConfig: any =
this.generateFrameworkControllerJobConfig(trialJobId, trialWorkingFolder, frameworkcontrollerJobName, podResources);
await this.generateFrameworkControllerJobConfig(trialJobId, trialWorkingFolder, frameworkcontrollerJobName, podResources);
return Promise.resolve(frameworkcontrollerJobConfig);
}
......@@ -329,8 +329,8 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
* @param frameworkcontrollerJobName job name
* @param podResources pod template
*/
private generateFrameworkControllerJobConfig(trialJobId: string, trialWorkingFolder: string,
frameworkcontrollerJobName : string, podResources : any) : any {
private async generateFrameworkControllerJobConfig(trialJobId: string, trialWorkingFolder: string,
frameworkcontrollerJobName : string, podResources : any) : Promise<any> {
if (this.fcClusterConfig === undefined) {
throw new Error('frameworkcontroller Cluster config is not initialized');
}
......@@ -345,12 +345,14 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
if (containerPort === undefined) {
throw new Error('Container port is not initialized');
}
const taskRole: any = this.generateTaskRoleConfig(
trialWorkingFolder,
this.fcTrialConfig.taskRoles[index].image,
`run_${this.fcTrialConfig.taskRoles[index].name}.sh`,
podResources[index],
containerPort
containerPort,
await this.createRegistrySecret(this.fcTrialConfig.taskRoles[index].privateRegistryAuthPath)
);
taskRoles.push({
name: this.fcTrialConfig.taskRoles[index].name,
......@@ -363,7 +365,7 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
});
}
return {
return Promise.resolve({
apiVersion: `frameworkcontroller.microsoft.com/v1`,
kind: 'Framework',
metadata: {
......@@ -379,11 +381,11 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
executionType: 'Start',
taskRoles: taskRoles
}
};
});
}
private generateTaskRoleConfig(trialWorkingFolder: string, replicaImage: string, runScriptFile: string,
podResources: any, containerPort: number): any {
private generateTaskRoleConfig(trialWorkingFolder: string, replicaImage: string, runScriptFile: string,
podResources: any, containerPort: number, privateRegistrySecretName: string | undefined): any {
if (this.fcClusterConfig === undefined) {
throw new Error('frameworkcontroller Cluster config is not initialized');
}
......@@ -451,13 +453,22 @@ class FrameworkControllerTrainingService extends KubernetesTrainingService imple
mountPath: '/mnt/frameworkbarrier'
}]
}];
const spec: any = {
containers: containers,
initContainers: initContainers,
restartPolicy: 'OnFailure',
volumes: volumeSpecMap.get('nniVolumes'),
hostNetwork: false
let spec: any = {
containers: containers,
initContainers: initContainers,
restartPolicy: 'OnFailure',
volumes: volumeSpecMap.get('nniVolumes'),
hostNetwork: false
};
if(privateRegistrySecretName) {
spec.imagePullSecrets = [
{
name: privateRegistrySecretName
}
]
}
if (this.fcClusterConfig.serviceAccountName !== undefined) {
spec.serviceAccountName = this.fcClusterConfig.serviceAccountName;
}
......
src/nni_manager/training_service/kubernetes/kubeflow/kubeflowConfig.ts
View file @ 88ef6c04
......@@ -135,8 +135,8 @@ export class KubeflowTrialConfig extends KubernetesTrialConfig {
export class KubeflowTrialConfigTemplate extends KubernetesTrialConfigTemplate {
public readonly replicas: number;
constructor(replicas: number, command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string) {
super(command, gpuNum, cpuNum, memoryMB, image);
cpuNum: number, memoryMB: number, image: string, privateRegistryAuthPath?: string) {
super(command, gpuNum, cpuNum, memoryMB, image, privateRegistryAuthPath);
this.replicas = replicas;
}
}
......
src/nni_manager/training_service/kubernetes/kubeflow/kubeflowTrainingService.ts
View file @ 88ef6c04
......@@ -347,7 +347,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
}
// Generate kubeflow job resource config object
const kubeflowJobConfig: any = this.generateKubeflowJobConfig(trialJobId, trialWorkingFolder, kubeflowJobName, workerPodResources,
const kubeflowJobConfig: any = await this.generateKubeflowJobConfig(trialJobId, trialWorkingFolder, kubeflowJobName, workerPodResources,
nonWorkerResources);
return Promise.resolve(kubeflowJobConfig);
......@@ -361,8 +361,8 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
* @param workerPodResources worker pod template
* @param nonWorkerPodResources non-worker pod template, like ps or master
*/
private generateKubeflowJobConfig(trialJobId: string, trialWorkingFolder: string, kubeflowJobName : string, workerPodResources : any,
nonWorkerPodResources?: any) : any {
private async generateKubeflowJobConfig(trialJobId: string, trialWorkingFolder: string, kubeflowJobName : string, workerPodResources : any,
nonWorkerPodResources?: any) : Promise<any> {
if (this.kubeflowClusterConfig === undefined) {
throw new Error('Kubeflow Cluster config is not initialized');
}
......@@ -377,29 +377,32 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
const replicaSpecsObj: any = {};
const replicaSpecsObjMap: Map<string, object> = new Map<string, object>();
if (this.kubeflowTrialConfig.operatorType === 'tf-operator') {
const tensorflowTrialConfig: KubeflowTrialConfigTensorflow = <KubeflowTrialConfigTensorflow>this.kubeflowTrialConfig;
let privateRegistrySecretName = await this.createRegistrySecret(tensorflowTrialConfig.worker.privateRegistryAuthPath);
replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.worker.replicas,
tensorflowTrialConfig.worker.image, 'run_worker.sh', workerPodResources);
tensorflowTrialConfig.worker.image, 'run_worker.sh', workerPodResources, privateRegistrySecretName);
if (tensorflowTrialConfig.ps !== undefined) {
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(tensorflowTrialConfig.ps.privateRegistryAuthPath);
replicaSpecsObj.Ps = this.generateReplicaConfig(trialWorkingFolder, tensorflowTrialConfig.ps.replicas,
tensorflowTrialConfig.ps.image, 'run_ps.sh', nonWorkerPodResources);
tensorflowTrialConfig.ps.image, 'run_ps.sh', nonWorkerPodResources, privateRegistrySecretName);
}
replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {tfReplicaSpecs: replicaSpecsObj});
} else if (this.kubeflowTrialConfig.operatorType === 'pytorch-operator') {
const pytorchTrialConfig: KubeflowTrialConfigPytorch = <KubeflowTrialConfigPytorch>this.kubeflowTrialConfig;
if (pytorchTrialConfig.worker !== undefined) {
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(pytorchTrialConfig.worker.privateRegistryAuthPath);
replicaSpecsObj.Worker = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.worker.replicas,
pytorchTrialConfig.worker.image, 'run_worker.sh', workerPodResources);
pytorchTrialConfig.worker.image, 'run_worker.sh', workerPodResources, privateRegistrySecretName);
}
let privateRegistrySecretName: string | undefined = await this.createRegistrySecret(pytorchTrialConfig.master.privateRegistryAuthPath);
replicaSpecsObj.Master = this.generateReplicaConfig(trialWorkingFolder, pytorchTrialConfig.master.replicas,
pytorchTrialConfig.master.image, 'run_master.sh', nonWorkerPodResources);
pytorchTrialConfig.master.image, 'run_master.sh', nonWorkerPodResources, privateRegistrySecretName);
replicaSpecsObjMap.set(this.kubernetesCRDClient.jobKind, {pytorchReplicaSpecs: replicaSpecsObj});
}
return {
return Promise.resolve({
apiVersion: `kubeflow.org/${this.kubernetesCRDClient.apiVersion}`,
kind: this.kubernetesCRDClient.jobKind,
metadata: {
......@@ -412,7 +415,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
}
},
spec: replicaSpecsObjMap.get(this.kubernetesCRDClient.jobKind)
};
});
}
/**
......@@ -424,7 +427,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
* @param podResources pod resource config section
*/
private generateReplicaConfig(trialWorkingFolder: string, replicaNumber: number, replicaImage: string, runScriptFile: string,
podResources: any): any {
podResources: any, privateRegistrySecretName: string | undefined): any {
if (this.kubeflowClusterConfig === undefined) {
throw new Error('Kubeflow Cluster config is not initialized');
}
......@@ -436,7 +439,7 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
if (this.kubernetesCRDClient === undefined) {
throw new Error('Kubeflow operator client is not initialized');
}
// The config spec for volume field
const volumeSpecMap: Map<string, object> = new Map<string, object>();
if (this.kubeflowClusterConfig.storageType === 'azureStorage') {
volumeSpecMap.set('nniVolumes', [
......@@ -459,7 +462,34 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
}
}]);
}
// The config spec for container field
const containersSpecMap: Map<string, object> = new Map<string, object>();
containersSpecMap.set('containers', [
{
// Kubeflow tensorflow operator requires that containers' name must be tensorflow
// TODO: change the name based on operator's type
name: this.kubernetesCRDClient.containerName,
image: replicaImage,
args: ['sh', `${path.join(trialWorkingFolder, runScriptFile)}`],
volumeMounts: [
{
name: 'nni-vol',
mountPath: this.CONTAINER_MOUNT_PATH
}],
resources: podResources
}
]);
let spec: any = {
containers: containersSpecMap.get('containers'),
restartPolicy: 'ExitCode',
volumes: volumeSpecMap.get('nniVolumes')
}
if (privateRegistrySecretName) {
spec.imagePullSecrets = [
{
name: privateRegistrySecretName
}]
}
return {
replicas: replicaNumber,
template: {
......@@ -467,26 +497,9 @@ class KubeflowTrainingService extends KubernetesTrainingService implements Kuber
// tslint:disable-next-line:no-null-keyword
creationTimestamp: null
},
spec: {
containers: [
{
// Kubeflow tensorflow operator requires that containers' name must be tensorflow
// TODO: change the name based on operator's type
name: this.kubernetesCRDClient.containerName,
image: replicaImage,
args: ['sh', `${path.join(trialWorkingFolder, runScriptFile)}`],
volumeMounts: [
{
name: 'nni-vol',
mountPath: this.CONTAINER_MOUNT_PATH
}],
resources: podResources
}],
restartPolicy: 'ExitCode',
volumes: volumeSpecMap.get('nniVolumes')
}
spec: spec
}
};
}
}
}
// tslint:enable: no-unsafe-any no-any
......
src/nni_manager/training_service/kubernetes/kubernetesConfig.ts
View file @ 88ef6c04
......@@ -179,6 +179,9 @@ export class KubernetesTrialConfigTemplate {
// Docker image
public readonly image: string;
// Private registry config file path to download docker iamge
public readonly privateRegistryAuthPath?: string;
// Trail command
public readonly command : string;
......@@ -186,12 +189,13 @@ export class KubernetesTrialConfigTemplate {
public readonly gpuNum : number;
constructor(command : string, gpuNum : number,
cpuNum: number, memoryMB: number, image: string) {
cpuNum: number, memoryMB: number, image: string, privateRegistryAuthPath?: string) {
this.command = command;
this.gpuNum = gpuNum;
this.cpuNum = cpuNum;
this.memoryMB = memoryMB;
this.image = image;
this.privateRegistryAuthPath = privateRegistryAuthPath;
}
}
......
src/nni_manager/training_service/kubernetes/kubernetesTrainingService.ts
View file @ 88ef6c04
......@@ -38,6 +38,8 @@ import { KubernetesClusterConfig } from './kubernetesConfig';
import { kubernetesScriptFormat, KubernetesTrialJobDetail } from './kubernetesData';
import { KubernetesJobRestServer } from './kubernetesJobRestServer';
var fs = require('fs');
/**
* Training Service implementation for Kubernetes
*/
......@@ -327,5 +329,34 @@ abstract class KubernetesTrainingService {
return Promise.resolve();
}
protected async createRegistrySecret(filePath: string | undefined): Promise<string | undefined> {
if(filePath === undefined || filePath === '') {
return undefined;
}
let body = fs.readFileSync(filePath).toString('base64');
let registrySecretName = String.Format('nni-secret-{0}', uniqueString(8)
.toLowerCase());
await this.genericK8sClient.createSecret(
{
apiVersion: 'v1',
kind: 'Secret',
metadata: {
name: registrySecretName,
namespace: 'default',
labels: {
app: this.NNI_KUBERNETES_TRIAL_LABEL,
expId: getExperimentId()
}
},
type: 'kubernetes.io/dockerconfigjson',
data: {
'.dockerconfigjson': body
}
}
);
return registrySecretName;
}
}
export { KubernetesTrainingService };
src/nni_manager/training_service/pai/paiConfig.ts
View file @ 88ef6c04
......@@ -71,6 +71,8 @@ export class PAIJobConfig {
public readonly image: string;
// Code directory on HDFS
public readonly codeDir: string;
//authentication file used for private Docker registry
public readonly authFile?: string;
// List of taskRole, one task role at least
public taskRoles: PAITaskRole[];
......@@ -87,12 +89,13 @@ export class PAIJobConfig {
* @param taskRoles List of taskRole, one task role at least
*/
constructor(jobName: string, image : string, codeDir : string,
taskRoles : PAITaskRole[], virtualCluster: string) {
taskRoles : PAITaskRole[], virtualCluster: string, authFile?: string) {
this.jobName = jobName;
this.image = image;
this.codeDir = codeDir;
this.taskRoles = taskRoles;
this.virtualCluster = virtualCluster;
this.authFile = authFile;
}
}
......@@ -129,14 +132,17 @@ export class NNIPAITrialConfig extends TrialConfig {
public virtualCluster?: string;
//Shared memory for one task in the task role
public shmMB?: number;
//authentication file used for private Docker registry
public authFile?: string;
constructor(command : string, codeDir : string, gpuNum : number, cpuNum: number, memoryMB: number,
image: string, virtualCluster?: string, shmMB?: number) {
image: string, virtualCluster?: string, shmMB?: number, authFile?: string) {
super(command, codeDir, gpuNum);
this.cpuNum = cpuNum;
this.memoryMB = memoryMB;
this.image = image;
this.virtualCluster = virtualCluster;
this.shmMB = shmMB;
this.authFile = authFile;
}
}
src/nni_manager/training_service/pai/paiTrainingService.ts
View file @ 88ef6c04
......@@ -442,7 +442,7 @@ class PAITrainingService implements TrainingService {
// Task command
nniPaiTrialCommand,
// Task shared memory
this.paiTrialConfig.shmMB
this.paiTrialConfig.shmMB,
)
];
......@@ -456,7 +456,9 @@ class PAITrainingService implements TrainingService {
// PAI Task roles
paiTaskRoles,
// Add Virutal Cluster
this.paiTrialConfig.virtualCluster === undefined ? 'default' : this.paiTrialConfig.virtualCluster.toString()
this.paiTrialConfig.virtualCluster === undefined ? 'default' : this.paiTrialConfig.virtualCluster.toString(),
//Task auth File
this.paiTrialConfig.authFile
);
// Step 2. Upload code files in codeDir onto HDFS
......
src/sdk/pycli/nnicli/__init__.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
from .nni_client import *
src/sdk/pycli/nnicli/nni_client.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
""" A python wrapper for nni rest api
Example:
import nnicli as nc
nc.start_nni('../../../../examples/trials/mnist/config.yml')
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_experiment_status())
print(nc.get_job_statistics())
print(nc.list_trial_jobs())
nc.stop_nni()
"""
import sys
import os
import subprocess
import requests
__all__ = [
'start_nni',
'stop_nni',
'set_endpoint',
'version',
'get_experiment_status',
'get_experiment_profile',
'get_trial_job',
'list_trial_jobs',
'get_job_statistics',
'get_job_metrics',
'export_data'
]
EXPERIMENT_PATH = 'experiment'
VERSION_PATH = 'version'
STATUS_PATH = 'check-status'
JOB_STATISTICS_PATH = 'job-statistics'
TRIAL_JOBS_PATH = 'trial-jobs'
METRICS_PATH = 'metric-data'
EXPORT_DATA_PATH = 'export-data'
API_ROOT_PATH = 'api/v1/nni'
_api_endpoint = None
def set_endpoint(endpoint):
"""set endpoint of nni rest server for nnicli, for example:
http://localhost:8080
"""
global _api_endpoint
_api_endpoint = endpoint
def _check_endpoint():
if _api_endpoint is None:
raise AssertionError("Please call set_endpoint to specify nni endpoint")
def _nni_rest_get(api_path, response_type='json'):
_check_endpoint()
uri = '{}/{}/{}'.format(_api_endpoint, API_ROOT_PATH, api_path)
res = requests.get(uri)
if _http_succeed(res.status_code):
if response_type == 'json':
return res.json()
elif response_type == 'text':
return res.text
else:
raise AssertionError('Incorrect response_type')
else:
return None
def _http_succeed(status_code):
return status_code // 100 == 2
def _create_process(cmd):
if sys.platform == 'win32':
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
else:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while process.poll() is None:
output = process.stdout.readline()
if output:
print(output.decode('utf-8').strip())
return process.returncode
def start_nni(config_file):
"""start nni experiment with specified configuration file"""
cmd = 'nnictl create --config {}'.format(config_file).split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to start nni.')
def stop_nni():
"""stop nni experiment"""
cmd = 'nnictl stop'.split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to stop nni.')
def version():
"""return version of nni"""
return _nni_rest_get(VERSION_PATH, 'text')
def get_experiment_status():
"""return experiment status as a dict"""
return _nni_rest_get(STATUS_PATH)
def get_experiment_profile():
"""return experiment profile as a dict"""
return _nni_rest_get(EXPERIMENT_PATH)
def get_trial_job(trial_job_id):
"""return trial job information as a dict"""
assert trial_job_id is not None
return _nni_rest_get(os.path.join(TRIAL_JOBS_PATH, trial_job_id))
def list_trial_jobs():
"""return information for all trial jobs as a list"""
return _nni_rest_get(TRIAL_JOBS_PATH)
def get_job_statistics():
"""return trial job statistics information as a dict"""
return _nni_rest_get(JOB_STATISTICS_PATH)
def get_job_metrics(trial_job_id=None):
"""return trial job metrics"""
api_path = METRICS_PATH if trial_job_id is None else os.path.join(METRICS_PATH, trial_job_id)
return _nni_rest_get(api_path)
def export_data():
"""return exported information for all trial jobs"""
return _nni_rest_get(EXPORT_DATA_PATH)
src/sdk/pycli/setup.py 0 → 100644
View file @ 88ef6c04
import setuptools
setuptools.setup(
name = 'nnicli',
version = '999.0.0-developing',
packages = setuptools.find_packages(),
python_requires = '>=3.5',
install_requires = [
'requests'
],
author = 'Microsoft NNI Team',
author_email = 'nni@microsoft.com',
description = 'nnicli for Neural Network Intelligence project',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
)
src/sdk/pynni/nni/hyperopt_tuner/hyperopt_tuner.py
View file @ 88ef6c04
......@@ -190,13 +190,19 @@ class HyperoptTuner(Tuner):
HyperoptTuner is a tuner which using hyperopt algorithm.
"""
def __init__(self, algorithm_name, optimize_mode='minimize'):
def __init__(self, algorithm_name, optimize_mode='minimize',
parallel_optimize=False, constant_liar_type='min'):
"""
Parameters
----------
algorithm_name : str
algorithm_name includes "tpe", "random_search" and anneal".
optimize_mode : str
parallel_optimize : bool
More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
constant_liar_type : str
constant_liar_type including "min", "max" and "mean"
More detail could reference: docs/en_US/Tuner/HyperoptTuner.md
"""
self.algorithm_name = algorithm_name
self.optimize_mode = OptimizeMode(optimize_mode)
......@@ -205,6 +211,13 @@ class HyperoptTuner(Tuner):
self.rval = None
self.supplement_data_num = 0
self.parallel = parallel_optimize
if self.parallel:
self.CL_rval = None
self.constant_liar_type = constant_liar_type
self.running_data = []
self.optimal_y = None
def _choose_tuner(self, algorithm_name):
"""
Parameters
......@@ -266,6 +279,10 @@ class HyperoptTuner(Tuner):
# but it can cause deplicate parameter rarely
total_params = self.get_suggestion(random_search=True)
self.total_data[parameter_id] = total_params
if self.parallel:
self.running_data.append(parameter_id)
params = split_index(total_params)
return params
......@@ -287,10 +304,39 @@ class HyperoptTuner(Tuner):
raise RuntimeError('Received parameter_id not in total_data.')
params = self.total_data[parameter_id]
# code for parallel
if self.parallel:
constant_liar = kwargs.get('constant_liar', False)
if constant_liar:
rval = self.CL_rval
else:
rval = self.rval
self.running_data.remove(parameter_id)
# update the reward of optimal_y
if self.optimal_y is None:
if self.constant_liar_type == 'mean':
self.optimal_y = [reward, 1]
else:
self.optimal_y = reward
else:
if self.constant_liar_type == 'mean':
_sum = self.optimal_y[0] + reward
_number = self.optimal_y[1] + 1
self.optimal_y = [_sum, _number]
elif self.constant_liar_type == 'min':
self.optimal_y = min(self.optimal_y, reward)
elif self.constant_liar_type == 'max':
self.optimal_y = max(self.optimal_y, reward)
logger.debug("Update optimal_y with reward, optimal_y = %s", self.optimal_y)
else:
rval = self.rval
if self.optimize_mode is OptimizeMode.Maximize:
reward = -reward
rval = self.rval
domain = rval.domain
trials = rval.trials
......@@ -375,13 +421,26 @@ class HyperoptTuner(Tuner):
total_params : dict
parameter suggestion
"""
if self.parallel and len(self.total_data)>20 and len(self.running_data) and self.optimal_y is not None:
self.CL_rval = copy.deepcopy(self.rval)
if self.constant_liar_type == 'mean':
_constant_liar_y = self.optimal_y[0] / self.optimal_y[1]
else:
_constant_liar_y = self.optimal_y
for _parameter_id in self.running_data:
self.receive_trial_result(parameter_id=_parameter_id, parameters=None, value=_constant_liar_y, constant_liar=True)
rval = self.CL_rval
rval = self.rval
random_state = np.random.randint(2**31 - 1)
else:
rval = self.rval
random_state = rval.rstate.randint(2**31 - 1)
trials = rval.trials
algorithm = rval.algo
new_ids = rval.trials.new_trial_ids(1)
rval.trials.refresh()
random_state = rval.rstate.randint(2**31 - 1)
if random_search:
new_trials = hp.rand.suggest(new_ids, rval.domain, trials,
random_state)
......
test/cli_test.py 0 → 100644
View file @ 88ef6c04
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge,
# to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import sys
import time
import traceback
from utils import GREEN, RED, CLEAR, setup_experiment
def test_nni_cli():
import nnicli as nc
config_file = 'config_test/examples/mnist.test.yml'
try:
# Sleep here to make sure previous stopped exp has enough time to exit to avoid port conflict
time.sleep(6)
print(GREEN + 'Testing nnicli:' + config_file + CLEAR)
nc.start_nni(config_file)
time.sleep(3)
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_job_statistics())
print(nc.get_experiment_status())
nc.list_trial_jobs()
print(GREEN + 'Test nnicli {}: TEST PASS'.format(config_file) + CLEAR)
except Exception as error:
print(RED + 'Test nnicli {}: TEST FAIL'.format(config_file) + CLEAR)
print('%r' % error)
traceback.print_exc()
raise error
finally:
nc.stop_nni()
if __name__ == '__main__':
installed = (sys.argv[-1] != '--preinstall')
setup_experiment(installed)
test_nni_cli()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment