Skip to content

GitLab

Unverified Commit c1811162 authored by chicm-ms's avatar chicm-ms Committed by GitHub
Browse files

Python wrapper for rest api (#1318)

parent 410ab1ca
Hide whitespace changes
Inline Side-by-side
Showing with 772 additions and 3 deletions
Makefile
View file @ c1811162
......@@ -167,6 +167,7 @@ dev-install-python-modules:
#$(_INFO) Installing Python SDK $(_END)
mkdir -p build
ln -sf ../src/sdk/pynni/nni build/nni
ln -sf ../src/sdk/pynni/nnicli build/nnicli
ln -sf ../tools/nni_annotation build/nni_annotation
ln -sf ../tools/nni_cmd build/nni_cmd
ln -sf ../tools/nni_trial_tool build/nni_trial_tool
......
azure-pipelines.yml
View file @ c1811162
......@@ -34,6 +34,10 @@ jobs:
cd test
PATH=$HOME/.local/bin:$PATH python3 metrics_test.py
displayName: 'Trial job metrics test'
- script: |
cd test
PATH=$HOME/.local/bin:$PATH python3 cli_test.py
displayName: 'nnicli test'
- job: 'basic_test_pr_macOS'
pool:
......@@ -61,3 +65,7 @@ jobs:
cd test
PATH=$HOME/Library/Python/3.7/bin:$PATH python3 tuner_test.py
displayName: 'Built-in tuners / assessors tests'
- script: |
cd test
PATH=$HOME/Library/Python/3.7/bin:$PATH python3 cli_test.py
displayName: 'nnicli test'
deployment/pypi/setup.py
View file @ c1811162
......@@ -53,13 +53,16 @@ setuptools.setup(
long_description_content_type = 'text/markdown',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
packages = setuptools.find_packages('../../tools') + setuptools.find_packages('../../src/sdk/pynni', exclude=['tests']),
packages = setuptools.find_packages('../../tools') \
+ setuptools.find_packages('../../src/sdk/pynni', exclude=['tests']) \
+ setuptools.find_packages('../../src/sdk/pycli'),
package_dir = {
'nni_annotation': '../../tools/nni_annotation',
'nni_cmd': '../../tools/nni_cmd',
'nni_trial_tool': '../../tools/nni_trial_tool',
'nni_gpu_tool': '../../tools/nni_gpu_tool',
'nni': '../../src/sdk/pynni/nni'
'nni': '../../src/sdk/pynni/nni',
'nnicli': '../../src/sdk/pycli/nnicli'
},
package_data = {'nni': ['**/requirements.txt']},
python_requires = '>=3.5',
......
examples/notebooks/retrieve_nni_info_with_python.ipynb 0 → 100644
View file @ c1811162
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python wrapper for nni restful APIs\n",
"\n",
"nni provides nnicli module as a python wrapper for its restful APIs, which can be used to retrieve nni experiment and trial job information in your python code. This notebook shows how to use nnicli module.\n",
"\n",
"Following are the functions available in nnicli module:\n",
"\n",
"#### start_nni(config_file)\n",
"Starts nni experiment with specified configuration file\n",
"\n",
"#### stop_nni()\n",
"Stop nni experiment.\n",
"\n",
"#### set_endpoint(endpoint)\n",
"Set nni endpoint for nnicli, the endpoint is showed while nni experiment is started successfully using nnictl command or start_nni function\n",
"\n",
"#### version()\n",
"Returns nni version\n",
"\n",
"#### get_experiment_profile()\n",
"Returns experiment profile.\n",
"\n",
"#### get_experiment_status()\n",
"Returns nni experiment status.\n",
"\n",
"#### get_job_metrics(trial_job_id)\n",
"Returns specified trial job metrics, including final results and intermediate results.\n",
"\n",
"#### get_job_statistics()\n",
"Returns trial job statistics information\n",
"\n",
"#### get_trial_job(trial_job_id)\n",
"Returns information of a specified trial job.\n",
"\n",
"#### list_trial_jobs()\n",
"Returns information of all trial jobs of current experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Start nni experiment using specified configuration file\n",
"Let's use a configruation file in nni examples directory to start an experiment."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"authorName: default\r\n",
"experimentName: example_mnist\r\n",
"trialConcurrency: 1\r\n",
"maxExecDuration: 1h\r\n",
"maxTrialNum: 10\r\n",
"#choice: local, remote, pai\r\n",
"trainingServicePlatform: local\r\n",
"searchSpacePath: search_space.json\r\n",
"#choice: true, false\r\n",
"useAnnotation: false\r\n",
"tuner:\r\n",
" #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner\r\n",
" #SMAC (SMAC should be installed through nnictl)\r\n",
" builtinTunerName: TPE\r\n",
" classArgs:\r\n",
" #choice: maximize, minimize\r\n",
" optimize_mode: maximize\r\n",
"trial:\r\n",
" command: python3 mnist.py\r\n",
" codeDir: .\r\n",
" gpuNum: 0\r\n"
]
}
],
"source": [
"! cat ../trials/mnist/config.yml"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: expand searchSpacePath: search_space.json to /mnt/d/Repos/nni/examples/trials/mnist/search_space.json\n",
"INFO: expand codeDir: . to /mnt/d/Repos/nni/examples/trials/mnist/.\n",
"INFO: Starting restful server...\n",
"INFO: Successfully started Restful server!\n",
"INFO: Setting local config...\n",
"INFO: Successfully set local config!\n",
"INFO: Starting experiment...\n",
"INFO: Successfully started experiment!\n",
"-----------------------------------------------------------------------\n",
"The experiment id is PlUIfDTR\n",
"The Web UI urls are: http://172.18.17.1:8080 http://10.172.121.40:8080 http://10.0.75.1:8080 http://127.0.0.1:8080\n",
"-----------------------------------------------------------------------\n",
"\n",
"You can use these commands to get more information about the experiment\n",
"-----------------------------------------------------------------------\n",
"commands description\n",
"1. nnictl experiment show show the information of experiments\n",
"2. nnictl trial ls list all of trial jobs\n",
"3. nnictl top monitor the status of running experiments\n",
"4. nnictl log stderr show stderr log content\n",
"5. nnictl log stdout show stdout log content\n",
"6. nnictl stop stop an experiment\n",
"7. nnictl trial kill kill a trial job by id\n",
"8. nnictl --help get help information about nnictl\n",
"-----------------------------------------------------------------------\n",
"\n"
]
}
],
"source": [
"import nnicli as nc\n",
"nc.start_nni(config_file='../trials/mnist/config.yml')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Connect nnicli module to started nni experiment\n",
"Call set_endpoint to connect nnicli moduele to the rest server of started nni experiment. Local mode training serviced is used in this notebook, but nnicli module can connect to any started nni experiment. The endpoint can be found in the output of start_nni function."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"nc.set_endpoint('http://127.0.0.1:8080')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Retrieve nni experiment and trial job information"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'errors': [], 'status': 'RUNNING'}"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_status()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'trialJobNumber': 4, 'trialJobStatus': 'SUCCEEDED'},\n",
" {'trialJobNumber': 1, 'trialJobStatus': 'RUNNING'}]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_job_statistics()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'execDuration': 1117,\n",
" 'id': 'PlUIfDTR',\n",
" 'logDir': '/home/chicm/nni/experiments/PlUIfDTR',\n",
" 'maxSequenceId': 3,\n",
" 'params': {'authorName': 'default',\n",
" 'clusterMetaData': [{'key': 'codeDir',\n",
" 'value': '/mnt/d/Repos/nni/examples/trials/mnist/.'},\n",
" {'key': 'command', 'value': 'python3 mnist.py'}],\n",
" 'experimentName': 'example_mnist',\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"hidden_size\": {\"_value\": [124, 512, 1024], \"_type\": \"choice\"}, \"batch_size\": {\"_value\": [1, 4, 8, 16, 32], \"_type\": \"choice\"}, \"conv_size\": {\"_value\": [2, 3, 5, 7], \"_type\": \"choice\"}, \"dropout_rate\": {\"_value\": [0.5, 0.9], \"_type\": \"uniform\"}, \"learning_rate\": {\"_value\": [0.0001, 0.001, 0.01, 0.1], \"_type\": \"choice\"}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'trialConcurrency': 1,\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'checkpointDir': '/home/chicm/nni/experiments/PlUIfDTR/checkpoint',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'className': 'TPE'},\n",
" 'versionCheck': True},\n",
" 'revision': 116,\n",
" 'startTime': 1564484985839}"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nc.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define an utility function to format json string returned by nnicli module."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"def show_json(res):\n",
" print(json.dumps(res, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"params\": {\n",
" \"searchSpace\": \"{\\\"hidden_size\\\": {\\\"_value\\\": [124, 512, 1024], \\\"_type\\\": \\\"choice\\\"}, \\\"batch_size\\\": {\\\"_value\\\": [1, 4, 8, 16, 32], \\\"_type\\\": \\\"choice\\\"}, \\\"conv_size\\\": {\\\"_value\\\": [2, 3, 5, 7], \\\"_type\\\": \\\"choice\\\"}, \\\"dropout_rate\\\": {\\\"_value\\\": [0.5, 0.9], \\\"_type\\\": \\\"uniform\\\"}, \\\"learning_rate\\\": {\\\"_value\\\": [0.0001, 0.001, 0.01, 0.1], \\\"_type\\\": \\\"choice\\\"}}\",\n",
" \"clusterMetaData\": [\n",
" {\n",
" \"key\": \"codeDir\",\n",
" \"value\": \"/mnt/d/Repos/nni/examples/trials/mnist/.\"\n",
" },\n",
" {\n",
" \"key\": \"command\",\n",
" \"value\": \"python3 mnist.py\"\n",
" }\n",
" ],\n",
" \"tuner\": {\n",
" \"classArgs\": {\n",
" \"optimize_mode\": \"maximize\"\n",
" },\n",
" \"builtinTunerName\": \"TPE\",\n",
" \"checkpointDir\": \"/home/chicm/nni/experiments/PlUIfDTR/checkpoint\",\n",
" \"className\": \"TPE\"\n",
" },\n",
" \"maxTrialNum\": 10,\n",
" \"maxExecDuration\": 3600,\n",
" \"experimentName\": \"example_mnist\",\n",
" \"authorName\": \"default\",\n",
" \"trialConcurrency\": 1,\n",
" \"trainingServicePlatform\": \"local\",\n",
" \"versionCheck\": true\n",
" },\n",
" \"execDuration\": 1192,\n",
" \"revision\": 124,\n",
" \"logDir\": \"/home/chicm/nni/experiments/PlUIfDTR\",\n",
" \"maxSequenceId\": 3,\n",
" \"id\": \"PlUIfDTR\",\n",
" \"startTime\": 1564484985839\n",
"}\n"
]
}
],
"source": [
"show_json(nc.get_experiment_profile())"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\n",
" {\n",
" \"startTime\": 1564484995992,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":0,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":1024,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.8055724367106529}}\"\n",
" ],\n",
" \"id\": \"BW0NR\",\n",
" \"endTime\": 1564485259753,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 0,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"0\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"BW0NR\",\n",
" \"timestamp\": 1564485258774,\n",
" \"data\": \"0.9078999757766724\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/BW0NR\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485271947,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":1,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":4,\\\"conv_size\\\":5,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.01,\\\"dropout_rate\\\":0.5547528540531742}}\"\n",
" ],\n",
" \"id\": \"x0P5w\",\n",
" \"endTime\": 1564485642784,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 1,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"1\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"x0P5w\",\n",
" \"timestamp\": 1564485642072,\n",
" \"data\": \"0.10100000351667404\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/x0P5w\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485652151,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":2,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.5584485925416655}}\"\n",
" ],\n",
" \"id\": \"V9jSG\",\n",
" \"endTime\": 1564485917057,\n",
" \"status\": \"SUCCEEDED\",\n",
" \"sequenceId\": 2,\n",
" \"finalMetricData\": [\n",
" {\n",
" \"parameterId\": \"2\",\n",
" \"type\": \"FINAL\",\n",
" \"trialJobId\": \"V9jSG\",\n",
" \"timestamp\": 1564485916403,\n",
" \"data\": \"0.928600013256073\",\n",
" \"sequence\": 0\n",
" }\n",
" ],\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/V9jSG\"\n",
" },\n",
" {\n",
" \"startTime\": 1564485927295,\n",
" \"hyperParameters\": [\n",
" \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":3,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":7,\\\"hidden_size\\\":124,\\\"learning_rate\\\":0.001,\\\"dropout_rate\\\":0.6281630602835235}}\"\n",
" ],\n",
" \"id\": \"CDlRX\",\n",
" \"status\": \"RUNNING\",\n",
" \"sequenceId\": 3,\n",
" \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/CDlRX\"\n",
" }\n",
"]\n"
]
}
],
"source": [
"show_json(nc.list_trial_jobs())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing nni experiment result\n",
"\n",
"With the retrieved trial job information, we can do some analysis by visualizing the metric data, below is a simple example."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3QAAAF4CAYAAAAczbvpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl8TNfj//F3MiQoGoQIsdTSVBFSitoiERIkoiFFPnSh0ZbSaq1dkBa1VKuWVqtaSz6WplJLpPiqvSUUReujVKkiIhJbQiWZ5PeHn/mYT0gGE8nl9Xw8PB6Ze889c85xM5P3vefe65CdnZ0tAAAAAIDhOBZ0AwAAAAAAd4ZABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgCAXcycOVOjRo2yqeyQIUM0ffr0m6777rvv9OKLL9pUT8+ePRUTE2NzG6OiovTUU0/J29tbly5dkre3t06dOmXz9rfy8ccfa8SIEXddz91q3bq14uPjC7oZAIB7qEhBNwAAUPh4e3tbfr5y5YqcnJxkMpkkSZGRkercuXOObQYMGGCX93766af19NNP26WuG129elUTJ05UTEyMateuLUnas2eP3d+nsPj444+VmJioCRMmFHRTAAD5iEAHAMjhxqDj5+ensWPHqnnz5rcsn5mZqSJFCvdXSlJSktLT0y1hrqAYYawAAMbBlEsAwG37+OOP9frrr+uNN96Qt7e3VqxYYTXtMCsrS4MGDVKLFi3UuHFj9e7dW0eOHLGp7ujoaPXu3dvy+ueff1ZoaKgaNWqkbt26ae/evVbl//rrL8v6AQMG6MKFCznqPHLkiIKCgiRdO/vYp08fZWZmytPTUydOnJB0bRro2LFj9eKLL8rb21vdu3fX33//banjvffeU+vWrfXEE0+oa9eu2r17t039+emnn+Tn56dZs2apRYsWevfddyVJP/zwgzp37qzGjRurZ8+eOnTokGWbWbNmqWXLlnriiScUGBhomUb5v1NVr9f9vzZs2KA5c+Zo5cqV8vb2VmhoqGVs/fz85O3trbZt22rVqlU29QEAUHgR6AAAd2TdunUKCgrSrl271LFjxxzr27RpozVr1ujHH39U7dq1NXTo0Nt+j5SUFL300kvq06eP4uPj1atXL/Xr188qtC1btkyTJk3Sli1blJ2drQ8++CBHPTVr1tTy5cslXTv7+NVXX930/VauXKnXXntNO3bskLu7uz755BPLOi8vL61YsUI7duxQQECAXnvtNaWnp9vUj9OnT+vy5cvasGGDRo8erX379undd9/V2LFjFR8fr65du6p///5KT0/X4cOHtWTJEn333XfavXu3Zs+erUqVKt3OsMnX11d9+/ZVcHCw9uzZo5iYGKWmpmrChAn66quvtGfPHi1atEienp63VS8AoPAh0AEA7sgTTzwhPz8/OTo6qlixYlbrHB0dFRoaqpIlS8rZ2VmvvvqqfvvtN12+fPm23mPDhg2qXbu2goKCVKRIEXXp0kUeHh7auHGjpUyXLl1Uq1YtlShRQoMGDdKqVauUnZ19R30KCAhQ/fr1VbRoUQUHB+vgwYNW7+Pi4qIiRYooIiJCqamp+uuvv2yq12Qy6dVXX5WTk5OKFSumb775RuHh4fLy8pLJZFK3bt0kSfv375fJZNLVq1f1xx9/KDMzU1WqVFGVKlXuqD//y8HBQYcPH9bVq1dVoUIF1apVyy71AgAKDpP4AQB3xN3d/ZbrzGazpkyZojVr1ujcuXNydLx2/PDcuXMqUaKEze9x5syZHGenKleurMTExJu2o3LlykpPT9f58+dVpkwZm9/nuvLly1t+Ll68uFUAnT17tpYuXaqkpCQ5ODjoypUrOnfunE31urq6ysnJyfL61KlTWrlypebOnWtZlpGRocTERDVq1EjDhw/XJ598oj///FMtW7bUyJEjrdp2J0qWLKkpU6bo66+/1siRI9WoUSONGDFCjzzyyF3VCwAoWJyhAwDcEQcHh1uuW7ZsmTZv3qx58+Zp165dWrt2rSTd9pmzChUq5HiswKlTp+Tm5mZ5nZCQYLXOyclJLi4ut/U+edm+fbvmzp2r6dOn6+eff9bOnTtVokQJm/vzv2NVsWJFDRgwQD///LPl3969ey1TV0NCQrR48WL98MMPMpvN+uijjyRJJUqU0JUrVyz1JCUl2fyekuTj46O5c+dq69atqlq1qs2PmQAAFF4EOgCA3aWlpVmC1ZUrVzR16tQ7qsfX11eHDx9WXFycMjMztXLlSh0/flxt2rSxlFm+fLmOHDmiy5cva/r06erQoUOuYfNOpKWlyWQyqUyZMsrIyND06dOtgtXteuaZZ7Rw4ULt27dP2dnZSktL0/r163X58mUdOXJE27dvV3p6upydnVWsWDFLfx577DFt2rRJFy5c0JkzZ7RgwYJbvoerq6tOnjxpCZ1nzpzR+vXrdeXKFRUtWlQlSpSwnDkFABgXn+QAALsLDQ1VhQoV1KpVKwUFBVk91+52lC1bVp999plmz56tpk2bau7cuZo1a5YefvhhS5mQkBANHTpULVu2lNls1ltvvWWvblj4+PioefPmat++vfz8/FSyZMm7mgLZsGFDjRkzRmPGjNGTTz6pgIAArVixQpKUnp6uyZMnq2nTpmrZsqUuXLigwYMHS7o2rjVr1pSvr69efPFFderU6Zbv0bFjR2VkZKhJkybq1q2bsrKyNGfOHLVs2VJNmzbV7t27OUMHAPcBh+w7vXIcAIB8sGTJEq1Zs+aWd6IEAAD/xRk6AEChcvjwYXl4eBR0MwAAMATucgkAKDReeuklnTp1yur5bwAA4NaYcgkAAAAABsWUSwAAAAAwKAIdAAAAABgUgQ4AAAAADKpQ3RTl3Lk0ZWVxSR8AAACAB4ujo4PKlHnotrcrVIEuKyubQAcAAAAANmLKJQAAAAAYFIEOAAAAAAyKQAcAAAAABpVnoJs4caL8/Pzk6empQ4cO3bSM2WxWZGSk/P391a5dO0VHR9u9oQAAAAAePEeOHFaHDm3VrJm3OnRoqz///CNHmcTERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZkqQNG35Qu3Y+8vBw1ejRb1vVmdu6wiTPQNe2bVv9+9//VuXKlW9ZZuXKlTp+/LjWrl2rJUuWaPr06Tpx4oRdGwoAAADgwTN06GD16ROh7dv3qE+fCA0Z8nqOMqNHj1SDBt7atGmbli9frfHj39PJk9fyyCeffKjatT21adM2bdy4Tfv2/aJVq1ZIkqpVq66PPpquAQMG5agzt3WFSZ6BrnHjxnJ3d8+1TFxcnMLCwuTo6KiyZcvK399fq1evtlsjAQAAADx4kpKStG/fXoWGhkmSQkPDtG/fXp09e9aq3G+//So/P39Jkqurq+rVq6/ly7+TJDk4OCg1NVVZWVm6evWq0tMzVLFiJUlSjRo1Vb++l0ymnDf/z21dYWKXa+gSEhJUqVIly2t3d3edPn3aHlUDAAAAeECdOnVC7u7uMplMkiSTyaSKFSvq1Cnr2YBeXg313XdLlZ2drb/+OqadO+N14sRxSdIbbwzTn3/+oXr1aqtevdry9W2rpk2b3fO+5JdCFTfLlStZ0E0AAAAAUEiUKfOQTCZHlS9fyrLMZHJUmTIPWS2bOXOaBg8erHbtWqlq1ary9/dXqVIlVL58KcXELFSjRt7avHmjLl26pA4dOmjTpjXq1q2bZfuHHnJWdnaGVZ22rCsM7BLo3N3dderUKXl5eUnKecbOVsnJqfn6YPEjRw7r1Vdf1rlzKSpTpqxmzvxcNWrUsiqTmJiooUNf019//aXMzAy9/voQhYX1sKxfvjxGH300SdnZ2XJwcFB09ApVqFBBkyaN19y5X8rN7dr01CZNmmrixI8s7/vmm6/pwoULSk+/qpCQUA0b9la+9RMAAAC4HxQvXkYnTpzU6dPnZTKZZDabdfLkKRUvXkZJSZduKFlMH3/8meVVz55d1axZKyUlXdLUqZ9o6tSZSk5Ok+Sotm0DFRe3Rj4+AZbyaWlXdfly+v/Umfc6e3J0dLijE1x2mXIZGBio6OhoZWVlKSUlRevWrVNAQEDeG95jd3tB5S+/7NbkyR/om2+Wa/PmeK1cuUalS5e2bBsW1lMbNvyoDRt+tIQ5SYqMHKXg4BBt2PCj1qzZqMWL/63du3/O/w4DAADY4G7vIihdO+jt49NMrVs3lY9PM505c0bStbuhDx/+hp580ktNmjRQVNQ8yzZTpkxUq1ZN5OPzlPz9W2v9+nX521EYTvny5VWvXn3FxFy7i35MTLTq1/eSq6urVbmUlGTLnSu3bNmk//zngOW6u6pVq1n2rfT0dG3evFGPPfb4PexF/soz0I0dO1atW7fW6dOn9cILL6hTp06SpIiICO3fv1+SFBISIg8PD7Vv317PPPOMBgwYoCpVquRvy2+TPS6onDVrpvr3HyQ3NzdJUunSD6tYsWJ5vreDg4MuXrwoSbpy5YocHBzk6lrebn2D8d3tF+mkSeP1+OM15OvbQr6+LTR8+BtWdXfp0lG+vi3UokVjTZo03rLu8uXLioh4Xk2aNFDz5o20du33+dtRAEChlJ8Hvb/9domOHv1T8fG/KC7uB02e/IGOH/9LkuTt3Uhr1mzUpk3bNHXqTPXr94KuXLly7zoOQ5g8earmzPlczZp5a86czzV58lRJ187C/fLLbknSnj271KJFYzVv3kgTJ47TggVLVKJECUnS++9P0Pbt2+Tj00x+fi1Us2ZN9e79vCRp+/ZtatDgMc2aNVPz53+tBg0es4S/3NYVJg7Z2dn5N8fxNuXnlMu9e/fo1Vdf0pYtOyzLWrZ8Up9+OlteXg0tywYM6Kdy5VwVGTlOx4//pYCANgoNDdP48ZPl59dS7dq117ZtPyktLU2dOgVr8OChcnBw0KRJ4/Xvf8+Xi0sZVahQQcOGvaUnn2wqSfr77+Pq1au7zp1L0YUL5zVq1Pvq27dfvvQTxhQaGqSePXspLKyHoqMXa9GiKMXExFqVefnlPqpd21NvvjlcZ8+eVbt2rRUbu1aVK3to0qTxSktLU2TkuBx1P/tsT/n4tFHfvi8pNTVVrVs31ZdfztMTTzTWhx9O0KlTJ/XRR9P1559/KDg4UPHxv6hkSa5nBYAHRVJSkp566gn9/vsxy5Q2T8/q2r59j9VZkFatmmjatM/k7d1IktS7d3c99VRL9e8/UC+/3FetW7dReHjvHPWHh3dTz569FBzcRZI0YsSb8vCoqldffc2qXHZ2tmrVqqItW+JVqdKtH5cF3K8KdMrl/SQycrySks7I17eF3n57mFq1amO5VanZbNaBA78pOnq5li+P0w8//J+++WaRJOm55/rq55/3a9OmbRow4DU991xPpaQkS5Lmz/9aYWE9tG/f79qxY6++/HKWdu3aWWB9ROFij7PHucntDPHy5TF69tkXJEk1atRSw4beWr/+/+zWNwBA4WePuwgeOvS7/vrrqDp3DlTbtq0s9xuQpJMnT8jD478ztypXrpKjbklasmShqld/hDAH3KZCdZfL/FSpkocSEhJkNpstR59Onz6tSpU8rMq5urrqs8++tLzu2bOrPD19JUkeHh4KCgqRs7OznJ2dFRjYSXv27FL37uGWaZiS1KaNnypV8tDBg/9R8+YtNXv2LO3cuU+S5OZWUS1b+mjbtp/UqNGT96DnKOxy+yK98cjo9S/Shg2f0PHjf2nnznhVqVLVsn7ZsqXauHF9jjPEY8dOUK9e3fX1119azhBXrVpNknTixAl5ePy3jsqVPXTy5Ml70W0AgMFERo7Xu++OkK9vC3l4eNzyoHdGRrq6dw9V5coe6t493Ka6f/ppqyZOHKfo6OX52QUUkNIuxeVc9IGJHbm6mpGpi+ftO634gRnZGy+oDAvrkesFlaVLP6wiRYpYLqicM2eBpGtnTtatW6tnnumpzMxMbdmy0TJ9ICHhlNzdr93Zc//+ffr7779Us2ZtSVK1atW0fv3/qXv3cKWmXlJ8/E8KDOxwD3uP+0FuX6TPPddXgwcPVdGiRbVx43o991xPbd26U2XLlrOcIX711deUmHhaXbp0VMOG3hxQAABIyv+D3pUre+jEib8tUzVPnvzb6mDizp3x6t8/QvPnL1KtWrXvQY9xrzkXLaI3vttU0M0oFD562sfudT5QUy7v9oLKp5/uJlfX8mrZ8kn5+bWQp2cd/etfz0qSxo2LVOvWTdWmTXO9+eZAzZz5heWs3bRpn2nevK/Upk1zBQb6qXPnp9W2bfsCGAEURjd+kUrK84t048afFBX1jVJTL8nT8zFJkpubm4oWLSrJ+gyxJM2ePctyhPTGM8TStS/g69NlpGvTYipXZqoLADxI7HEXwdDQMG3cuF7Z2dnKyMjQli0bVbdufUlS585Pa8GCucrKytLZs2f1/ferFBwcIuna3139+r2gOXPmW93TAIDtHpibogCFWZcuHfWvfz1ruSnKwoUL9N13q6zK/O/Z44EDX9ZPP+1SiRIlcpwh7tYtWJs375Cbm5t8fJqpf/9BljPEHTv6a/To99W2bXtNmjRep08nWG6KEhQUoB07flHJkoXzwZm2uttnTi5aFKVZs2bK0dFRWVlm9er1nCIiXslzu6SkJL322is6efKkMjMz1KJFK40fP1lFijwwkyEAGNThw4c0cOBLOn/+vFxcXDRjxheqVau2evbsquHD31bDhk/ohx/W6q23hslkMqls2XL64IMPVb/+tWcQZ2VlafTot7V+/f/J0dFRbdq0VWTkODk6OspsNmvEiCHauPEHSdLAgYMt12+3b++jv/8+rooV//v84pkzv9Djj9e994OAfFO+fCnO0P1/Hz3tc8vn2d3pTVEIdEAhcLdfpK+++pL27ftFjo4mOTkV1bBhb8nf/9qzIPfu3aORI4fq8uXLyszMUJcuXTVkyAhJUlpamgYNekX79++VyWTSqFHvq0OHTgU2DvZyt3cNvXTpokqWLCUHBwelpl5S69bNtGDBEtWtWy/X7d55Z7hMpiKKjBynjIwMBQe31yuvDFRISGgBjQQAAAWPQPdf+RHoDHPYuFTpYirmXLSgm1Eo/HM1Q5cu/lPQzYAd1a79qFav3pBj+aJFSy0/t23bXvHxN5+qO2PG57esu0EDb8XF3fyZKQ899JDmzJl/m60t3K7fNfT6hfWhoWEaOXKozp49azV96LffftVLLw2QZH3X0P79B6pUqdKWcpcvX1FGRoYcHBzy3O5aAExVVlaWrl69qvT0DKujzgCAB0N+zhS57o8/Dqtt25Z6/vkXLY8tio5erBkzPtGhQwc1duwE9e370r3pMAqUYQJdMeeiCh/274JuRqGwcNK/dEkEOuBm7HXX0NWr4zRu3BgdO3ZUb7892jL9J7ft3nhjmPr06a169Wrr8uXL6tu3n5o2bXYPew/gQfXww8Xk5MSBb0lKT8/QhQsF+3fS9Qe1X58pMmTI6zlmilx/UPv8+YstMz6aN2+pypU9FBTUWT16/Mtqpkjz5q1Ut249Sdeutx8y5LUcs2rq1fPSF198rWnTPrpnfUXBM0ygAwqz0g87y9nJqaCbUShcTU/XxQtXC7oZecrtrqGSFBjYUYGBHXXixN967rlw+fsHqFat2rlut2LFMj3+eF0tXbpSqamX1KNHV61cucxyN1wAyC9OTkU1ZcqUgm5GofDmm29KBXjgO79nikjStGkfqV27QKWlpSotLc2yvE6dxyVJjo4P1H0PH3gEOsAOnJ2c9PzXrxV0MwqFuS98IqngAp09br99Iw+PKvL2bqS1a1erVq3auW43Z87nmjr12hSZ0qUfVmBgJ23duplABwAPkPyeKfLrr/u1YcMP+u67VZoyZeK97RwKJeI7gPuKPW6/fejQ75ZyycnJ+vHHzZajnrltV7VqNa1ff+16xfT0dG3evFGPPfZ4PvYWAGBUkZHjlZR0Rr6+LfT228NuOlNky5Yd2rZtt6Kjl+iPPw4rIyNDQ4YM0uTJUy2BEeAMHYD7zuTJUzVw4EuaMmWi5a6hkqzuGrpnzy6ru4be+MzJBQu+1saN61WkSFFlZ2erb99+8vVtK0m5bvf++xM0dOhg+fg0k9lsVosWrdS79/MFMgYAgIKRnzNFOnfuomPHjio8vJsk6cKFC8rOzlZq6iVNmTItfzuGQotAB+C+c7d3DX3//Qm3rDu37R55pIa+/Xb5bbYWAHA/uXGmSFhYj1xnitz4fNn//OeA5sxZIOnaTJFHH/WU9N+ZIp06BcvDo4oOHjxmqWPSpPFKS0uz3OUSDyYCHQAAAGBH+TlTJDcxMdGKjHxXFy6c1+rVcZo27WN9880yeXo+lq/9RcEi0AEodFxKOaloMeeCbkahkPHPVZ2/lF7QzQAA3Ib8nClyo2HD3rJ6HRoaZrmuGw8OAh2AQqdoMWfFPftCQTejUOg4/2uJQAcAAG6BQAcAAADcoIyLs4oU5fmykpSZka5z5wv/82UfZAQ6AAAA4AZFijppc+yYgm5GodA6aIwK8vmyyBvPoQMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUEVsKXT06FGNGDFC58+fl4uLiyZOnKjq1atblUlOTtbIkSOVkJCgzMxMNW3aVO+8846KFLHpLQAAAAAAt8mmM3SjR49WeHi41qxZo/DwcI0aNSpHmVmzZqlmzZpauXKlVqxYod9++01r1661e4MBAAAAANfkGeiSk5N14MABBQUFSZKCgoJ04MABpaSkWJVzcHBQWlqasrKylJ6eroyMDLm5ueVPqwEAAAAAeQe6hIQEubm5yWQySZJMJpMqVKighIQEq3L9+/fX0aNH1bJlS8u/Ro0a5U+rAQAAAAC2XUNni9WrV8vT01Pz5s1TWlqaIiIitHr1agUGBtpcR7lyJe3VnPte+fKlCroJwC2xf9oX4wkAt4fPTftiPO3L3uOZZ6Bzd3dXYmKizGazTCaTzGazzpw5I3d3d6tyUVFRGj9+vBwdHVWqVCn5+fkpPj7+tgJdcnKqsrKyb7qOHclaUtKlgm4CbsD+ae1u90/G0xq/7wDywuemNb6H7IvxtK9bjaejo8MdneDKc8pluXLlVKdOHcXGxkqSYmNjVadOHZUtW9aqnIeHhzZv3ixJSk9P17Zt21S7du3bbhAAAAAAwDY23eVyzJgxioqKUkBAgKKiohQZGSlJioiI0P79+yVJb731lnbt2qXg4GB16dJF1atX1zPPPJN/LQcAAACAB5xN19DVrFlT0dHROZbPnj3b8nPVqlX19ddf269lAAAAAIBc2XSGDgAAAABQ+BDoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIOy6cHiAIAH05Ejh/Xqqy/r3LkUlSlTVjNnfq4aNWpZlRkwoJ8OHPjN8vrAgV81b94iBQZ21JQpE7Vs2VI5OppUtGhRvfXWKPn5+UuSunbtrJSUZEmS2Zypgwf/ow0bflLduvV05Mhhvfnma7pw4YLS068qJCRUw4a9de86DgCAQRDoAAC3NHToYPXpE6GwsB6Kjl6sIUNeV0xMrFWZmTO/sPz866/71bVrkHx920qSvL0b6ZVXBqpEiRL69df96tKlo/bvP6TixYtr6dIVlu3i4mI1YcL7qlu3niQpMnKUgoND1LfvS0pNTVXr1k3l799eTzzR+B70GgAA42DKJQDgppKSkrRv316FhoZJkkJDw7Rv316dPXv2ltssXDhfXbs+I2dnZ0mSn5+/SpQoIUmqW7eesrOzde5cSo7tFi1aoJ49e1teOzg46OLFi5KkK1euyMHBQa6u5e3WNwAA7hcEOgDATZ06dULu7u4ymUySJJPJpIoVK+rUqRM3LZ+enq6YmGirYHajJUsWqnr1R1SpUmWr5YmJidq8eaPCwnpYlo0dO0HLlsXIy8tTjRvXU//+g1S1ajU79QwAgPsHgQ4AYBfffx+rypWrqH59rxzrfvppqyZOHKfPP/8qx7pvvlkkX19/ubq6WpbNn/+1wsJ6aN++37Vjx159+eUs7dq1M1/bDwCAERHoAAA3VamShxISEmQ2myVJZrNZp0+fVqVKHjctv3DhAoWH98qxfOfOePXvH6F58xaqVq3aOdYvXhyVY7vZs2epe/dwSZKbW0W1bOmjbdt+utsuAQBw3yHQAQBuqnz58qpXr75iYqIlSTEx0apf38vqTNp1p06dVHz8NnXt+ozV8j17dqlfvxc0Z858eXk1zLHdjh3xunjxotq2bW+1vFq1alq//v8kSamplxQf/5Pq1Kljr64BAHDfINABAG5p8uSpmjPnczVr5q05cz7X5MlTJUk9e3bVL7/stpRbsmSh2rcPlItLGavthw9/Q//8c0VDhrwuX98W8vVtYfWIg8WLo/TMMz0t1+ldN23aZ5o37yu1adNcgYF+6tz56RyhDwAA8NgCAEAuatd+VKtXb8ixfNGipVavBw8eetPt167dlGv9H300/abLGzTwVlzcOhtbCQDAg4szdAAAAABgUJyhA4D72MOli8vJmY96SUq/mqkLF68UdDMAALArvuUB4D7m5FxE49/+tqCbUSi8Na5bQTcBAAC7Y8olAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMZtZ4OAAAgAElEQVQi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADMqmQHf06FF1795dAQEB6t69u44dO3bTcnFxcQoODlZQUJCCg4N19uxZe7YVAAAAAHCDIrYUGj16tMLDwxUSEqLly5dr1KhRmj9/vlWZ/fv3a8aMGZo3b57Kly+vS5cuycnJKV8aDQAAAACw4QxdcnKyDhw4oKCgIElSUFCQDhw4oJSUFKtyc+fOVZ8+fVS+fHlJUqlSpeTs7JwPTQYAAAAASDYEuoSEBLm5uclkMkmSTCaTKlSooISEBKtyR44c0d9//61//etfevrpp/Xpp58qOzs7f1oNAAAAALBtyqUtzGazfv/9d3399ddKT0/Xiy++qEqVKqlLly4211GuXEl7Nee+V758qYJuAnBL7J/2xXjaD2MJPBj4XbcvxtO+7D2eeQY6d3d3JSYmymw2y2QyyWw268yZM3J3d7cqV6lSJQUGBsrJyUlOTk5q27at9u3bd1uBLjk5VVlZNz+rx45kLSnpUkE3ATdg/7R2t/sn42ntbsaTsbTGZyfuV/yuW+N7yL4YT/u61Xg6Ojrc0QmuPKdclitXTnXq1FFsbKwkKTY2VnXq1FHZsmWtygUFBWnr1q3Kzs5WRkaGtm/frscee+y2GwQAAAAAsI1Njy0YM2aMoqKiFBAQoKioKEVGRkqSIiIitH//fklSp06dVK5cOXXs2FFdunRRrVq11K1bt/xrOQAAAAA84Gy6hq5mzZqKjo7OsXz27NmWnx0dHTVy5EiNHDnSfq0DAAAAANySTWfoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQNgW6o0ePqnv37goICFD37t117NixW5b9888/1aBBA02cONFebQQAAAAA3IRNgW706NEKDw/XmjVrFB4erlGjRt20nNls1ujRo+Xv72/XRgIAAAAAcsoz0CUnJ+vAgQMKCgqSJAUFBenAgQNKSUnJUfaLL75QmzZtVL16dbs3FAAAAABgLc9Al5CQIDc3N5lMJkmSyWRShQoVlJCQYFXu4MGD2rp1q55//vl8aSgAAAAAwFoRe1SSkZGhd999Vx988IEl+N2JcuVK2qM5D4Ty5UsVdBOAW2L/tC/G034YS+DBwO+6fTGe9mXv8cwz0Lm7uysxMVFms1kmk0lms1lnzpyRu7u7pUxSUpKOHz+ufv36SZIuXryo7Oxspaam6v3337e5McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5RnoypUrpzp16ig2NlYhISGKjY1VnTp1VLZsWUuZSpUqKT4+3vJ6+vTpunz5soYPH37bDQIAAAAA2Mamu1yOGTNGUVFRCggIUFRUlCIjIyVJERER2r9/f742EAAAAABwczZdQ1ezZk1FR0fnWD579uyblh84cODdtQoAAAAAkCebztABAAAAAAofAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBEegAAAAAwKAIdAAAAABgUAQ6AAAAADAoAh0AAAAAGBSBDgAAAAAMikAHAAAAAAZFoAMAAAAAgyLQAQAAAIBBFbGl0NGjRzVixAidP39eLi4umjhxoqpXr25VZubMmYqLi5Ojo6OKFi2qwYMHq1WrVvnRZgAAAACAbAx0o0ePVnh4uEJCQrR8+XKNGjVK8+fPtyrj5eWlPn36qHjx4jp48KB69eqlrVu3qlixYvnScAAAAAB40OU55TI5OVkHDhxQUFCQJCkoKEgHDhxQSkqKVblWrVqpePHikiRPT09lZ2fr/Pnz+dBkAAAAAIBkQ6BLSEiQm5ubTCaTJMlkMqlChQpKSEi45TbLli1T1apVVbFiRfu1FAAAAABgxaYpl7djx44d+uSTT/TVV1/d9rblypW0d3PuW+XLlyroJgC3xP5pX4yn/TCWwIOB33X7Yjzty97jmWegc3d3V2Jiosxms0wmk8xms86cOSN3d/ccZffs2aOhQ4fq008/VY0aNW67McnJqcrKyr7pOnYka0lJlwq6CbgB+6e1u90/GU9rdzOejKU1Pjtxv+J33RrfQ/bFeNrXrcbT0dHhjk5w5Tnlsly5cqpTp45iY2MlSbGxsapTp47Kli1rVW7fvn0aPHiwpk2bprp16952QwAAAAAAt8em59CNGTNGUVFRCggIUFRUlCIjIyVJERER2r9/vyQpMjJS//zzj0aNGqWQkBCFhITo999/z7+WAwAAAMADzqZr6GrWrKno6Ogcy2fPnm35eenSpfZrFQAAAAAgTzadoQMAAAAAFD4EOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHQAAAAAYFAEOgAAAAAwKAIdAAAAABgUgQ4AAAAADIpABwAAAAAGRaADAAAAAIMi0AEAAACAQRHoAAAAAMCgCHS4I0eOHFaHDm3VrJm3OnRoqz///CNHmQ0bflC7dj7y8HDV6NFvW60zm80aPvwNPfmkl5o0aaCoqHmWdYsWRcnH5yn5+raQj08zzZ79mWXdlCkT1apVE/n4PCV//9Zav35d/nUSAAAAKOQIdLgjQ4cOVp8+Edq+fY/69InQkCGv5yhTrVp1ffTRdA0YMCjHum+/XaKjR/9UfPwviov7QZMnf6Djx/+SJAUFddbGjT9pw4YftWrV/+mzz2bot99+lSR5ezfSmjUbtWnTNk2dOlP9+r2gK1eu5G9nAcBObDkYltsBr8TERD37bA/5+DylFi0aKzp6sWVdUlKSwsO7WdYNGzZYmZmZedYJADA2Ah1uW1JSkvbt26vQ0DBJUmhomPbt26uzZ89alatRo6bq1/eSyVQkRx3Ll8eod+/n5ejoKFdXV3Xo0EkrViyTJJUqVVoODg6SpMuXrygjI8Py2s/PXyVKlJAk1a1bT9nZ2Tp3LiXf+goA9mTLwbDcDniNHj1SDRp4a9OmbVq+fLXGj39PJ0+ekCR98smHql3bU5s2bdPGjdu0b98vWrVqRZ51Gll+BuQBA/rJ17eF5Z+b28NavTrOqu4//jisatXccsxCAYB7iUCH23bq1Am5u7vLZDJJkkwmkypWrKhTp07YXMfJkyfk4VHF8rpy5SpW269eHadWrZqoUaO6GjBgkB5/vG6OOpYsWajq1R9RpUqV76I3AHBv2HowLLcDXr/99qv8/PwlSa6urqpXr76WL/9OkuTg4KDU1FRlZWXp6tWrSk/PUMWKlfKs08jyMyDPnPmFNmz4URs2/Kjp02fJxcVFvr5tLfWazWYNGfKaOnTodG86CwC3QKBDoRQY2FFbtuzQtm27FR29RH/8cdhq/U8/bdXEieP0+edfFVALAeD22HowLLcDXl5eDfXdd0uVnZ2tv/46pp0743XixHFJ0htvDNOff/6hevVqq1692vL1baumTZvlWadR5XdAvtHChfPVteszcnZ2tiybNu0jtWsXqBo1auVXFwHAJgQ63LZKlTyUkJAgs9ks6dpRytOnT6tSJQ+b66hc2UMnTvxteX3y5N833d7Do4q8vRtp7drVlmU7d8arf/8IzZu3ULVq1b6LngCAsURGjldS0hn5+rbQ228PU6tWbSzT2lesWKbHH6+rX389rH37Dmrbth+1cqXxz8LdSn4H5OvS09MVExOtnj17W5b9+ut+bdjwg15+eUB+dQ8AbEagw20rX7686tWrr5iYaElSTEy06tf3kqurq811dO78tBYsmKusrCydPXtW33+/SsHBIZKkQ4d+t5RLTk7Wjz9uVp06j0uS9uzZpX79XtCcOfPl5dXQjr0CgPxl68Gw3A54ubq66rPPvtTGjT8pKuobpaZekqfnY5KkOXM+V9euz8jR0VGlSz+swMBO2rp1c551PshyC8jXff99rCpXrqL69b0kSRkZGRoyZJAmT55qCZMAUJAIdLgjkydP1Zw5n6tZM2/NmfO5Jk+eKknq2bOrfvlltyRp+/ZtatDgMc2aNVPz53+tBg0eszxmICysh6pVe0RNmzZUhw5+evPN4apWrbokacGCr9WqVRP5+rZQ167B6tu3n+W6heHD39A//1zRkCGvWy5UP3Dgt3s/AABwm2w9GJbbAa+UlGTLnSu3bNmk//zngGXKYdWq1Syfsenp6dq8eaMee+zxPOs0qvwOyNctXLhA4eG9LK8TE0/r2LGjCg/vpkaN6umLLz5TVNQ8vflmzjs6A8C9kPP2g4ANatd+VKtXb8ixfNGipZafmzV7Snv3Hrzp9iaTSZMnf3zTde+/P+GW77t27abbbCkAFB6TJ0/VwIEvacqUiXJxcdGMGV9IunYwbPjwt9Ww4RMKC+uhXbt+VtOm12Yh3HjAa8+eXXrrrWEymUwqW7acFixYYrnz7/vvT9DQoYPl49NMZrNZLVq0Uu/ez0tSrnUa1Y0BOSysR54BuVOnzkpJSdH336/SihXXpvGnpCSrdOmHVaRIEUtAnjNngWXbU6dOKj5+m9X12h4eVXTw4DHL60mTxistLU2RkePyt8MAcAsEOgAA7hFbDobldsCrbdv2io9vf9N1jzxSQ99+u/ym63Kr08jyMyBL1+6m3L59oFxcytzzvgGArQh0D6gyDzupiJNz3gUfAJnpV3XuQnpBNwMAcJvyMyBL0uDBQ/Nsw7Bhb9nQUgDIPwS6B1QRJ2ftmvRiQTejUGg07EtJBDoAeXu4tJOcnDkYJknpV6/qwsW7++ws83BxFXHiTxFJykzP1LkLVwq6GQAMiE9RAABs5OTsrI9GvlTQzSgU3vjgc93twbAiTkW099ONdmmP0TXo36agmwDAoLjLJQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYFIEOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6AAAAADAoAh0AAAAAGBQBDoAAAAAMCgCHQAAAAAYlE2B7ujRo+revbsCAgLUvXt3HTt2LEcZs9msyMhI+fv7q127doqOjrZ3WwEAAAAAN7Ap0I0ePVrh4eFas2aNwsPDNWrUqBxlVq5cqePHj2vt2rVasmSJpk+frhMnTti9wQAAAACAa/IMdMnJyTpw4ICCgoIkSUFBQTpw4IBSUlKsysXFxSksLEyOjo4qW7as/P39tXr16vxpNQAAAABARfIqkJCQIDc3N5lMJkmSyWRShQoVlJCQoLJly1qVq1SpkuW1u7u7Tp8+fVuNcXR0yHW9a5mHbqu++1leY2ULp9Ll7NCS+4M9xtO1ZNm8Cz0g7DGexV3ZP6+72/F82KWEnVpifPbYN0u7sG9eZ4/xLFqqmB1acn+42/EsXbq0nVpifPbYN52Lu9ihJfcHe4xnmRLOdmjJ/eFW43mn4+yQnZ2dnVuBX3/9VcOHD9eqVassyzp27KjJkyerbt26lmXBwcEaN26cvLy8JEmzZ89WYmKi3nnnnTtqGAAAAAAgd3lOuXR3d1diYqLMZrOkazc/OXPmjNzd3XOUO3XqlOV1QkKCKlasaOfmAgAAAACuyzPQlStXTnXq1FFsbKwkKTY2VnXq1LGabilJgYGBio6OVlZWllJSUrRu3ToFBATkT6sBAAAAAHlPuZSkI0eOaMSIEbp48aJKly6tiRMnqkaNGoqIiNCgQYNUv359mc1mvffee/rxxx8lSREREerevXu+dwAAAAAAHlQ2BToAAAAAQOFj03PoAAAAAACFD4EOAAAAAAyKQAcAAAAABkWgAwAAAACDItABAAAAgEER6Gy0bt06Pf300woMDJS/v78mTJig9PT0PLeLj4/X1q1b70ELC8aFCxfk5eWlsWPHWpbFxMRo0KBBOcreanleMjIyNH36dAUEBKhTp07q3LmzBg0apD/++OOu2l6YpKena8KECfL391dgYKC6dOmidevWSbq2D4WGhlqVP3TokPz8/HLUM2PGDHl6eurQoUNWy48cOSJPT0/NnTvXavnJkyfVr18/BQcHKzg4WKGhoTm2vZ/87ziHhITo+++/l3RtnBs0aKCQkBDLv2HDhlnW3fh/4OnpqbS0NElS7969tWHDhnvfmXvEz89PgYGB6ty5s9q1a6dXXnlFu3fvtqyfOXOm/P395e/vr5kzZ9pc77Zt2xQaGqpOnTqpU6dOOnjwYH403xCmT5+e5/dJ79691bZtW6v98/r/g5+f3y1/b3Nbdz+53s8RI0aodevWCgkJUUBAgD788ENLGU9PTwUHB6tz584KDg7WDz/8YFk3ffp0TZw48aZ1x8TEyNPTU3FxcVbL7uT77EF14/j+72dtWFhYAbfu3rrx+yMiIkLHjx+XJI0YMUJRUVGSru1fjRs3VkhIiDp27Kh+/fopKSnJqh6z2axWrVrplVdeubcdKGQSExPVu3dvSdL58+fVunVr7du3z7J+1qxZGjhwoK5cuaJ69erp7NmzlnWhoaFWv8f79++Xj4+PpGv7rKenp/bu3WtZn9vnREErUtANMIKdO3fqvffe0+zZs+Xp6amrV69q+PDhioyM1Lhx43LddseOHbp8+bJatmx5j1p7b8XGxqpBgwZatWqVhg0bJicnJ7u/x8iRI/XPP/8oOjpapUuXVnZ2tjZt2qSjR4+qVq1adn+/gjBmzBhdvnxZq1atkrOzsw4dOqQXX3xRDz/8sM11/Pbbb/rll19UuXLlHOu+/fZbNWvWTEuXLtXzzz9vWR4ZGanWrVurV69ekq59MBYpcv9+LNxsnPv27SsXFxc5OjqqZs2aiomJKehmFjrTpk3To48+Kklau3at+vXrpzlz5ig9PV2rV69WbGysJCksLExNmjTRk08+mWt9iYmJevvtt/Xll1+qRo0a+ueff5SZmZnv/SisZsyYoT59+uT5+fnOO+/I19f3HrXKuPr166devXrp0qVLCgkJkbe3t9q2bStJWrx4sR566CFt2rRJr7/+unbu3GnTZ17lypX1ySefqH379vf1Z+S9wmftNbNnz77luubNm2vatGnKzs7WG2+8oRkzZigyMtKyfsuWLapQoYJ2796ts2fPytXV9V40udBxc3PTggULJEkuLi4aNWqURo4cqe+++05Hjx7Vv//9by1btkzFixeXl5eXduzYoY4dOyo1NVX//POP1QGvHTt2qEmTJpbXlStX1pQpUzR//vx73q/b9cCdofv00081fvx4y+tz586padOmGjRokOXIiGR9pGT69Ol65ZVX5OnpKUlydnbWmDFjFBcXp5MnT+rIkSPy8fHRyZMnJV37ch48eLB+//13LV68WMuWLVNISIi++OKLe9jTe2Pp0qXq37+/PD09rY525iUhIUGhoaGWI55r1qyxnJmaNWuW5QjWsWPHtG7dOo0bN06lS5eWJDk4OKhNmzZq165dvvTpXjt58qS+//57jRkzRs7OzpKkRx99VC+//LJmzJhhUx3p6el67733NGbMmBzrMjMztXLlSr333nu6evWq1ZGr06dPy83NzfLazc1N5cqVu7sOFVK3GudXXnnF5nGG1L59e/Xo0UNz5sxRXFycunTpomLFiqlYsWLq0qWL5Xf60qVLGjRokAIDA9W7d28NHTrUcmRz4cKFCgkJUY0aNSRJxYoVU8mSJSVJrVq1UnJysqRrR6/79esnSUpOTlbr1q3vdXfz3fU/0Hr06KGQkBC1a9fOcuaiRYsWGjBggE31rFixQqGhoWrXrp3Vd9mDrFSpUqpfv76OHj2aY13Tpk11+fJlXbx40bIsMTFRERERCgwMVL9+/XTlyhXLunr16umRRx7Rt99+e0/aXhisXbvWMpNh5syZlu/lW31f3+rvq8uXLxdgLwo3W86gOzg46Mknn1RCQoLV8qVLl6pHjx7y9/fXsmXL8rOZhcbevXvVu3dvhYaGKjQ0VBs3btSJEyfUtGlTSxl/f3/VqVNHH374oUaMGKGRI0da/q5p0qSJ4uPjJUm7du1S48aNVb16dR0+fFjStUB3Y13t27fX+fPntWXLlnvYyzvzwAW6639wXD8aHBsbKz8/P5UoUeKW2/z+++9q2LCh1TIXFxdVqVJFhw4dUs2aNTV48GANHjxYW7du1cqVK/X+++/L09NTPXr0UJcuXbR8+XLLHyb3i4MH/1979x5TZf0HcPx9BghMEMXCJBHyBqHjYigHZxuB0yMUcEjLrHCRKIph0ooNEHA1WDlNKi6jDLUtGnIOHMfqlEXguMoERcd+GDZMUmSMWBgSxyO/P8545iPgJTPh8H1t/PPceJ4vj8/39vl8/R+9vb0olUoiIyPRaDT3fN6WLVtISkoiJCSE7u5uUlNTycvLo7S0FBsbG+nYlpYWXF1d72umaqI5f/48c+fOZfr06bLtPj4+UhjahQsXZKFWt4f6ZGVlERYWxpw5c0Zcv6KiAldXV1xdXVGr1bK/0+bNm3nvvfd47bXX2Lt3r6yzZ27uVM7DFert5Sw6eqPz9vamra2NK1eu4OzsLG2fPXu21OjIzs5m6tSp6PV6srKyaGhokI5ra2vjr7/+IioqioiICDIzM6WQQ39/f+rq6jAYDHR0dNDR0YHBYKC2tlZW0ZqLtLQ0wDRzpNPpOH78ODqdjtzcXGxtbYmJiZGO/eCDD2Tv53DHF0wdXq1WS2FhIXl5eZM6hHXY1atXaWxsxNPTc8S+48ePo1QqcXR0lLadO3eOffv28d1330kDYbfatWsXubm5DAwMPPR7f9S6u7vZvXs3OTk56HQ6afa4t7d3zPr6ftpX7e3tqNVq1q9fT0lJyX/zUBPU4OAgJ06cICQkRNrW09NDXV0da9euJTIyclLMdv7555+kpaWxb98+tFoteXl5pKam0tfXN+LY3bt3U1xczJNPPikrN39/f06ePAmYQn+XL1+On58fJ0+exGg0curUKVk9o1AoSEhI4OOPP2ZoaOjhP+QDmHQdOmdnZxYsWEBlZSUAJSUlI/KT/omIiAjmzZtHXFwc+/btk0abzVlxcTHh4eEoFApWr15Nc3MzV69eveM5ra2t7NixgwMHDuDn5weYRlw8PT1xc3MD4MUXXxzz/La2Nik34ta8vYnsXj4S8+fPR6fTST+ffPKJtK+pqYlz586xcePGUc/VaDSo1WrA9J7q9Xr+/vtvAMLCwigvL2fjxo1cv36dTZs2SeFz5uZO5axQKICR5bxjx47/6vYmlHt5Z+vr61m3bh0Ajo6Oshl1o9FIY2Mj2dnZFBUVceXKFSmCISAggJqaGs6cOYOPjw9eXl6cOXOGmpoalErlw3mgcaavr4/Y2Fjeffdd2WBiSkqK7P28dTZ9uKwfe+wxAgMDpUbLZJSfn094eDjbtm1j8+bNrFixQtq3YcMGgoKCSE5O5u2335adt3LlSqZNm4ZCocDLy0vKbRrm7u7OsmXLpPAuczZcLw/Por/88suAaUB2rPr6XttXixcvprKykpKSEvbv3092djY1NTUP+YkmnpqaGsLDwwkICKCnp4e1a9dK+44dO8Zzzz2HnZ0dzzzzDEajkaampkd4tw9fU1MTHR0dxMTEEB4eTkxMDAqFYtRw/draWuzs7Pj1119l+cm+vr50dHTQ3d1NQ0MDy5cvl2btWlpasLe3x8XFRXatwMBArK2tpXz78WrSdegA1Go1paWltLAYJCIAAAefSURBVLa20tfXh5+fHxYWFty8eVM6ZrjBC6aP+OnTp2XX6O3t5dKlSyxcuBAwjaD88ssv2Nvby0ZNzdXg4CBlZWVoNBqCgoIICQnBYDDcdZRo1qxZODg4SFPed+Pp6cnFixelsJgFCxag0+l4/fXXuXbt2gM/x3iwaNEifvvtN3p7e2XbT58+LYX53klDQwMXLlwgODiYoKAgOjs7efPNN6mqqqK7u5uqqio+++wzgoKCePXVV7l+/Trff/+9dP6MGTMICQkhNTWVbdu2mW2H7k7l7Ovr+4juamI6e/YsCxcuZPbs2Vy+fFnafuXKFWbPnn3X852dnQkMDMTe3p4pU6agUqk4e/YsAEqlktraWmpra1EqlSiVSurq6qirqyMgIOChPdN4YTAYeOutt1Cr1axZs+ZR386EtGXLFnQ6HVqtlk2bNsn2ffPNN5SXl7Nz504SEhJkdf1wKDaAhYUFRqNxxLV37tzJoUOHRp0VEEZvX93Ozs4Oe3t7AFxcXFi1apVsoSXBZMWKFeh0OiorK1EoFGRlZUn7NBoN1dXVBAUFERQURE9Pzz1HSU1UQ0NDuLu7ywa1KisrmTFjhuy4np4eMjIyyM/PZ8mSJbIBcBsbG7y9vfn555/p7+/HyckJT09PWlpaRuTP3eqdd94hKytrXOd6T8oO3erVq2loaKCgoAC1Wo1CocDV1VVqUHR1dck6HHFxceTm5tLa2gqYOnvp6emoVCopxO2jjz5i8eLFFBQUkJaWRmdnJ2D6cJnjh/+nn37iqaee4sSJE5SXl1NeXs6XX35519CJ6dOnc+jQIY4dO0ZBQQFgCt9qaWmRRkNvvYabmxvBwcGkpKTIytGcYvLnzJmDSqUiPT1dalycP3+evLy8e5oh2rJlC1VVVdLf4YknnuDgwYOsXLmS0tJS1qxZQ0VFhbQ/IyND+vBXVFRIv9NoNNLa2jpq2KY5GKucDx8+PGKkXhjbjz/+SGFhIdHR0ahUKkpLSxkYGGBgYIDS0lJpFFmpVEoDPH/88Ye0aivA888/T319PYODgwwNDVFVVYWHhwdgSkK3sLCgpKSEgIAAAgIC0Gq1WFpaysI7zcnUqVOlAarU1FTmzZvHG2+8cV/XGP5u9vT0UFlZaZbhqf+m6OhoZs6cSWFh4X2d5+Liwpo1azh8+PBDurPxwcfHh5aWFtrb2wE4evQoAB4eHmPW1zB6++p2XV1d0ix/b28v1dXV0r9/YSQ7Ozv27NlDYWEhXV1dNDc309fXJ6v3y8rK0Ov1srxPc+Pr68vFixepq6uTtjU3N4+IGNmzZw8vvfQSHh4eJCcnU1ZWJrXvwZRH98UXX7B06VIALC0tmTt3LkVFRWN+N4dz7W4Pwx5PJuVSTba2tgQHB6PVaqWFPNavX098fDwhISG4ubnh5eUlHe/v709KSgqJiYkMDAxgMBgIDg4mISEBMDVwTp48ydGjR7G2tiYuLo6EhASOHDkiJauGh4cTGhpqNnl0Go2GF154QbbN19eXmzdvcvnyZSorK2ULGERGRjJ37lzAlKh+8OBBYmNj6e/vJy4ujvT0dGJiYrC1tSUwMBArKytsbW0ByMzMJCcnh3Xr1mFpacm0adNwcnIym7IEUx7N/v37CQkJwcrKCmtra5KTk2UJvP+EVqslMTFRti04OJi0tDQ6Ojqor6/nww8/xNLSEqPRyJIlS9i5c+eDPs64dWs5KxQKrl69SlFREU8//TT19fVSDt0wJyenO65CNlnEx8czZcoUrl+/zvz588nPz8fb2xswNeBCQ0MBU0jv8Ajn9u3bSUpKQqVS8fjjj8tG6pcuXcqzzz5LREQEFhYWeHp6snXrVml/QEAAp06dwsnJCTCNqo420m8uoqOjiYqKwmAw8Pvvv7No0SLpPfT39ycpKQkw5dAdOHBAOi8+Pl5avXHGjBlERkbS19fH1q1b72l235zcuHFDNsN2NwqFgsTERHbt2sWGDRvu63dt377d7PO+Zs6cyfvvv09sbCw2NjasXr0aMA3K3qm+Hq19dbsffviBwsJCqd6JiIhg1apV/9mzjTfDnd47vcMeHh6oVCo+//xzBgcHCQ0NlXWWZ82ahaenJ3q9XkqxMDcODg7k5OSwd+9eMjIyMBgMuLi4sHv3bumYb7/9lvb2dum/K3FwcCA1NZWkpCQ0Gg1TpkzB39+f7Oxs2YJTy5Yto7q6eswZOjDl0I7nslUMjfcsP2FSuHbtmpR3qNFoKC4uvu+RU0G4HwaDgdTUVDo7O8nLy7uvxqBw/z799FP6+/tHDDAIwoPq6upi7dq1VFdXyxbpEP5d7u7uNDY2MjQ0JOrrf8GNGzfw9/dHr9fj6OiIWq0mJSXljp0KQRjLpJyhE8afr776Cr1ej9FoxMHBwWwWPBHGLysrKzIzMx/1bQiC8ACOHDnC119/TWJioujM/UdEff3guru7eeWVVwgLC+PSpUtERUXh5+cnOnPCPyZm6ARBEARBEARBECaoSbkoiiAIgiAIgiAIgjkQHTpBEARBEARBEIQJSnToBEEQBEEQBEEQJijRoRMEQRAEQRAEQZigRIdOEARBEARBEARhghIdOkEQBEEQBEEQhAnq/9qDRoCeJ1AFAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1080x432 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"sns.set(style=\"whitegrid\")\n",
"\n",
"jobs = nc.list_trial_jobs()\n",
"job_ids = [x['id'] for x in jobs]\n",
"final_metrics = [float(x['finalMetricData'][0]['data']) for x in jobs]\n",
"\n",
"data = {'job id': job_ids, 'final metrics': final_metrics}\n",
"sns.set(rc={'figure.figsize':(15, 6)})\n",
"\n",
"plt.title('Trial job final results')\n",
"ax = sns.barplot(x='job id', y='final metrics', data=data) \n",
"\n",
"for i,p in enumerate(ax.patches):\n",
" ax.annotate('{:.4f}'.format(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()),\n",
" ha='center', va='center', fontsize=11, color='black', rotation=0, xytext=(0, 5),\n",
" textcoords='offset points') "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stop nni experiment"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO: Stoping experiment PlUIfDTR\n",
"INFO: Stop experiment success.\n"
]
}
],
"source": [
"nc.stop_nni()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
setup.py
View file @ c1811162
......@@ -35,9 +35,10 @@ setup(
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('tools'),
packages = find_packages('src/sdk/pynni', exclude=['tests']) + find_packages('src/sdk/pycli') + find_packages('tools'),
package_dir = {
'nni': 'src/sdk/pynni/nni',
'nnicli': 'src/sdk/pycli/nnicli',
'nni_annotation': 'tools/nni_annotation',
'nni_cmd': 'tools/nni_cmd',
'nni_trial_tool':'tools/nni_trial_tool',
......
src/sdk/pycli/nnicli/__init__.py 0 → 100644
View file @ c1811162
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
from .nni_client import *
src/sdk/pycli/nnicli/nni_client.py 0 → 100644
View file @ c1811162
# Copyright (c) Microsoft Corporation. All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
# associated documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish, distribute,
# sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
# NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
# OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# ==================================================================================================
""" A python wrapper for nni rest api
Example:
import nnicli as nc
nc.start_nni('../../../../examples/trials/mnist/config.yml')
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_experiment_status())
print(nc.get_job_statistics())
print(nc.list_trial_jobs())
nc.stop_nni()
"""
import sys
import os
import subprocess
import requests
__all__ = [
'start_nni',
'stop_nni',
'set_endpoint',
'version',
'get_experiment_status',
'get_experiment_profile',
'get_trial_job',
'list_trial_jobs',
'get_job_statistics',
'get_job_metrics',
'export_data'
]
EXPERIMENT_PATH = 'experiment'
VERSION_PATH = 'version'
STATUS_PATH = 'check-status'
JOB_STATISTICS_PATH = 'job-statistics'
TRIAL_JOBS_PATH = 'trial-jobs'
METRICS_PATH = 'metric-data'
EXPORT_DATA_PATH = 'export-data'
API_ROOT_PATH = 'api/v1/nni'
_api_endpoint = None
def set_endpoint(endpoint):
"""set endpoint of nni rest server for nnicli, for example:
http://localhost:8080
"""
global _api_endpoint
_api_endpoint = endpoint
def _check_endpoint():
if _api_endpoint is None:
raise AssertionError("Please call set_endpoint to specify nni endpoint")
def _nni_rest_get(api_path, response_type='json'):
_check_endpoint()
uri = '{}/{}/{}'.format(_api_endpoint, API_ROOT_PATH, api_path)
res = requests.get(uri)
if _http_succeed(res.status_code):
if response_type == 'json':
return res.json()
elif response_type == 'text':
return res.text
else:
raise AssertionError('Incorrect response_type')
else:
return None
def _http_succeed(status_code):
return status_code // 100 == 2
def _create_process(cmd):
if sys.platform == 'win32':
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
else:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while process.poll() is None:
output = process.stdout.readline()
if output:
print(output.decode('utf-8').strip())
return process.returncode
def start_nni(config_file):
"""start nni experiment with specified configuration file"""
cmd = 'nnictl create --config {}'.format(config_file).split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to start nni.')
def stop_nni():
"""stop nni experiment"""
cmd = 'nnictl stop'.split(' ')
if _create_process(cmd) != 0:
raise RuntimeError('Failed to stop nni.')
def version():
"""return version of nni"""
return _nni_rest_get(VERSION_PATH, 'text')
def get_experiment_status():
"""return experiment status as a dict"""
return _nni_rest_get(STATUS_PATH)
def get_experiment_profile():
"""return experiment profile as a dict"""
return _nni_rest_get(EXPERIMENT_PATH)
def get_trial_job(trial_job_id):
"""return trial job information as a dict"""
assert trial_job_id is not None
return _nni_rest_get(os.path.join(TRIAL_JOBS_PATH, trial_job_id))
def list_trial_jobs():
"""return information for all trial jobs as a list"""
return _nni_rest_get(TRIAL_JOBS_PATH)
def get_job_statistics():
"""return trial job statistics information as a dict"""
return _nni_rest_get(JOB_STATISTICS_PATH)
def get_job_metrics(trial_job_id=None):
"""return trial job metrics"""
api_path = METRICS_PATH if trial_job_id is None else os.path.join(METRICS_PATH, trial_job_id)
return _nni_rest_get(api_path)
def export_data():
"""return exported information for all trial jobs"""
return _nni_rest_get(EXPORT_DATA_PATH)
src/sdk/pycli/setup.py 0 → 100644
View file @ c1811162
import setuptools
setuptools.setup(
name = 'nnicli',
version = '999.0.0-developing',
packages = setuptools.find_packages(),
python_requires = '>=3.5',
install_requires = [
'requests'
],
author = 'Microsoft NNI Team',
author_email = 'nni@microsoft.com',
description = 'nnicli for Neural Network Intelligence project',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
)
test/cli_test.py 0 → 100644
View file @ c1811162
# Copyright (c) Microsoft Corporation
# All rights reserved.
#
# MIT License
#
# Permission is hereby granted, free of charge,
# to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and
# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import sys
import time
import traceback
from utils import GREEN, RED, CLEAR, setup_experiment
def test_nni_cli():
import nnicli as nc
config_file = 'config_test/examples/mnist.test.yml'
try:
# Sleep here to make sure previous stopped exp has enough time to exit to avoid port conflict
time.sleep(6)
print(GREEN + 'Testing nnicli:' + config_file + CLEAR)
nc.start_nni(config_file)
time.sleep(3)
nc.set_endpoint('http://localhost:8080')
print(nc.version())
print(nc.get_job_statistics())
print(nc.get_experiment_status())
nc.list_trial_jobs()
print(GREEN + 'Test nnicli {}: TEST PASS'.format(config_file) + CLEAR)
except Exception as error:
print(RED + 'Test nnicli {}: TEST FAIL'.format(config_file) + CLEAR)
print('%r' % error)
traceback.print_exc()
raise error
finally:
nc.stop_nni()
if __name__ == '__main__':
installed = (sys.argv[-1] != '--preinstall')
setup_experiment(installed)
test_nni_cli()
test/pipelines-it-local-windows.yml
View file @ c1811162
......@@ -36,3 +36,7 @@ jobs:
cd test
python metrics_test.py
displayName: 'Trial job metrics test'
- script: |
cd test
PATH=$HOME/.local/bin:$PATH python3 cli_test.py
displayName: 'nnicli test'
test/pipelines-it-local.yml
View file @ c1811162
......@@ -37,3 +37,7 @@ jobs:
cd test
PATH=$HOME/.local/bin:$PATH python3 metrics_test.py
displayName: 'Trial job metrics test'
- script: |
cd test
PATH=$HOME/.local/bin:$PATH python3 cli_test.py
displayName: 'nnicli test'
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment