Skip to content

GitLab

"include/ck/utility/statically_indexed_array.hpp" did not exist on "64350affc5767e7ce3fb211d8145b5c9d18017d8"
Unverified Commit 464e33fe authored by J-shang's avatar J-shang Committed by GitHub
Browse files

add python api example and update doc (#3396)

parent a8f05579
Hide whitespace changes
Inline Side-by-side
Showing with 933 additions and 35 deletions
dependencies/required.txt
View file @ 464e33fe
......@@ -13,6 +13,7 @@ scikit-learn >= 0.23.2
websockets
filelock
prettytable
ipython
dataclasses ; python_version < "3.7"
numpy < 1.19.4 ; sys_platform == "win32"
numpy < 1.20 ; sys_platform != "win32" and python_version < "3.7"
......
docs/en_US/Tutorial/HowToLaunchFromPython.rst
View file @ 464e33fe
**How to Launch an experiment from Python**
**How to Launch an Experiment from Python**
===========================================
.. toctree::
:hidden:
Start Usage <python_api_start>
Connect Usage <python_api_connect>
Overview
--------
Since ``nni v2.0``, we provide a new way to launch experiments. Before that, you need to configure the experiment in the yaml configuration file and then use the experiment ``nnictl`` command to launch the experiment. Now, you can also configure and run experiments directly in python file. If you are familiar with python programming, this will undoubtedly bring you more convenience.
How to Use
----------
Run a New Experiment
--------------------
After successfully installing ``nni``, you can start the experiment with a python script in the following 3 steps.
..
......@@ -59,9 +65,11 @@ See `parameter configuration <../reference/experiment_config.rst>`__ required by
Now, you have successfully launched an NNI experiment. And you can type ``localhost:8081`` in your browser to observe your experiment in real time.
.. Note:: In this way, experiment will run in the foreground and will automatically exit when the experiment finished. If you want to run an experiment in an interactive way, use ``start()`` in Step 3.
Example
-------
Below is an example for this new launching approach. You can also find this code in :githublink:`mnist-tfv2/launch.py <examples/trials/mnist-tfv2/launch.py>` .
^^^^^^^
Below is an example for this new launching approach. You can also find this code in :githublink:`mnist-tfv2/launch.py <examples/trials/mnist-tfv2/launch.py>`.
.. code-block:: python
......@@ -90,6 +98,23 @@ Below is an example for this new launching approach. You can also find this code
experiment.run(8081)
Start and Manage a New Experiment
---------------------------------
We migrate the API in ``NNI Client`` to this new launching approach.
Launch the experiment by ``start()`` instead of ``run()``, then you can use these APIs in interactive mode.
Please refer to `example usage <./python_api_start.rst>`__ and code file :githublink:`python_api_start.ipynb <examples/trials/sklearn/classification/python_api_start.ipynb>`.
.. Note:: ``run()`` polls the experiment status and will automatically call ``stop()`` when the experiment finished. ``start()`` just launched a new experiment, so you need to manually stop the experiment by calling ``stop()``.
Connect and Manage an Exist Experiment
--------------------------------------
If you launch the experiment by ``nnictl`` and also want to use these APIs, you can use ``Experiment.connect()`` to connect to an existing experiment.
Please refer to `example usage <./python_api_connect.rst>`__ and code file :githublink:`python_api_connect.ipynb <examples/trials/sklearn/classification/python_api_connect.ipynb>`.
.. Note:: You can use ``stop()`` to stop the experiment when connecting to an existing experiment.
API
---
......
docs/en_US/Tutorial/python_api_connect.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "white-electron",
"metadata": {},
"source": [
"## Connect and Manage an Exist Experiment"
]
},
{
"cell_type": "markdown",
"id": "recent-italic",
"metadata": {},
"source": [
"### 1. Connect Experiment"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "statistical-repair",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:38] Tuner not set, wait for connect...\n",
"[2021-02-25 07:50:38] Connect to port 8080 success, experiment id is IF0JnfLE, status is RUNNING.\n"
]
}
],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment.connect(8080)"
]
},
{
"cell_type": "markdown",
"id": "defensive-scratch",
"metadata": {},
"source": [
"### 2. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "independent-touch",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 6,\n",
" 'execDuration': 28,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 5,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "printable-bookmark",
"metadata": {},
"outputs": [],
"source": [
"experiment.update_max_trial_number(10)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "marine-serial",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 8,\n",
" 'execDuration': 32,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"id": "opened-lounge",
"metadata": {},
"source": [
"### 3. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "emotional-machinery",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:49] Stopping experiment, please wait...\n",
"[2021-02-25 07:50:49] Experiment stopped\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/en_US/Tutorial/python_api_start.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "technological-script",
"metadata": {},
"source": [
"## Start and Manage a New Experiment"
]
},
{
"cell_type": "markdown",
"id": "immediate-daily",
"metadata": {},
"source": [
"### 1. Initialize Tuner"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "formed-grounds",
"metadata": {},
"outputs": [],
"source": [
"from nni.algorithms.hpo.gridsearch_tuner import GridSearchTuner\n",
"tuner = GridSearchTuner()"
]
},
{
"cell_type": "markdown",
"id": "reported-somerset",
"metadata": {},
"source": [
"### 2. Configure Search Space"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "potential-williams",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"C\": {\"_type\":\"quniform\",\"_value\":[0.1, 1, 0.1]},\n",
" \"kernel\": {\"_type\":\"choice\",\"_value\":[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]},\n",
" \"degree\": {\"_type\":\"choice\",\"_value\":[1, 2, 3, 4]},\n",
" \"gamma\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]},\n",
" \"coef0\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]}\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "greek-archive",
"metadata": {},
"source": [
"### 3. Configure Experiment "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fiscal-expansion",
"metadata": {},
"outputs": [],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment(tuner, 'local')\n",
"experiment.config.experiment_name = 'test'\n",
"experiment.config.trial_concurrency = 2\n",
"experiment.config.max_trial_number = 5\n",
"experiment.config.search_space = search_space\n",
"experiment.config.trial_command = 'python3 main.py'\n",
"experiment.config.trial_code_directory = './'"
]
},
{
"cell_type": "markdown",
"id": "received-tattoo",
"metadata": {},
"source": [
"### 4. Start Experiment"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "pleasant-patent",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:27:11] Creating experiment, Experiment ID: bj025qo4\n",
"[2021-02-22 12:27:11] Connecting IPC pipe...\n",
"[2021-02-22 12:27:15] Statring web server...\n",
"[2021-02-22 12:27:16] Setting up...\n",
"[2021-02-22 12:27:16] Dispatcher started\n",
"[2021-02-22 12:27:16] Web UI URLs: http://127.0.0.1:8081 http://10.0.1.5:8081 http://172.17.0.1:8081\n"
]
}
],
"source": [
"experiment.start(8081)"
]
},
{
"cell_type": "markdown",
"id": "miniature-prison",
"metadata": {},
"source": [
"### 5. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "animated-english",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'RUNNING'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_status()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "alpha-ottawa",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[TrialResult(parameter={'coef0': 0.01, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='B55mT'),\n",
" TrialResult(parameter={'coef0': 0.02, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='QkhD0')]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.export_data()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "unique-rendering",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'B55mT': [TrialMetricData(timestamp=1613996853005, trialJobId='B55mT', parameterId='0', type='FINAL', sequence=0, data=0.9866666666666667)],\n",
" 'QkhD0': [TrialMetricData(timestamp=1613996853843, trialJobId='QkhD0', parameterId='1', type='FINAL', sequence=0, data=0.9866666666666667)]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_job_metrics()"
]
},
{
"cell_type": "markdown",
"id": "welsh-difference",
"metadata": {},
"source": [
"### 6. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "technological-cleanup",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:28:16] Stopping experiment, please wait...\n",
"[2021-02-22 12:28:16] Dispatcher exiting...\n",
"[2021-02-22 12:28:17] Experiment stopped\n",
"[2021-02-22 12:28:19] Dispatcher terminiated\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/en_US/conf.py
View file @ 464e33fe
......@@ -47,6 +47,7 @@ extensions = [
'sphinx.ext.intersphinx',
'nbsphinx',
'sphinx.ext.extlinks',
'IPython.sphinxext.ipython_console_highlighting',
]
# Add mock modules
......@@ -72,7 +73,7 @@ language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md', '**.ipynb_checkpoints']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
......
examples/trials/sklearn/classification/python_api_connect.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "white-electron",
"metadata": {},
"source": [
"## Connect and Manage an Exist Experiment"
]
},
{
"cell_type": "markdown",
"id": "recent-italic",
"metadata": {},
"source": [
"### 1. Connect Experiment"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "statistical-repair",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:38] Tuner not set, wait for connect...\n",
"[2021-02-25 07:50:38] Connect to port 8080 success, experiment id is IF0JnfLE, status is RUNNING.\n"
]
}
],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment.connect(8080)"
]
},
{
"cell_type": "markdown",
"id": "defensive-scratch",
"metadata": {},
"source": [
"### 2. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "independent-touch",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 6,\n",
" 'execDuration': 28,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 5,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "printable-bookmark",
"metadata": {},
"outputs": [],
"source": [
"experiment.update_max_trial_number(10)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "marine-serial",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 8,\n",
" 'execDuration': 32,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"id": "opened-lounge",
"metadata": {},
"source": [
"### 3. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "emotional-machinery",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:49] Stopping experiment, please wait...\n",
"[2021-02-25 07:50:49] Experiment stopped\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/trials/sklearn/classification/python_api_start.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "technological-script",
"metadata": {},
"source": [
"## Start and Manage a New Experiment"
]
},
{
"cell_type": "markdown",
"id": "immediate-daily",
"metadata": {},
"source": [
"### 1. Initialize Tuner"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "formed-grounds",
"metadata": {},
"outputs": [],
"source": [
"from nni.algorithms.hpo.gridsearch_tuner import GridSearchTuner\n",
"tuner = GridSearchTuner()"
]
},
{
"cell_type": "markdown",
"id": "reported-somerset",
"metadata": {},
"source": [
"### 2. Configure Search Space"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "potential-williams",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"C\": {\"_type\":\"quniform\",\"_value\":[0.1, 1, 0.1]},\n",
" \"kernel\": {\"_type\":\"choice\",\"_value\":[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]},\n",
" \"degree\": {\"_type\":\"choice\",\"_value\":[1, 2, 3, 4]},\n",
" \"gamma\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]},\n",
" \"coef0\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]}\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "greek-archive",
"metadata": {},
"source": [
"### 3. Configure Experiment "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fiscal-expansion",
"metadata": {},
"outputs": [],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment(tuner, 'local')\n",
"experiment.config.experiment_name = 'test'\n",
"experiment.config.trial_concurrency = 2\n",
"experiment.config.max_trial_number = 5\n",
"experiment.config.search_space = search_space\n",
"experiment.config.trial_command = 'python3 main.py'\n",
"experiment.config.trial_code_directory = './'"
]
},
{
"cell_type": "markdown",
"id": "received-tattoo",
"metadata": {},
"source": [
"### 4. Start Experiment"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "pleasant-patent",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:27:11] Creating experiment, Experiment ID: bj025qo4\n",
"[2021-02-22 12:27:11] Connecting IPC pipe...\n",
"[2021-02-22 12:27:15] Statring web server...\n",
"[2021-02-22 12:27:16] Setting up...\n",
"[2021-02-22 12:27:16] Dispatcher started\n",
"[2021-02-22 12:27:16] Web UI URLs: http://127.0.0.1:8081 http://10.0.1.5:8081 http://172.17.0.1:8081\n"
]
}
],
"source": [
"experiment.start(8081)"
]
},
{
"cell_type": "markdown",
"id": "miniature-prison",
"metadata": {},
"source": [
"### 5. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "animated-english",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'RUNNING'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_status()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "alpha-ottawa",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[TrialResult(parameter={'coef0': 0.01, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='B55mT'),\n",
" TrialResult(parameter={'coef0': 0.02, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='QkhD0')]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.export_data()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "unique-rendering",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'B55mT': [TrialMetricData(timestamp=1613996853005, trialJobId='B55mT', parameterId='0', type='FINAL', sequence=0, data=0.9866666666666667)],\n",
" 'QkhD0': [TrialMetricData(timestamp=1613996853843, trialJobId='QkhD0', parameterId='1', type='FINAL', sequence=0, data=0.9866666666666667)]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_job_metrics()"
]
},
{
"cell_type": "markdown",
"id": "welsh-difference",
"metadata": {},
"source": [
"### 6. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "technological-cleanup",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:28:16] Stopping experiment, please wait...\n",
"[2021-02-22 12:28:16] Dispatcher exiting...\n",
"[2021-02-22 12:28:17] Experiment stopped\n",
"[2021-02-22 12:28:19] Dispatcher terminiated\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
nni/experiment/experiment.py
View file @ 464e33fe
......@@ -77,16 +77,6 @@ class Experiment:
"""
...
@overload
def __init__(self) -> None:
"""
Prepare an empty experiment, for `connect_experiment`.
Use `Experiment.connect_experiment` to manage experiment.
"""
...
def __init__(self, tuner=None, config=None, training_service=None):
self.config: Optional[ExperimentConfig] = None
self.id: Optional[str] = None
......@@ -204,7 +194,8 @@ class Experiment:
finally:
self.stop()
def connect_experiment(self, port: int):
@classmethod
def connect(cls, port: int):
"""
Connect to an existing experiment.
......@@ -213,8 +204,17 @@ class Experiment:
port
The port of web UI.
"""
self.port = port
self.get_status()
experiment = Experiment()
experiment.port = port
experiment.id = experiment.get_experiment_profile().get('id')
status = experiment.get_status()
pid = experiment.get_experiment_metadata(experiment.id).get('pid')
if pid is None:
_logger.warning('Get experiment pid failed, can not stop experiment by stop().')
else:
experiment._proc = psutil.Process(pid)
_logger.info('Connect to port %d success, experiment id is %s, status is %s.', port, experiment.id, status)
return experiment
def _experiment_rest_get(self, port: int, api: str) -> Any:
if self.port is None:
......@@ -248,7 +248,7 @@ class Experiment:
Trial job id.
Returns
----------
-------
TrialJob
A `TrialJob` instance corresponding to `trial_job_id`.
"""
......@@ -260,7 +260,7 @@ class Experiment:
Return information for all trial jobs as a list.
Returns
----------
-------
list
List of `TrialJob`.
"""
......@@ -272,7 +272,7 @@ class Experiment:
Return trial job statistics information as a dict.
Returns
----------
-------
dict
Job statistics information.
"""
......@@ -289,7 +289,7 @@ class Experiment:
trial job id. if this parameter is None, all trail jobs' metrics will be returned.
Returns
----------
-------
dict
Each key is a trialJobId, the corresponding value is a list of `TrialMetricData`.
"""
......@@ -309,19 +309,46 @@ class Experiment:
Return experiment profile as a dict.
Returns
----------
-------
dict
The profile of the experiment.
"""
resp = self._experiment_rest_get(self.port, '/experiment')
return resp
def get_experiment_metadata(self, exp_id: str):
"""
Return experiment metadata with specified exp_id as a dict.
Returns
-------
dict
The specified experiment metadata.
"""
experiments_metadata = self.get_all_experiments_metadata()
for metadata in experiments_metadata:
if metadata['id'] == exp_id:
return metadata
return {}
def get_all_experiments_metadata(self):
"""
Return all experiments metadata as a list.
Returns
-------
list
The experiments metadata.
"""
resp = self._experiment_rest_get(self.port, '/experiments-info')
return resp
def export_data(self):
"""
Return exported information for all trial jobs.
Returns
----------
-------
list
List of `TrialResult`.
"""
......@@ -329,13 +356,13 @@ class Experiment:
return [TrialResult(**trial_result) for trial_result in resp]
def _get_query_type(self, key: str):
if key == 'trial_concurrency':
if key == 'trialConcurrency':
return '?update_type=TRIAL_CONCURRENCY'
if key == 'max_experiment_duration':
if key == 'maxExecDuration':
return '?update_type=MAX_EXEC_DURATION'
if key == 'search_space':
if key == 'searchSpace':
return '?update_type=SEARCH_SPACE'
if key == 'max_trial_number':
if key == 'maxTrialNum':
return '?update_type=MAX_TRIAL_NUM'
def _update_experiment_profile(self, key: str, value: Any):
......@@ -363,7 +390,7 @@ class Experiment:
value: int
New trial_concurrency value.
"""
self._update_experiment_profile('trial_concurrency', value)
self._update_experiment_profile('trialConcurrency', value)
def update_max_experiment_duration(self, value: str):
"""
......@@ -375,7 +402,7 @@ class Experiment:
Strings like '1m' for one minute or '2h' for two hours.
SUFFIX may be 's' for seconds, 'm' for minutes, 'h' for hours or 'd' for days.
"""
self._update_experiment_profile('max_experiment_duration', value)
self._update_experiment_profile('maxExecDuration', value)
def update_search_space(self, value: dict):
"""
......@@ -387,9 +414,9 @@ class Experiment:
value: dict
New search_space.
"""
self._update_experiment_profile('search_space', value)
self._update_experiment_profile('searchSpace', value)
def update_max_trial_number(self, value):
def update_max_trial_number(self, value: int):
"""
Update an experiment's max_trial_number
......@@ -398,4 +425,4 @@ class Experiment:
value: int
New max_trial_number value.
"""
self._update_experiment_profile('max_trial_number', value)
self._update_experiment_profile('maxTrialNum', value)
test/nni_test/nnitest/validators.py
View file @ 464e33fe
......@@ -93,7 +93,7 @@ class NnicliValidator(ITValidator):
def __call__(self, rest_endpoint, experiment_dir, nni_source_dir, **kwargs):
print(rest_endpoint)
exp = Experiment()
exp.connect_experiment(int(rest_endpoint.split(':')[-1]))
exp.connect(int(rest_endpoint.split(':')[-1]))
print(exp.get_job_statistics())
print(exp.get_experiment_status())
print(exp.list_trial_jobs())
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment