Skip to content

GitLab

Unverified Commit 464e33fe authored by J-shang's avatar J-shang Committed by GitHub
Browse files

add python api example and update doc (#3396)

parent a8f05579
Show whitespace changes
Inline Side-by-side
Showing with 933 additions and 35 deletions
dependencies/required.txt
View file @ 464e33fe
...@@ -13,6 +13,7 @@ scikit-learn >= 0.23.2 ...@@ -13,6 +13,7 @@ scikit-learn >= 0.23.2
websockets websockets
filelock filelock
prettytable prettytable
ipython
dataclasses ; python_version < "3.7" dataclasses ; python_version < "3.7"
numpy < 1.19.4 ; sys_platform == "win32" numpy < 1.19.4 ; sys_platform == "win32"
numpy < 1.20 ; sys_platform != "win32" and python_version < "3.7" numpy < 1.20 ; sys_platform != "win32" and python_version < "3.7"
......
docs/en_US/Tutorial/HowToLaunchFromPython.rst
View file @ 464e33fe
**How to Launch an experiment from Python** **How to Launch an Experiment from Python**
=========================================== ===========================================
.. toctree::
:hidden:
Start Usage <python_api_start>
Connect Usage <python_api_connect>
Overview Overview
-------- --------
Since ``nni v2.0``, we provide a new way to launch experiments. Before that, you need to configure the experiment in the yaml configuration file and then use the experiment ``nnictl`` command to launch the experiment. Now, you can also configure and run experiments directly in python file. If you are familiar with python programming, this will undoubtedly bring you more convenience. Since ``nni v2.0``, we provide a new way to launch experiments. Before that, you need to configure the experiment in the yaml configuration file and then use the experiment ``nnictl`` command to launch the experiment. Now, you can also configure and run experiments directly in python file. If you are familiar with python programming, this will undoubtedly bring you more convenience.
How to Use Run a New Experiment
---------- --------------------
After successfully installing ``nni``, you can start the experiment with a python script in the following 3 steps. After successfully installing ``nni``, you can start the experiment with a python script in the following 3 steps.
.. ..
...@@ -59,9 +65,11 @@ See `parameter configuration <../reference/experiment_config.rst>`__ required by ...@@ -59,9 +65,11 @@ See `parameter configuration <../reference/experiment_config.rst>`__ required by
Now, you have successfully launched an NNI experiment. And you can type ``localhost:8081`` in your browser to observe your experiment in real time. Now, you have successfully launched an NNI experiment. And you can type ``localhost:8081`` in your browser to observe your experiment in real time.
.. Note:: In this way, experiment will run in the foreground and will automatically exit when the experiment finished. If you want to run an experiment in an interactive way, use ``start()`` in Step 3.
Example Example
------- ^^^^^^^
Below is an example for this new launching approach. You can also find this code in :githublink:`mnist-tfv2/launch.py <examples/trials/mnist-tfv2/launch.py>` . Below is an example for this new launching approach. You can also find this code in :githublink:`mnist-tfv2/launch.py <examples/trials/mnist-tfv2/launch.py>`.
.. code-block:: python .. code-block:: python
...@@ -90,6 +98,23 @@ Below is an example for this new launching approach. You can also find this code ...@@ -90,6 +98,23 @@ Below is an example for this new launching approach. You can also find this code
experiment.run(8081) experiment.run(8081)
Start and Manage a New Experiment
---------------------------------
We migrate the API in ``NNI Client`` to this new launching approach.
Launch the experiment by ``start()`` instead of ``run()``, then you can use these APIs in interactive mode.
Please refer to `example usage <./python_api_start.rst>`__ and code file :githublink:`python_api_start.ipynb <examples/trials/sklearn/classification/python_api_start.ipynb>`.
.. Note:: ``run()`` polls the experiment status and will automatically call ``stop()`` when the experiment finished. ``start()`` just launched a new experiment, so you need to manually stop the experiment by calling ``stop()``.
Connect and Manage an Exist Experiment
--------------------------------------
If you launch the experiment by ``nnictl`` and also want to use these APIs, you can use ``Experiment.connect()`` to connect to an existing experiment.
Please refer to `example usage <./python_api_connect.rst>`__ and code file :githublink:`python_api_connect.ipynb <examples/trials/sklearn/classification/python_api_connect.ipynb>`.
.. Note:: You can use ``stop()`` to stop the experiment when connecting to an existing experiment.
API API
--- ---
......
docs/en_US/Tutorial/python_api_connect.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "white-electron",
"metadata": {},
"source": [
"## Connect and Manage an Exist Experiment"
]
},
{
"cell_type": "markdown",
"id": "recent-italic",
"metadata": {},
"source": [
"### 1. Connect Experiment"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "statistical-repair",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:38] Tuner not set, wait for connect...\n",
"[2021-02-25 07:50:38] Connect to port 8080 success, experiment id is IF0JnfLE, status is RUNNING.\n"
]
}
],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment.connect(8080)"
]
},
{
"cell_type": "markdown",
"id": "defensive-scratch",
"metadata": {},
"source": [
"### 2. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "independent-touch",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 6,\n",
" 'execDuration': 28,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 5,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "printable-bookmark",
"metadata": {},
"outputs": [],
"source": [
"experiment.update_max_trial_number(10)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "marine-serial",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 8,\n",
" 'execDuration': 32,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"id": "opened-lounge",
"metadata": {},
"source": [
"### 3. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "emotional-machinery",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:49] Stopping experiment, please wait...\n",
"[2021-02-25 07:50:49] Experiment stopped\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/en_US/Tutorial/python_api_start.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "technological-script",
"metadata": {},
"source": [
"## Start and Manage a New Experiment"
]
},
{
"cell_type": "markdown",
"id": "immediate-daily",
"metadata": {},
"source": [
"### 1. Initialize Tuner"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "formed-grounds",
"metadata": {},
"outputs": [],
"source": [
"from nni.algorithms.hpo.gridsearch_tuner import GridSearchTuner\n",
"tuner = GridSearchTuner()"
]
},
{
"cell_type": "markdown",
"id": "reported-somerset",
"metadata": {},
"source": [
"### 2. Configure Search Space"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "potential-williams",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"C\": {\"_type\":\"quniform\",\"_value\":[0.1, 1, 0.1]},\n",
" \"kernel\": {\"_type\":\"choice\",\"_value\":[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]},\n",
" \"degree\": {\"_type\":\"choice\",\"_value\":[1, 2, 3, 4]},\n",
" \"gamma\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]},\n",
" \"coef0\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]}\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "greek-archive",
"metadata": {},
"source": [
"### 3. Configure Experiment "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fiscal-expansion",
"metadata": {},
"outputs": [],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment(tuner, 'local')\n",
"experiment.config.experiment_name = 'test'\n",
"experiment.config.trial_concurrency = 2\n",
"experiment.config.max_trial_number = 5\n",
"experiment.config.search_space = search_space\n",
"experiment.config.trial_command = 'python3 main.py'\n",
"experiment.config.trial_code_directory = './'"
]
},
{
"cell_type": "markdown",
"id": "received-tattoo",
"metadata": {},
"source": [
"### 4. Start Experiment"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "pleasant-patent",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:27:11] Creating experiment, Experiment ID: bj025qo4\n",
"[2021-02-22 12:27:11] Connecting IPC pipe...\n",
"[2021-02-22 12:27:15] Statring web server...\n",
"[2021-02-22 12:27:16] Setting up...\n",
"[2021-02-22 12:27:16] Dispatcher started\n",
"[2021-02-22 12:27:16] Web UI URLs: http://127.0.0.1:8081 http://10.0.1.5:8081 http://172.17.0.1:8081\n"
]
}
],
"source": [
"experiment.start(8081)"
]
},
{
"cell_type": "markdown",
"id": "miniature-prison",
"metadata": {},
"source": [
"### 5. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "animated-english",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'RUNNING'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_status()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "alpha-ottawa",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[TrialResult(parameter={'coef0': 0.01, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='B55mT'),\n",
" TrialResult(parameter={'coef0': 0.02, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='QkhD0')]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.export_data()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "unique-rendering",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'B55mT': [TrialMetricData(timestamp=1613996853005, trialJobId='B55mT', parameterId='0', type='FINAL', sequence=0, data=0.9866666666666667)],\n",
" 'QkhD0': [TrialMetricData(timestamp=1613996853843, trialJobId='QkhD0', parameterId='1', type='FINAL', sequence=0, data=0.9866666666666667)]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_job_metrics()"
]
},
{
"cell_type": "markdown",
"id": "welsh-difference",
"metadata": {},
"source": [
"### 6. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "technological-cleanup",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:28:16] Stopping experiment, please wait...\n",
"[2021-02-22 12:28:16] Dispatcher exiting...\n",
"[2021-02-22 12:28:17] Experiment stopped\n",
"[2021-02-22 12:28:19] Dispatcher terminiated\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/en_US/conf.py
View file @ 464e33fe
...@@ -47,6 +47,7 @@ extensions = [ ...@@ -47,6 +47,7 @@ extensions = [
'sphinx.ext.intersphinx', 'sphinx.ext.intersphinx',
'nbsphinx', 'nbsphinx',
'sphinx.ext.extlinks', 'sphinx.ext.extlinks',
'IPython.sphinxext.ipython_console_highlighting',
] ]
# Add mock modules # Add mock modules
...@@ -72,7 +73,7 @@ language = None ...@@ -72,7 +73,7 @@ language = None
# List of patterns, relative to source directory, that match files and # List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files. # directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path. # This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'Release_v1.0.md', '**.ipynb_checkpoints']
# The name of the Pygments (syntax highlighting) style to use. # The name of the Pygments (syntax highlighting) style to use.
pygments_style = None pygments_style = None
......
examples/trials/sklearn/classification/python_api_connect.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "white-electron",
"metadata": {},
"source": [
"## Connect and Manage an Exist Experiment"
]
},
{
"cell_type": "markdown",
"id": "recent-italic",
"metadata": {},
"source": [
"### 1. Connect Experiment"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "statistical-repair",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:38] Tuner not set, wait for connect...\n",
"[2021-02-25 07:50:38] Connect to port 8080 success, experiment id is IF0JnfLE, status is RUNNING.\n"
]
}
],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment.connect(8080)"
]
},
{
"cell_type": "markdown",
"id": "defensive-scratch",
"metadata": {},
"source": [
"### 2. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "independent-touch",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 6,\n",
" 'execDuration': 28,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 5,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "printable-bookmark",
"metadata": {},
"outputs": [],
"source": [
"experiment.update_max_trial_number(10)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "marine-serial",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 8,\n",
" 'execDuration': 32,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_experiment_profile()"
]
},
{
"cell_type": "markdown",
"id": "opened-lounge",
"metadata": {},
"source": [
"### 3. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "emotional-machinery",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:49] Stopping experiment, please wait...\n",
"[2021-02-25 07:50:49] Experiment stopped\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/trials/sklearn/classification/python_api_start.ipynb 0 → 100644
View file @ 464e33fe
{
"cells": [
{
"cell_type": "markdown",
"id": "technological-script",
"metadata": {},
"source": [
"## Start and Manage a New Experiment"
]
},
{
"cell_type": "markdown",
"id": "immediate-daily",
"metadata": {},
"source": [
"### 1. Initialize Tuner"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "formed-grounds",
"metadata": {},
"outputs": [],
"source": [
"from nni.algorithms.hpo.gridsearch_tuner import GridSearchTuner\n",
"tuner = GridSearchTuner()"
]
},
{
"cell_type": "markdown",
"id": "reported-somerset",
"metadata": {},
"source": [
"### 2. Configure Search Space"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "potential-williams",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"C\": {\"_type\":\"quniform\",\"_value\":[0.1, 1, 0.1]},\n",
" \"kernel\": {\"_type\":\"choice\",\"_value\":[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]},\n",
" \"degree\": {\"_type\":\"choice\",\"_value\":[1, 2, 3, 4]},\n",
" \"gamma\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]},\n",
" \"coef0\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]}\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "greek-archive",
"metadata": {},
"source": [
"### 3. Configure Experiment "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fiscal-expansion",
"metadata": {},
"outputs": [],
"source": [
"from nni.experiment import Experiment\n",
"experiment = Experiment(tuner, 'local')\n",
"experiment.config.experiment_name = 'test'\n",
"experiment.config.trial_concurrency = 2\n",
"experiment.config.max_trial_number = 5\n",
"experiment.config.search_space = search_space\n",
"experiment.config.trial_command = 'python3 main.py'\n",
"experiment.config.trial_code_directory = './'"
]
},
{
"cell_type": "markdown",
"id": "received-tattoo",
"metadata": {},
"source": [
"### 4. Start Experiment"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "pleasant-patent",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:27:11] Creating experiment, Experiment ID: bj025qo4\n",
"[2021-02-22 12:27:11] Connecting IPC pipe...\n",
"[2021-02-22 12:27:15] Statring web server...\n",
"[2021-02-22 12:27:16] Setting up...\n",
"[2021-02-22 12:27:16] Dispatcher started\n",
"[2021-02-22 12:27:16] Web UI URLs: http://127.0.0.1:8081 http://10.0.1.5:8081 http://172.17.0.1:8081\n"
]
}
],
"source": [
"experiment.start(8081)"
]
},
{
"cell_type": "markdown",
"id": "miniature-prison",
"metadata": {},
"source": [
"### 5. Experiment View & Control"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "animated-english",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'RUNNING'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_status()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "alpha-ottawa",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[TrialResult(parameter={'coef0': 0.01, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='B55mT'),\n",
" TrialResult(parameter={'coef0': 0.02, 'gamma': 0.01, 'degree': 1, 'kernel': 'linear', 'C': 0.1}, value=0.9866666666666667, trialJobId='QkhD0')]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.export_data()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "unique-rendering",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'B55mT': [TrialMetricData(timestamp=1613996853005, trialJobId='B55mT', parameterId='0', type='FINAL', sequence=0, data=0.9866666666666667)],\n",
" 'QkhD0': [TrialMetricData(timestamp=1613996853843, trialJobId='QkhD0', parameterId='1', type='FINAL', sequence=0, data=0.9866666666666667)]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiment.get_job_metrics()"
]
},
{
"cell_type": "markdown",
"id": "welsh-difference",
"metadata": {},
"source": [
"### 6. Stop Experiment"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "technological-cleanup",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-22 12:28:16] Stopping experiment, please wait...\n",
"[2021-02-22 12:28:16] Dispatcher exiting...\n",
"[2021-02-22 12:28:17] Experiment stopped\n",
"[2021-02-22 12:28:19] Dispatcher terminiated\n"
]
}
],
"source": [
"experiment.stop()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "nni-dev",
"language": "python",
"name": "nni-dev"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
nni/experiment/experiment.py
View file @ 464e33fe
...@@ -77,16 +77,6 @@ class Experiment: ...@@ -77,16 +77,6 @@ class Experiment:
""" """
... ...
@overload
def __init__(self) -> None:
"""
Prepare an empty experiment, for `connect_experiment`.
Use `Experiment.connect_experiment` to manage experiment.
"""
...
def __init__(self, tuner=None, config=None, training_service=None): def __init__(self, tuner=None, config=None, training_service=None):
self.config: Optional[ExperimentConfig] = None self.config: Optional[ExperimentConfig] = None
self.id: Optional[str] = None self.id: Optional[str] = None
...@@ -204,7 +194,8 @@ class Experiment: ...@@ -204,7 +194,8 @@ class Experiment:
finally: finally:
self.stop() self.stop()
def connect_experiment(self, port: int): @classmethod
def connect(cls, port: int):
""" """
Connect to an existing experiment. Connect to an existing experiment.
...@@ -213,8 +204,17 @@ class Experiment: ...@@ -213,8 +204,17 @@ class Experiment:
port port
The port of web UI. The port of web UI.
""" """
self.port = port experiment = Experiment()
self.get_status() experiment.port = port
experiment.id = experiment.get_experiment_profile().get('id')
status = experiment.get_status()
pid = experiment.get_experiment_metadata(experiment.id).get('pid')
if pid is None:
_logger.warning('Get experiment pid failed, can not stop experiment by stop().')
else:
experiment._proc = psutil.Process(pid)
_logger.info('Connect to port %d success, experiment id is %s, status is %s.', port, experiment.id, status)
return experiment
def _experiment_rest_get(self, port: int, api: str) -> Any: def _experiment_rest_get(self, port: int, api: str) -> Any:
if self.port is None: if self.port is None:
...@@ -248,7 +248,7 @@ class Experiment: ...@@ -248,7 +248,7 @@ class Experiment:
Trial job id. Trial job id.
Returns Returns
---------- -------
TrialJob TrialJob
A `TrialJob` instance corresponding to `trial_job_id`. A `TrialJob` instance corresponding to `trial_job_id`.
""" """
...@@ -260,7 +260,7 @@ class Experiment: ...@@ -260,7 +260,7 @@ class Experiment:
Return information for all trial jobs as a list. Return information for all trial jobs as a list.
Returns Returns
---------- -------
list list
List of `TrialJob`. List of `TrialJob`.
""" """
...@@ -272,7 +272,7 @@ class Experiment: ...@@ -272,7 +272,7 @@ class Experiment:
Return trial job statistics information as a dict. Return trial job statistics information as a dict.
Returns Returns
---------- -------
dict dict
Job statistics information. Job statistics information.
""" """
...@@ -289,7 +289,7 @@ class Experiment: ...@@ -289,7 +289,7 @@ class Experiment:
trial job id. if this parameter is None, all trail jobs' metrics will be returned. trial job id. if this parameter is None, all trail jobs' metrics will be returned.
Returns Returns
---------- -------
dict dict
Each key is a trialJobId, the corresponding value is a list of `TrialMetricData`. Each key is a trialJobId, the corresponding value is a list of `TrialMetricData`.
""" """
...@@ -309,19 +309,46 @@ class Experiment: ...@@ -309,19 +309,46 @@ class Experiment:
Return experiment profile as a dict. Return experiment profile as a dict.
Returns Returns
---------- -------
dict dict
The profile of the experiment. The profile of the experiment.
""" """
resp = self._experiment_rest_get(self.port, '/experiment') resp = self._experiment_rest_get(self.port, '/experiment')
return resp return resp
def get_experiment_metadata(self, exp_id: str):
"""
Return experiment metadata with specified exp_id as a dict.
Returns
-------
dict
The specified experiment metadata.
"""
experiments_metadata = self.get_all_experiments_metadata()
for metadata in experiments_metadata:
if metadata['id'] == exp_id:
return metadata
return {}
def get_all_experiments_metadata(self):
"""
Return all experiments metadata as a list.
Returns
-------
list
The experiments metadata.
"""
resp = self._experiment_rest_get(self.port, '/experiments-info')
return resp
def export_data(self): def export_data(self):
""" """
Return exported information for all trial jobs. Return exported information for all trial jobs.
Returns Returns
---------- -------
list list
List of `TrialResult`. List of `TrialResult`.
""" """
...@@ -329,13 +356,13 @@ class Experiment: ...@@ -329,13 +356,13 @@ class Experiment:
return [TrialResult(**trial_result) for trial_result in resp] return [TrialResult(**trial_result) for trial_result in resp]
def _get_query_type(self, key: str): def _get_query_type(self, key: str):
if key == 'trial_concurrency': if key == 'trialConcurrency':
return '?update_type=TRIAL_CONCURRENCY' return '?update_type=TRIAL_CONCURRENCY'
if key == 'max_experiment_duration': if key == 'maxExecDuration':
return '?update_type=MAX_EXEC_DURATION' return '?update_type=MAX_EXEC_DURATION'
if key == 'search_space': if key == 'searchSpace':
return '?update_type=SEARCH_SPACE' return '?update_type=SEARCH_SPACE'
if key == 'max_trial_number': if key == 'maxTrialNum':
return '?update_type=MAX_TRIAL_NUM' return '?update_type=MAX_TRIAL_NUM'
def _update_experiment_profile(self, key: str, value: Any): def _update_experiment_profile(self, key: str, value: Any):
...@@ -363,7 +390,7 @@ class Experiment: ...@@ -363,7 +390,7 @@ class Experiment:
value: int value: int
New trial_concurrency value. New trial_concurrency value.
""" """
self._update_experiment_profile('trial_concurrency', value) self._update_experiment_profile('trialConcurrency', value)
def update_max_experiment_duration(self, value: str): def update_max_experiment_duration(self, value: str):
""" """
...@@ -375,7 +402,7 @@ class Experiment: ...@@ -375,7 +402,7 @@ class Experiment:
Strings like '1m' for one minute or '2h' for two hours. Strings like '1m' for one minute or '2h' for two hours.
SUFFIX may be 's' for seconds, 'm' for minutes, 'h' for hours or 'd' for days. SUFFIX may be 's' for seconds, 'm' for minutes, 'h' for hours or 'd' for days.
""" """
self._update_experiment_profile('max_experiment_duration', value) self._update_experiment_profile('maxExecDuration', value)
def update_search_space(self, value: dict): def update_search_space(self, value: dict):
""" """
...@@ -387,9 +414,9 @@ class Experiment: ...@@ -387,9 +414,9 @@ class Experiment:
value: dict value: dict
New search_space. New search_space.
""" """
self._update_experiment_profile('search_space', value) self._update_experiment_profile('searchSpace', value)
def update_max_trial_number(self, value): def update_max_trial_number(self, value: int):
""" """
Update an experiment's max_trial_number Update an experiment's max_trial_number
...@@ -398,4 +425,4 @@ class Experiment: ...@@ -398,4 +425,4 @@ class Experiment:
value: int value: int
New max_trial_number value. New max_trial_number value.
""" """
self._update_experiment_profile('max_trial_number', value) self._update_experiment_profile('maxTrialNum', value)
test/nni_test/nnitest/validators.py
View file @ 464e33fe
...@@ -93,7 +93,7 @@ class NnicliValidator(ITValidator): ...@@ -93,7 +93,7 @@ class NnicliValidator(ITValidator):
def __call__(self, rest_endpoint, experiment_dir, nni_source_dir, **kwargs): def __call__(self, rest_endpoint, experiment_dir, nni_source_dir, **kwargs):
print(rest_endpoint) print(rest_endpoint)
exp = Experiment() exp = Experiment()
exp.connect_experiment(int(rest_endpoint.split(':')[-1])) exp.connect(int(rest_endpoint.split(':')[-1]))
print(exp.get_job_statistics()) print(exp.get_job_statistics())
print(exp.get_experiment_status()) print(exp.get_experiment_status())
print(exp.list_trial_jobs()) print(exp.list_trial_jobs())
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment