Skip to content

GitLab

Unverified Commit fbee0df1 authored by liuzhe-lz's avatar liuzhe-lz Committed by GitHub
Browse files

Fix HPO doc fixme (#4661)

parent f886ae5d
Hide whitespace changes
Inline Side-by-side
Showing with 464 additions and 253 deletions
docs/source/tutorials/hpo_quickstart_pytorch/main.rst
View file @ fbee0df1
...@@ -23,20 +23,22 @@ NNI HPO Quickstart with PyTorch ...@@ -23,20 +23,22 @@ NNI HPO Quickstart with PyTorch
=============================== ===============================
This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning. This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.
There is also a :doc:`TensorFlow version<../hpo_quickstart_tensorflow/main>` if you prefer it.
The tutorial consists of 4 steps: The tutorial consists of 4 steps:
1. Modify the model for auto-tuning. 1. Modify the model for auto-tuning.
2. Define hyperparameters' search space. 2. Define hyperparameters' search space.
3. Configure the experiment. 3. Configure the experiment.
4. Run the experiment. 4. Run the experiment.
.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html .. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
.. GENERATED FROM PYTHON SOURCE LINES 17-28 .. GENERATED FROM PYTHON SOURCE LINES 19-36
Step 1: Prepare the model Step 1: Prepare the model
------------------------- -------------------------
In first step, you need to prepare the model to be tuned. In first step, we need to prepare the model to be tuned.
The model should be put in a separate script. The model should be put in a separate script.
It will be evaluated many times concurrently, It will be evaluated many times concurrently,
...@@ -44,9 +46,15 @@ and possibly will be trained on distributed platforms. ...@@ -44,9 +46,15 @@ and possibly will be trained on distributed platforms.
In this tutorial, the model is defined in :doc:`model.py <model>`. In this tutorial, the model is defined in :doc:`model.py <model>`.
In short, it is a PyTorch model with 3 additional API calls:
1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.
2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.
3. Use :func:`nni.report_final_result` to report final accuracy.
Please understand the model code before continue to next step. Please understand the model code before continue to next step.
.. GENERATED FROM PYTHON SOURCE LINES 30-51 .. GENERATED FROM PYTHON SOURCE LINES 38-59
Step 2: Define search space Step 2: Define search space
--------------------------- ---------------------------
...@@ -57,9 +65,9 @@ Here we need to define their *search space* so the tuning algorithm can sample t ...@@ -57,9 +65,9 @@ Here we need to define their *search space* so the tuning algorithm can sample t
Assuming we have following prior knowledge for these hyperparameters: Assuming we have following prior knowledge for these hyperparameters:
1. *features* should be one of 128, 256, 512, 1024. 1. *features* should be one of 128, 256, 512, 1024.
2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution. 2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
3. *momentum* should be a float between 0 and 1. 3. *momentum* should be a float between 0 and 1.
In NNI, the space of *features* is called ``choice``; In NNI, the space of *features* is called ``choice``;
the space of *lr* is called ``loguniform``; the space of *lr* is called ``loguniform``;
...@@ -70,7 +78,7 @@ For full specification of search space, check :doc:`the reference </hpo/search_s ...@@ -70,7 +78,7 @@ For full specification of search space, check :doc:`the reference </hpo/search_s
Now we can define the search space as follow: Now we can define the search space as follow:
.. GENERATED FROM PYTHON SOURCE LINES 51-58 .. GENERATED FROM PYTHON SOURCE LINES 59-66
.. code-block:: default .. code-block:: default
...@@ -88,7 +96,7 @@ Now we can define the search space as follow: ...@@ -88,7 +96,7 @@ Now we can define the search space as follow:
.. GENERATED FROM PYTHON SOURCE LINES 59-66 .. GENERATED FROM PYTHON SOURCE LINES 67-74
Step 3: Configure the experiment Step 3: Configure the experiment
-------------------------------- --------------------------------
...@@ -98,7 +106,7 @@ The *experiment config* defines how to train the models and how to explore the s ...@@ -98,7 +106,7 @@ The *experiment config* defines how to train the models and how to explore the s
In this tutorial we use a *local* mode experiment, In this tutorial we use a *local* mode experiment,
which means models will be trained on local machine, without using any special training platform. which means models will be trained on local machine, without using any special training platform.
.. GENERATED FROM PYTHON SOURCE LINES 66-69 .. GENERATED FROM PYTHON SOURCE LINES 74-77
.. code-block:: default .. code-block:: default
...@@ -112,20 +120,16 @@ which means models will be trained on local machine, without using any special t ...@@ -112,20 +120,16 @@ which means models will be trained on local machine, without using any special t
.. GENERATED FROM PYTHON SOURCE LINES 70-80 .. GENERATED FROM PYTHON SOURCE LINES 78-84
Now we start to configure the experiment. Now we start to configure the experiment.
Firstly, specify the model code. Configure trial code
^^^^^^^^^^^^^^^^^^^^
In NNI evaluation of each hyperparameter set is called a *trial*. In NNI evaluation of each hyperparameter set is called a *trial*.
So the model script is called *trial code*. So the model script is called *trial code*.
If you are using Linux system without Conda, you many need to change ``python`` to ``python3``. .. GENERATED FROM PYTHON SOURCE LINES 84-86
When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
.. GENERATED FROM PYTHON SOURCE LINES 80-83
.. code-block:: default .. code-block:: default
...@@ -138,12 +142,24 @@ To run ``main.py`` from a different path, you can set trial code directory to `` ...@@ -138,12 +142,24 @@ To run ``main.py`` from a different path, you can set trial code directory to ``
.. GENERATED FROM PYTHON SOURCE LINES 87-96
When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.
(`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__
is only available in standard Python, not in Jupyter Notebook.)
.. attention::
If you are using Linux system without Conda,
you may need to change ``"python model.py"`` to ``"python3 model.py"``.
.. GENERATED FROM PYTHON SOURCE LINES 84-85 .. GENERATED FROM PYTHON SOURCE LINES 98-100
Then specify the search space we defined above: Configure search space
^^^^^^^^^^^^^^^^^^^^^^
.. GENERATED FROM PYTHON SOURCE LINES 85-87 .. GENERATED FROM PYTHON SOURCE LINES 100-102
.. code-block:: default .. code-block:: default
...@@ -156,12 +172,13 @@ Then specify the search space we defined above: ...@@ -156,12 +172,13 @@ Then specify the search space we defined above:
.. GENERATED FROM PYTHON SOURCE LINES 88-90 .. GENERATED FROM PYTHON SOURCE LINES 103-106
Choose a tuning algorithm. Configure tuning algorithm
^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we use :doc:`TPE tuner </hpo/tuners>`. Here we use :doc:`TPE tuner </hpo/tuners>`.
.. GENERATED FROM PYTHON SOURCE LINES 90-93 .. GENERATED FROM PYTHON SOURCE LINES 106-109
.. code-block:: default .. code-block:: default
...@@ -175,43 +192,47 @@ Here we use :doc:`TPE tuner </hpo/tuners>`. ...@@ -175,43 +192,47 @@ Here we use :doc:`TPE tuner </hpo/tuners>`.
.. GENERATED FROM PYTHON SOURCE LINES 94-105 .. GENERATED FROM PYTHON SOURCE LINES 110-113
Specify how many trials to run. Configure how many trials to run
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.
Please note that ``max_trial_number`` here is merely for a quick example. .. GENERATED FROM PYTHON SOURCE LINES 113-115
With default config TPE tuner requires 20 trials to warm up.
In real world max trial number is commonly set to 100+.
You can also set ``max_experiment_duration = '1h'`` to limit running time. .. code-block:: default
experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 2
And alternatively, you can skip this part and set no limit at all.
The experiment will run forever until you press Ctrl-C.
.. GENERATED FROM PYTHON SOURCE LINES 105-108
.. code-block:: default
experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 4
.. GENERATED FROM PYTHON SOURCE LINES 116-126
.. note::
``max_trial_number`` is set to 10 here for a fast example.
In real world it should be set to a larger number.
With default config TPE tuner requires 20 trials to warm up.
You may also set ``max_experiment_duration = '1h'`` to limit running time.
If neither ``max_trial_number`` nor ``max_experiment_duration`` are set,
the experiment will run forever until you press Ctrl-C.
.. GENERATED FROM PYTHON SOURCE LINES 109-114 .. GENERATED FROM PYTHON SOURCE LINES 128-133
Step 4: Run the experiment Step 4: Run the experiment
-------------------------- --------------------------
Now the experiment is ready. Choose a port and launch it. Now the experiment is ready. Choose a port and launch it. (Here we use port 8080.)
You can use the web portal to view experiment status: http://localhost:8080. You can use the web portal to view experiment status: http://localhost:8080.
.. GENERATED FROM PYTHON SOURCE LINES 114-115 .. GENERATED FROM PYTHON SOURCE LINES 133-135
.. code-block:: default .. code-block:: default
...@@ -220,27 +241,73 @@ You can use the web portal to view experiment status: http://localhost:8080. ...@@ -220,27 +241,73 @@ You can use the web portal to view experiment status: http://localhost:8080.
.. rst-class:: sphx-glr-script-out .. rst-class:: sphx-glr-script-out
Out: Out:
.. code-block:: none .. code-block:: none
[2022-03-18 12:58:19] Creating experiment, Experiment ID: 2ijcdvau [2022-03-20 21:07:36] Creating experiment, Experiment ID: p43ny6ew
[2022-03-18 12:58:19] Starting web server... [2022-03-20 21:07:36] Starting web server...
[2022-03-18 12:58:21] Setting up... [2022-03-20 21:07:37] Setting up...
[2022-03-18 12:58:21] Web UI URLs: http://127.0.0.1:8080 http://172.22.104.196:8080 [2022-03-20 21:07:37] Web portal URLs: http://127.0.0.1:8080 http://192.168.100.103:8080
[2022-03-18 13:57:44] Stopping experiment, please wait...
[2022-03-18 13:57:47] Experiment stopped
True True
.. GENERATED FROM PYTHON SOURCE LINES 136-143
After the experiment is done
----------------------------
Everything is done and it is safe to exit now. The following are optional.
If you are using standard Python instead of Jupyter Notebook,
you can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,
allowing you to view the web portal after the experiment is done.
.. GENERATED FROM PYTHON SOURCE LINES 143-147
.. code-block:: default
# input('Press enter to quit')
experiment.stop()
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[2022-03-20 21:08:57] Stopping experiment, please wait...
[2022-03-20 21:09:00] Experiment stopped
.. GENERATED FROM PYTHON SOURCE LINES 148-158
:meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,
so it can be omitted in your code.
After the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.
.. tip::
This example uses :doc:`Python API </reference/experiment>` to create experiment.
You can also create and manage experiments with :doc:`command line tool </reference/nnictl>`.
.. rst-class:: sphx-glr-timing .. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 59 minutes 28.055 seconds) **Total running time of the script:** ( 1 minutes 24.393 seconds)
.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py: .. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py:
......
docs/source/tutorials/hpo_quickstart_pytorch/model.ipynb
View file @ fbee0df1
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"\n# Port PyTorch Quickstart to NNI\nThis is a modified version of `PyTorch quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.\n\nFor now, we recommend to run this script directly to verify the environment.\n\nThere are only 2 key differences from the original version:\n\n1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.\n2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set.\n\n" "\n# Port PyTorch Quickstart to NNI\nThis is a modified version of `PyTorch quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.\n\nIt is recommended to run this script directly first to verify the environment.\n\nThere are 2 key differences from the original version:\n\n1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.\n2. In `Train model and report accuracy`_ part, it reports accuracy metrics to NNI.\n\n"
] ]
}, },
{ {
...@@ -33,7 +33,7 @@ ...@@ -33,7 +33,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Hyperparameters to be tuned\n\n" "## Hyperparameters to be tuned\nThese are the hyperparameters that will be tuned.\n\n"
] ]
}, },
{ {
...@@ -51,7 +51,7 @@ ...@@ -51,7 +51,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Get optimized hyperparameters\nIf run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n" "## Get optimized hyperparameters\nIf run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n"
] ]
}, },
{ {
...@@ -105,7 +105,7 @@ ...@@ -105,7 +105,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Define train() and test()\n\n" "## Define train and test\n\n"
] ]
}, },
{ {
...@@ -123,7 +123,7 @@ ...@@ -123,7 +123,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Train the model and report accuracy\nReport accuracy to NNI so the tuning algorithm can predict best hyperparameters.\n\n" "## Train model and report accuracy\nReport accuracy metrics to NNI so the tuning algorithm can suggest better hyperparameters.\n\n"
] ]
}, },
{ {
...@@ -154,7 +154,7 @@ ...@@ -154,7 +154,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.10.2" "version": "3.10.3"
} }
}, },
"nbformat": 4, "nbformat": 4,
......
docs/source/tutorials/hpo_quickstart_pytorch/model.py
View file @ fbee0df1
...@@ -5,14 +5,14 @@ This is a modified version of `PyTorch quickstart`_. ...@@ -5,14 +5,14 @@ This is a modified version of `PyTorch quickstart`_.
It can be run directly and will have the exact same result as original version. It can be run directly and will have the exact same result as original version.
Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later. Furthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.
For now, we recommend to run this script directly to verify the environment. It is recommended to run this script directly first to verify the environment.
There are only 2 key differences from the original version: There are 2 key differences from the original version:
1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters. 1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.
2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set. 2. In `Train model and report accuracy`_ part, it reports accuracy metrics to NNI.
.. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html .. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
""" """
...@@ -28,6 +28,7 @@ from torchvision.transforms import ToTensor ...@@ -28,6 +28,7 @@ from torchvision.transforms import ToTensor
# %% # %%
# Hyperparameters to be tuned # Hyperparameters to be tuned
# --------------------------- # ---------------------------
# These are the hyperparameters that will be tuned.
params = { params = {
'features': 512, 'features': 512,
'lr': 0.001, 'lr': 0.001,
...@@ -37,7 +38,7 @@ params = { ...@@ -37,7 +38,7 @@ params = {
# %% # %%
# Get optimized hyperparameters # Get optimized hyperparameters
# ----------------------------- # -----------------------------
# If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict. # If run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.
# But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm. # But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
optimized_params = nni.get_next_parameter() optimized_params = nni.get_next_parameter()
params.update(optimized_params) params.update(optimized_params)
...@@ -83,8 +84,8 @@ loss_fn = nn.CrossEntropyLoss() ...@@ -83,8 +84,8 @@ loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum']) optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])
# %% # %%
# Define train() and test() # Define train and test
# ------------------------- # ---------------------
def train(dataloader, model, loss_fn, optimizer): def train(dataloader, model, loss_fn, optimizer):
size = len(dataloader.dataset) size = len(dataloader.dataset)
model.train() model.train()
...@@ -112,9 +113,9 @@ def test(dataloader, model, loss_fn): ...@@ -112,9 +113,9 @@ def test(dataloader, model, loss_fn):
return correct return correct
# %% # %%
# Train the model and report accuracy # Train model and report accuracy
# ----------------------------------- # -------------------------------
# Report accuracy to NNI so the tuning algorithm can predict best hyperparameters. # Report accuracy metrics to NNI so the tuning algorithm can suggest better hyperparameters.
epochs = 5 epochs = 5
for t in range(epochs): for t in range(epochs):
print(f"Epoch {t+1}\n-------------------------------") print(f"Epoch {t+1}\n-------------------------------")
......
docs/source/tutorials/hpo_quickstart_pytorch/model.py.md5
View file @ fbee0df1
8250278967847abcd7b16ec4e811e825 ed8bfc27e3d555d842fc4eec2635e619
\ No newline at end of file \ No newline at end of file
docs/source/tutorials/hpo_quickstart_pytorch/model.rst
View file @ fbee0df1
...@@ -25,14 +25,14 @@ This is a modified version of `PyTorch quickstart`_. ...@@ -25,14 +25,14 @@ This is a modified version of `PyTorch quickstart`_.
It can be run directly and will have the exact same result as original version. It can be run directly and will have the exact same result as original version.
Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later. Furthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.
For now, we recommend to run this script directly to verify the environment. It is recommended to run this script directly first to verify the environment.
There are only 2 key differences from the original version: There are 2 key differences from the original version:
1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters. 1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.
2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set. 2. In `Train model and report accuracy`_ part, it reports accuracy metrics to NNI.
.. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html .. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
...@@ -54,12 +54,13 @@ There are only 2 key differences from the original version: ...@@ -54,12 +54,13 @@ There are only 2 key differences from the original version:
.. GENERATED FROM PYTHON SOURCE LINES 29-31 .. GENERATED FROM PYTHON SOURCE LINES 29-32
Hyperparameters to be tuned Hyperparameters to be tuned
--------------------------- ---------------------------
These are the hyperparameters that will be tuned.
.. GENERATED FROM PYTHON SOURCE LINES 31-37 .. GENERATED FROM PYTHON SOURCE LINES 32-38
.. code-block:: default .. code-block:: default
...@@ -76,14 +77,14 @@ Hyperparameters to be tuned ...@@ -76,14 +77,14 @@ Hyperparameters to be tuned
.. GENERATED FROM PYTHON SOURCE LINES 38-42 .. GENERATED FROM PYTHON SOURCE LINES 39-43
Get optimized hyperparameters Get optimized hyperparameters
----------------------------- -----------------------------
If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict. If run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.
But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm. But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
.. GENERATED FROM PYTHON SOURCE LINES 42-46 .. GENERATED FROM PYTHON SOURCE LINES 43-47
.. code-block:: default .. code-block:: default
...@@ -101,19 +102,17 @@ But with an NNI *experiment*, it will receive optimized hyperparameters from tun ...@@ -101,19 +102,17 @@ But with an NNI *experiment*, it will receive optimized hyperparameters from tun
.. code-block:: none .. code-block:: none
/home/lz/nnisrc/nni/runtime/platform/standalone.py:32: RuntimeWarning: Running NNI code without runtime. Check the following tutorial if you are new to NNI: https://nni.readthedocs.io/en/stable/Tutorial/QuickStart.html#id1
warnings.warn(warning_message, RuntimeWarning)
{'features': 512, 'lr': 0.001, 'momentum': 0} {'features': 512, 'lr': 0.001, 'momentum': 0}
.. GENERATED FROM PYTHON SOURCE LINES 47-49 .. GENERATED FROM PYTHON SOURCE LINES 48-50
Load dataset Load dataset
------------ ------------
.. GENERATED FROM PYTHON SOURCE LINES 49-57 .. GENERATED FROM PYTHON SOURCE LINES 50-58
.. code-block:: default .. code-block:: default
...@@ -132,12 +131,12 @@ Load dataset ...@@ -132,12 +131,12 @@ Load dataset
.. GENERATED FROM PYTHON SOURCE LINES 58-60 .. GENERATED FROM PYTHON SOURCE LINES 59-61
Build model with hyperparameters Build model with hyperparameters
-------------------------------- --------------------------------
.. GENERATED FROM PYTHON SOURCE LINES 60-85 .. GENERATED FROM PYTHON SOURCE LINES 61-86
.. code-block:: default .. code-block:: default
...@@ -181,12 +180,12 @@ Build model with hyperparameters ...@@ -181,12 +180,12 @@ Build model with hyperparameters
.. GENERATED FROM PYTHON SOURCE LINES 86-88 .. GENERATED FROM PYTHON SOURCE LINES 87-89
Define train() and test() Define train and test
------------------------- ---------------------
.. GENERATED FROM PYTHON SOURCE LINES 88-114 .. GENERATED FROM PYTHON SOURCE LINES 89-115
.. code-block:: default .. code-block:: default
...@@ -223,13 +222,13 @@ Define train() and test() ...@@ -223,13 +222,13 @@ Define train() and test()
.. GENERATED FROM PYTHON SOURCE LINES 115-118 .. GENERATED FROM PYTHON SOURCE LINES 116-119
Train the model and report accuracy Train model and report accuracy
----------------------------------- -------------------------------
Report accuracy to NNI so the tuning algorithm can predict best hyperparameters. Report accuracy metrics to NNI so the tuning algorithm can suggest better hyperparameters.
.. GENERATED FROM PYTHON SOURCE LINES 118-125 .. GENERATED FROM PYTHON SOURCE LINES 119-126
.. code-block:: default .. code-block:: default
...@@ -252,20 +251,20 @@ Report accuracy to NNI so the tuning algorithm can predict best hyperparameters. ...@@ -252,20 +251,20 @@ Report accuracy to NNI so the tuning algorithm can predict best hyperparameters.
Epoch 1 Epoch 1
------------------------------- -------------------------------
[2022-03-18 14:02:46] INFO (nni/MainThread) Intermediate result: 0.5418 (Index 0) [2022-03-21 01:09:37] INFO (nni/MainThread) Intermediate result: 0.461 (Index 0)
Epoch 2 Epoch 2
------------------------------- -------------------------------
[2022-03-18 14:02:52] INFO (nni/MainThread) Intermediate result: 0.5945 (Index 1) [2022-03-21 01:09:42] INFO (nni/MainThread) Intermediate result: 0.5529 (Index 1)
Epoch 3 Epoch 3
------------------------------- -------------------------------
[2022-03-18 14:02:58] INFO (nni/MainThread) Intermediate result: 0.6202 (Index 2) [2022-03-21 01:09:47] INFO (nni/MainThread) Intermediate result: 0.6155 (Index 2)
Epoch 4 Epoch 4
------------------------------- -------------------------------
[2022-03-18 14:03:04] INFO (nni/MainThread) Intermediate result: 0.6376 (Index 3) [2022-03-21 01:09:52] INFO (nni/MainThread) Intermediate result: 0.6345 (Index 3)
Epoch 5 Epoch 5
------------------------------- -------------------------------
[2022-03-18 14:03:09] INFO (nni/MainThread) Intermediate result: 0.652 (Index 4) [2022-03-21 01:09:56] INFO (nni/MainThread) Intermediate result: 0.6505 (Index 4)
[2022-03-18 14:03:09] INFO (nni/MainThread) Final result: 0.652 [2022-03-21 01:09:56] INFO (nni/MainThread) Final result: 0.6505
...@@ -273,7 +272,7 @@ Report accuracy to NNI so the tuning algorithm can predict best hyperparameters. ...@@ -273,7 +272,7 @@ Report accuracy to NNI so the tuning algorithm can predict best hyperparameters.
.. rst-class:: sphx-glr-timing .. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 29.026 seconds) **Total running time of the script:** ( 0 minutes 24.441 seconds)
.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_model.py: .. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_model.py:
......
docs/source/tutorials/hpo_quickstart_pytorch/sg_execution_times.rst
View file @ fbee0df1
...@@ -5,10 +5,10 @@ ...@@ -5,10 +5,10 @@
Computation times Computation times
================= =================
**00:29.026** total execution time for **tutorials_hpo_quickstart_pytorch** files: **00:24.441** total execution time for **tutorials_hpo_quickstart_pytorch** files:
+--------------------------------------------------------------------------+-----------+--------+ +--------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py` (``model.py``) | 00:29.026 | 0.0 MB | | :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py` (``model.py``) | 00:24.441 | 0.0 MB |
+--------------------------------------------------------------------------+-----------+--------+ +--------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py` (``main.py``) | 00:00.000 | 0.0 MB | | :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py` (``main.py``) | 00:00.000 | 0.0 MB |
+--------------------------------------------------------------------------+-----------+--------+ +--------------------------------------------------------------------------+-----------+--------+
docs/source/tutorials/hpo_quickstart_tensorflow/main.ipynb
View file @ fbee0df1
...@@ -15,21 +15,21 @@ ...@@ -15,21 +15,21 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"\n# NNI HPO Quickstart with TensorFlow\nThis tutorial optimizes the model in `official TensorFlow quickstart`_ with auto-tuning.\n\nThe tutorial consists of 4 steps: \n\n 1. Modify the model for auto-tuning.\n 2. Define hyperparameters' search space.\n 3. Configure the experiment.\n 4. Run the experiment.\n\n" "\n# NNI HPO Quickstart with TensorFlow\nThis tutorial optimizes the model in `official TensorFlow quickstart`_ with auto-tuning.\n\nThe tutorial consists of 4 steps: \n\n1. Modify the model for auto-tuning.\n2. Define hyperparameters' search space.\n3. Configure the experiment.\n4. Run the experiment.\n\n"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Step 1: Prepare the model\nIn first step, you need to prepare the model to be tuned.\n\nThe model should be put in a separate script.\nIt will be evaluated many times concurrently,\nand possibly will be trained on distributed platforms.\n\nIn this tutorial, the model is defined in :doc:`model.py <model>`.\n\nPlease understand the model code before continue to next step.\n\n" "## Step 1: Prepare the model\nIn first step, we need to prepare the model to be tuned.\n\nThe model should be put in a separate script.\nIt will be evaluated many times concurrently,\nand possibly will be trained on distributed platforms.\n\nIn this tutorial, the model is defined in :doc:`model.py <model>`.\n\nIn short, it is a TensorFlow model with 3 additional API calls:\n\n1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.\n2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.\n3. Use :func:`nni.report_final_result` to report final accuracy.\n\nPlease understand the model code before continue to next step.\n\n"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Step 2: Define search space\nIn model code, we have prepared 4 hyperparameters to be tuned:\n*dense_units*, *activation_type*, *dropout_rate*, and *learning_rate*.\n\nHere we need to define their *search space* so the tuning algorithm can sample them in desired range.\n\nAssuming we have following prior knowledge for these hyperparameters:\n\n 1. *dense_units* should be one of 64, 128, 256.\n 2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.\n 3. *dropout_rate* should be a float between 0.5 and 0.9.\n 4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.\n\nIn NNI, the space of *dense_units* and *activation_type* is called ``choice``;\nthe space of *dropout_rate* is called ``uniform``;\nand the space of *learning_rate* is called ``loguniform``.\nYou may have noticed, these names are derived from ``numpy.random``.\n\nFor full specification of search space, check :doc:`the reference </hpo/search_space>`.\n\nNow we can define the search space as follow:\n\n" "## Step 2: Define search space\nIn model code, we have prepared 4 hyperparameters to be tuned:\n*dense_units*, *activation_type*, *dropout_rate*, and *learning_rate*.\n\nHere we need to define their *search space* so the tuning algorithm can sample them in desired range.\n\nAssuming we have following prior knowledge for these hyperparameters:\n\n1. *dense_units* should be one of 64, 128, 256.\n2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.\n3. *dropout_rate* should be a float between 0.5 and 0.9.\n4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.\n\nIn NNI, the space of *dense_units* and *activation_type* is called ``choice``;\nthe space of *dropout_rate* is called ``uniform``;\nand the space of *learning_rate* is called ``loguniform``.\nYou may have noticed, these names are derived from ``numpy.random``.\n\nFor full specification of search space, check :doc:`the reference </hpo/search_space>`.\n\nNow we can define the search space as follow:\n\n"
] ]
}, },
{ {
...@@ -65,7 +65,7 @@ ...@@ -65,7 +65,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Now we start to configure the experiment.\n\nFirstly, specify the model code.\nIn NNI evaluation of each hyperparameter set is called a *trial*.\nSo the model script is called *trial code*.\n\nIf you are using Linux system without Conda, you many need to change ``python`` to ``python3``.\n\nWhen ``trial_code_directory`` is a relative path, it relates to current working directory.\nTo run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.\n\n" "Now we start to configure the experiment.\n\n### Configure trial code\nIn NNI evaluation of each hyperparameter set is called a *trial*.\nSo the model script is called *trial code*.\n\n"
] ]
}, },
{ {
...@@ -83,7 +83,14 @@ ...@@ -83,7 +83,14 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Then specify the search space we defined above:\n\n" "When ``trial_code_directory`` is a relative path, it relates to current working directory.\nTo run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.\n(`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__\nis only available in standard Python, not in Jupyter Notebook.)\n\n.. attention::\n\n If you are using Linux system without Conda,\n you may need to change ``\"python model.py\"`` to ``\"python3 model.py\"``.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Configure search space\n\n"
] ]
}, },
{ {
...@@ -101,7 +108,7 @@ ...@@ -101,7 +108,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Choose a tuning algorithm.\nHere we use :doc:`TPE tuner </hpo/tuners>`.\n\n" "### Configure tuning algorithm\nHere we use :doc:`TPE tuner </hpo/tuners>`.\n\n"
] ]
}, },
{ {
...@@ -119,7 +126,7 @@ ...@@ -119,7 +126,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Specify how many trials to run.\nHere we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.\n\nPlease note that ``max_trial_number`` here is merely for a quick example.\nWith default config TPE tuner requires 20 trials to warm up.\nIn real world max trial number is commonly set to 100+.\n\nYou can also set ``max_experiment_duration = '1h'`` to limit running time.\n\nAnd alternatively, you can skip this part and set no limit at all.\nThe experiment will run forever until you press Ctrl-C.\n\n" "### Configure how many trials to run\nHere we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.\n\n"
] ]
}, },
{ {
...@@ -130,14 +137,21 @@ ...@@ -130,14 +137,21 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"experiment.config.max_trial_number = 10\nexperiment.config.trial_concurrency = 4" "experiment.config.max_trial_number = 10\nexperiment.config.trial_concurrency = 2"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Step 4: Run the experiment\nNow the experiment is ready. Choose a port and launch it.\n\nYou can use the web portal to view experiment status: http://localhost:8080.\n\n" "<div class=\"alert alert-info\"><h4>Note</h4><p>``max_trial_number`` is set to 10 here for a fast example.\n In real world it should be set to a larger number.\n With default config TPE tuner requires 20 trials to warm up.</p></div>\n\nYou may also set ``max_experiment_duration = '1h'`` to limit running time.\n\nIf neither ``max_trial_number`` nor ``max_experiment_duration`` are set,\nthe experiment will run forever until you press Ctrl-C.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Run the experiment\nNow the experiment is ready. Choose a port and launch it. (Here we use port 8080.)\n\nYou can use the web portal to view experiment status: http://localhost:8080.\n\n"
] ]
}, },
{ {
...@@ -150,6 +164,31 @@ ...@@ -150,6 +164,31 @@
"source": [ "source": [
"experiment.run(8080)" "experiment.run(8080)"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## After the experiment is done\nEverything is done and it is safe to exit now. The following are optional.\n\nIf you are using standard Python instead of Jupyter Notebook,\nyou can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,\nallowing you to view the web portal after the experiment is done.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# input('Press enter to quit')\nexperiment.stop()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,\nso it can be omitted in your code.\n\nAfter the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.\n\n.. tip::\n\n This example uses :doc:`Python API </reference/experiment>` to create experiment.\n\n You can also create and manage experiments with :doc:`command line tool </reference/nnictl>`.\n\n"
]
} }
], ],
"metadata": { "metadata": {
...@@ -168,7 +207,7 @@ ...@@ -168,7 +207,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.10.2" "version": "3.10.3"
} }
}, },
"nbformat": 4, "nbformat": 4,
......
docs/source/tutorials/hpo_quickstart_tensorflow/main.py
View file @ fbee0df1
...@@ -5,10 +5,10 @@ This tutorial optimizes the model in `official TensorFlow quickstart`_ with auto ...@@ -5,10 +5,10 @@ This tutorial optimizes the model in `official TensorFlow quickstart`_ with auto
The tutorial consists of 4 steps: The tutorial consists of 4 steps:
1. Modify the model for auto-tuning. 1. Modify the model for auto-tuning.
2. Define hyperparameters' search space. 2. Define hyperparameters' search space.
3. Configure the experiment. 3. Configure the experiment.
4. Run the experiment. 4. Run the experiment.
.. _official TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner .. _official TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner
""" """
...@@ -16,7 +16,7 @@ The tutorial consists of 4 steps: ...@@ -16,7 +16,7 @@ The tutorial consists of 4 steps:
# %% # %%
# Step 1: Prepare the model # Step 1: Prepare the model
# ------------------------- # -------------------------
# In first step, you need to prepare the model to be tuned. # In first step, we need to prepare the model to be tuned.
# #
# The model should be put in a separate script. # The model should be put in a separate script.
# It will be evaluated many times concurrently, # It will be evaluated many times concurrently,
...@@ -24,6 +24,12 @@ The tutorial consists of 4 steps: ...@@ -24,6 +24,12 @@ The tutorial consists of 4 steps:
# #
# In this tutorial, the model is defined in :doc:`model.py <model>`. # In this tutorial, the model is defined in :doc:`model.py <model>`.
# #
# In short, it is a TensorFlow model with 3 additional API calls:
#
# 1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.
# 2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.
# 3. Use :func:`nni.report_final_result` to report final accuracy.
#
# Please understand the model code before continue to next step. # Please understand the model code before continue to next step.
# %% # %%
...@@ -36,10 +42,10 @@ The tutorial consists of 4 steps: ...@@ -36,10 +42,10 @@ The tutorial consists of 4 steps:
# #
# Assuming we have following prior knowledge for these hyperparameters: # Assuming we have following prior knowledge for these hyperparameters:
# #
# 1. *dense_units* should be one of 64, 128, 256. # 1. *dense_units* should be one of 64, 128, 256.
# 2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None. # 2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.
# 3. *dropout_rate* should be a float between 0.5 and 0.9. # 3. *dropout_rate* should be a float between 0.5 and 0.9.
# 4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution. # 4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
# #
# In NNI, the space of *dense_units* and *activation_type* is called ``choice``; # In NNI, the space of *dense_units* and *activation_type* is called ``choice``;
# the space of *dropout_rate* is called ``uniform``; # the space of *dropout_rate* is called ``uniform``;
...@@ -71,46 +77,81 @@ experiment = Experiment('local') ...@@ -71,46 +77,81 @@ experiment = Experiment('local')
# %% # %%
# Now we start to configure the experiment. # Now we start to configure the experiment.
# #
# Firstly, specify the model code. # Configure trial code
# ^^^^^^^^^^^^^^^^^^^^
# In NNI evaluation of each hyperparameter set is called a *trial*. # In NNI evaluation of each hyperparameter set is called a *trial*.
# So the model script is called *trial code*. # So the model script is called *trial code*.
#
# If you are using Linux system without Conda, you many need to change ``python`` to ``python3``.
#
# When ``trial_code_directory`` is a relative path, it relates to current working directory.
# To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
experiment.config.trial_command = 'python model.py' experiment.config.trial_command = 'python model.py'
experiment.config.trial_code_directory = '.' experiment.config.trial_code_directory = '.'
# %%
# When ``trial_code_directory`` is a relative path, it relates to current working directory.
# To run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.
# (`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__
# is only available in standard Python, not in Jupyter Notebook.)
#
# .. attention::
#
# If you are using Linux system without Conda,
# you may need to change ``"python model.py"`` to ``"python3 model.py"``.
# %% # %%
# Then specify the search space we defined above: # Configure search space
# ^^^^^^^^^^^^^^^^^^^^^^
experiment.config.search_space = search_space experiment.config.search_space = search_space
# %% # %%
# Choose a tuning algorithm. # Configure tuning algorithm
# ^^^^^^^^^^^^^^^^^^^^^^^^^^
# Here we use :doc:`TPE tuner </hpo/tuners>`. # Here we use :doc:`TPE tuner </hpo/tuners>`.
experiment.config.tuner.name = 'TPE' experiment.config.tuner.name = 'TPE'
experiment.config.tuner.class_args['optimize_mode'] = 'maximize' experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
# %% # %%
# Specify how many trials to run. # Configure how many trials to run
# Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time. # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.
experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 2
# %%
# .. note::
# #
# Please note that ``max_trial_number`` here is merely for a quick example. # ``max_trial_number`` is set to 10 here for a fast example.
# With default config TPE tuner requires 20 trials to warm up. # In real world it should be set to a larger number.
# In real world max trial number is commonly set to 100+. # With default config TPE tuner requires 20 trials to warm up.
# #
# You can also set ``max_experiment_duration = '1h'`` to limit running time. # You may also set ``max_experiment_duration = '1h'`` to limit running time.
# #
# And alternatively, you can skip this part and set no limit at all. # If neither ``max_trial_number`` nor ``max_experiment_duration`` are set,
# The experiment will run forever until you press Ctrl-C. # the experiment will run forever until you press Ctrl-C.
experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 4
# %% # %%
# Step 4: Run the experiment # Step 4: Run the experiment
# -------------------------- # --------------------------
# Now the experiment is ready. Choose a port and launch it. # Now the experiment is ready. Choose a port and launch it. (Here we use port 8080.)
# #
# You can use the web portal to view experiment status: http://localhost:8080. # You can use the web portal to view experiment status: http://localhost:8080.
experiment.run(8080) experiment.run(8080)
# %%
# After the experiment is done
# ----------------------------
# Everything is done and it is safe to exit now. The following are optional.
#
# If you are using standard Python instead of Jupyter Notebook,
# you can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,
# allowing you to view the web portal after the experiment is done.
# input('Press enter to quit')
experiment.stop()
# %%
# :meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,
# so it can be omitted in your code.
#
# After the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.
#
# .. tip::
#
# This example uses :doc:`Python API </reference/experiment>` to create experiment.
#
# You can also create and manage experiments with :doc:`command line tool </reference/nnictl>`.
docs/source/tutorials/hpo_quickstart_tensorflow/main.py.md5
View file @ fbee0df1
911c32a84d08c02c02821ba2badc056c fe5546e4ae3f3dbf5e852af322dae15f
\ No newline at end of file \ No newline at end of file
docs/source/tutorials/hpo_quickstart_tensorflow/main.rst
View file @ fbee0df1
...@@ -25,18 +25,18 @@ This tutorial optimizes the model in `official TensorFlow quickstart`_ with auto ...@@ -25,18 +25,18 @@ This tutorial optimizes the model in `official TensorFlow quickstart`_ with auto
The tutorial consists of 4 steps: The tutorial consists of 4 steps:
1. Modify the model for auto-tuning. 1. Modify the model for auto-tuning.
2. Define hyperparameters' search space. 2. Define hyperparameters' search space.
3. Configure the experiment. 3. Configure the experiment.
4. Run the experiment. 4. Run the experiment.
.. _official TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner .. _official TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner
.. GENERATED FROM PYTHON SOURCE LINES 17-28 .. GENERATED FROM PYTHON SOURCE LINES 17-34
Step 1: Prepare the model Step 1: Prepare the model
------------------------- -------------------------
In first step, you need to prepare the model to be tuned. In first step, we need to prepare the model to be tuned.
The model should be put in a separate script. The model should be put in a separate script.
It will be evaluated many times concurrently, It will be evaluated many times concurrently,
...@@ -44,9 +44,15 @@ and possibly will be trained on distributed platforms. ...@@ -44,9 +44,15 @@ and possibly will be trained on distributed platforms.
In this tutorial, the model is defined in :doc:`model.py <model>`. In this tutorial, the model is defined in :doc:`model.py <model>`.
In short, it is a TensorFlow model with 3 additional API calls:
1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.
2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.
3. Use :func:`nni.report_final_result` to report final accuracy.
Please understand the model code before continue to next step. Please understand the model code before continue to next step.
.. GENERATED FROM PYTHON SOURCE LINES 30-52 .. GENERATED FROM PYTHON SOURCE LINES 36-58
Step 2: Define search space Step 2: Define search space
--------------------------- ---------------------------
...@@ -57,10 +63,10 @@ Here we need to define their *search space* so the tuning algorithm can sample t ...@@ -57,10 +63,10 @@ Here we need to define their *search space* so the tuning algorithm can sample t
Assuming we have following prior knowledge for these hyperparameters: Assuming we have following prior knowledge for these hyperparameters:
1. *dense_units* should be one of 64, 128, 256. 1. *dense_units* should be one of 64, 128, 256.
2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None. 2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.
3. *dropout_rate* should be a float between 0.5 and 0.9. 3. *dropout_rate* should be a float between 0.5 and 0.9.
4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution. 4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
In NNI, the space of *dense_units* and *activation_type* is called ``choice``; In NNI, the space of *dense_units* and *activation_type* is called ``choice``;
the space of *dropout_rate* is called ``uniform``; the space of *dropout_rate* is called ``uniform``;
...@@ -71,7 +77,7 @@ For full specification of search space, check :doc:`the reference </hpo/search_s ...@@ -71,7 +77,7 @@ For full specification of search space, check :doc:`the reference </hpo/search_s
Now we can define the search space as follow: Now we can define the search space as follow:
.. GENERATED FROM PYTHON SOURCE LINES 52-60 .. GENERATED FROM PYTHON SOURCE LINES 58-66
.. code-block:: default .. code-block:: default
...@@ -90,7 +96,7 @@ Now we can define the search space as follow: ...@@ -90,7 +96,7 @@ Now we can define the search space as follow:
.. GENERATED FROM PYTHON SOURCE LINES 61-68 .. GENERATED FROM PYTHON SOURCE LINES 67-74
Step 3: Configure the experiment Step 3: Configure the experiment
-------------------------------- --------------------------------
...@@ -100,7 +106,7 @@ The *experiment config* defines how to train the models and how to explore the s ...@@ -100,7 +106,7 @@ The *experiment config* defines how to train the models and how to explore the s
In this tutorial we use a *local* mode experiment, In this tutorial we use a *local* mode experiment,
which means models will be trained on local machine, without using any special training platform. which means models will be trained on local machine, without using any special training platform.
.. GENERATED FROM PYTHON SOURCE LINES 68-71 .. GENERATED FROM PYTHON SOURCE LINES 74-77
.. code-block:: default .. code-block:: default
...@@ -114,20 +120,16 @@ which means models will be trained on local machine, without using any special t ...@@ -114,20 +120,16 @@ which means models will be trained on local machine, without using any special t
.. GENERATED FROM PYTHON SOURCE LINES 72-82 .. GENERATED FROM PYTHON SOURCE LINES 78-84
Now we start to configure the experiment. Now we start to configure the experiment.
Firstly, specify the model code. Configure trial code
^^^^^^^^^^^^^^^^^^^^
In NNI evaluation of each hyperparameter set is called a *trial*. In NNI evaluation of each hyperparameter set is called a *trial*.
So the model script is called *trial code*. So the model script is called *trial code*.
If you are using Linux system without Conda, you many need to change ``python`` to ``python3``. .. GENERATED FROM PYTHON SOURCE LINES 84-86
When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
.. GENERATED FROM PYTHON SOURCE LINES 82-85
.. code-block:: default .. code-block:: default
...@@ -140,12 +142,24 @@ To run ``main.py`` from a different path, you can set trial code directory to `` ...@@ -140,12 +142,24 @@ To run ``main.py`` from a different path, you can set trial code directory to ``
.. GENERATED FROM PYTHON SOURCE LINES 87-96
.. GENERATED FROM PYTHON SOURCE LINES 86-87 When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.
(`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__
is only available in standard Python, not in Jupyter Notebook.)
Then specify the search space we defined above: .. attention::
.. GENERATED FROM PYTHON SOURCE LINES 87-89 If you are using Linux system without Conda,
you may need to change ``"python model.py"`` to ``"python3 model.py"``.
.. GENERATED FROM PYTHON SOURCE LINES 98-100
Configure search space
^^^^^^^^^^^^^^^^^^^^^^
.. GENERATED FROM PYTHON SOURCE LINES 100-102
.. code-block:: default .. code-block:: default
...@@ -158,12 +172,13 @@ Then specify the search space we defined above: ...@@ -158,12 +172,13 @@ Then specify the search space we defined above:
.. GENERATED FROM PYTHON SOURCE LINES 90-92 .. GENERATED FROM PYTHON SOURCE LINES 103-106
Choose a tuning algorithm. Configure tuning algorithm
^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we use :doc:`TPE tuner </hpo/tuners>`. Here we use :doc:`TPE tuner </hpo/tuners>`.
.. GENERATED FROM PYTHON SOURCE LINES 92-95 .. GENERATED FROM PYTHON SOURCE LINES 106-109
.. code-block:: default .. code-block:: default
...@@ -177,43 +192,47 @@ Here we use :doc:`TPE tuner </hpo/tuners>`. ...@@ -177,43 +192,47 @@ Here we use :doc:`TPE tuner </hpo/tuners>`.
.. GENERATED FROM PYTHON SOURCE LINES 96-107 .. GENERATED FROM PYTHON SOURCE LINES 110-113
Specify how many trials to run. Configure how many trials to run
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.
Please note that ``max_trial_number`` here is merely for a quick example. .. GENERATED FROM PYTHON SOURCE LINES 113-115
With default config TPE tuner requires 20 trials to warm up.
In real world max trial number is commonly set to 100+. .. code-block:: default
You can also set ``max_experiment_duration = '1h'`` to limit running time. experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 2
And alternatively, you can skip this part and set no limit at all.
The experiment will run forever until you press Ctrl-C.
.. GENERATED FROM PYTHON SOURCE LINES 107-110
.. code-block:: default
experiment.config.max_trial_number = 10
experiment.config.trial_concurrency = 4
.. GENERATED FROM PYTHON SOURCE LINES 116-126
.. note::
``max_trial_number`` is set to 10 here for a fast example.
In real world it should be set to a larger number.
With default config TPE tuner requires 20 trials to warm up.
You may also set ``max_experiment_duration = '1h'`` to limit running time.
If neither ``max_trial_number`` nor ``max_experiment_duration`` are set,
the experiment will run forever until you press Ctrl-C.
.. GENERATED FROM PYTHON SOURCE LINES 111-116 .. GENERATED FROM PYTHON SOURCE LINES 128-133
Step 4: Run the experiment Step 4: Run the experiment
-------------------------- --------------------------
Now the experiment is ready. Choose a port and launch it. Now the experiment is ready. Choose a port and launch it. (Here we use port 8080.)
You can use the web portal to view experiment status: http://localhost:8080. You can use the web portal to view experiment status: http://localhost:8080.
.. GENERATED FROM PYTHON SOURCE LINES 116-117 .. GENERATED FROM PYTHON SOURCE LINES 133-135
.. code-block:: default .. code-block:: default
...@@ -222,27 +241,73 @@ You can use the web portal to view experiment status: http://localhost:8080. ...@@ -222,27 +241,73 @@ You can use the web portal to view experiment status: http://localhost:8080.
.. rst-class:: sphx-glr-script-out .. rst-class:: sphx-glr-script-out
Out: Out:
.. code-block:: none .. code-block:: none
[2022-03-07 03:24:07] Creating experiment, Experiment ID: f4q1xjki [2022-03-20 21:12:19] Creating experiment, Experiment ID: 8raiuoyb
[2022-03-07 03:24:07] Starting web server... [2022-03-20 21:12:19] Starting web server...
[2022-03-07 03:24:08] Setting up... [2022-03-20 21:12:20] Setting up...
[2022-03-07 03:24:08] Web UI URLs: http://127.0.0.1:8080 http://192.168.100.103:8080 [2022-03-20 21:12:20] Web portal URLs: http://127.0.0.1:8080 http://192.168.100.103:8080
[2022-03-07 03:36:50] Stopping experiment, please wait...
[2022-03-07 03:36:53] Experiment stopped
True True
.. GENERATED FROM PYTHON SOURCE LINES 136-143
After the experiment is done
----------------------------
Everything is done and it is safe to exit now. The following are optional.
If you are using standard Python instead of Jupyter Notebook,
you can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,
allowing you to view the web portal after the experiment is done.
.. GENERATED FROM PYTHON SOURCE LINES 143-147
.. code-block:: default
# input('Press enter to quit')
experiment.stop()
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[2022-03-20 21:13:41] Stopping experiment, please wait...
[2022-03-20 21:13:44] Experiment stopped
.. GENERATED FROM PYTHON SOURCE LINES 148-158
:meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,
so it can be omitted in your code.
After the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.
.. tip::
This example uses :doc:`Python API </reference/experiment>` to create experiment.
You can also create and manage experiments with :doc:`command line tool </reference/nnictl>`.
.. rst-class:: sphx-glr-timing .. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 12 minutes 45.612 seconds) **Total running time of the script:** ( 1 minutes 24.257 seconds)
.. _sphx_glr_download_tutorials_hpo_quickstart_tensorflow_main.py: .. _sphx_glr_download_tutorials_hpo_quickstart_tensorflow_main.py:
......
docs/source/tutorials/hpo_quickstart_tensorflow/model.ipynb
View file @ fbee0df1
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"\n# Port TensorFlow Quickstart to NNI\nThis is a modified version of `TensorFlow quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.\n\nFor now, we recommend to run this script directly to verify the environment.\n\nThere are only 3 key differences from the original version:\n\n 1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.\n 2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy for visualization.\n 3. In `Report final result`_ part, it reports final accuracy for tuner to generate next hyperparameter set.\n\n" "\n# Port TensorFlow Quickstart to NNI\nThis is a modified version of `TensorFlow quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.\n\nIt is recommended to run this script directly first to verify the environment.\n\nThere are 3 key differences from the original version:\n\n1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.\n2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy metrics.\n3. In `Report final result`_ part, it reports final accuracy.\n\n"
] ]
}, },
{ {
...@@ -33,7 +33,7 @@ ...@@ -33,7 +33,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Hyperparameters to be tuned\n\n" "## Hyperparameters to be tuned\nThese are the hyperparameters that will be tuned later.\n\n"
] ]
}, },
{ {
...@@ -51,7 +51,7 @@ ...@@ -51,7 +51,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Get optimized hyperparameters\nIf run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n" "## Get optimized hyperparameters\nIf run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n"
] ]
}, },
{ {
...@@ -62,7 +62,7 @@ ...@@ -62,7 +62,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"optimized_params = nni.get_next_parameter()\nparams.update(optimized_params)" "optimized_params = nni.get_next_parameter()\nparams.update(optimized_params)\nprint(params)"
] ]
}, },
{ {
...@@ -98,14 +98,14 @@ ...@@ -98,14 +98,14 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"model = tf.keras.models.Sequential([\n tf.keras.layers.Flatten(input_shape=(28, 28)),\n tf.keras.layers.Dense(params['dense_units'], activation=params['activation_type']),\n tf.keras.layers.Dropout(params['dropout_rate']),\n tf.keras.layers.Dense(10)\n])\n\nadam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate'])\n\nloss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n\nmodel.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy'])" "model = tf.keras.models.Sequential([\n tf.keras.layers.Flatten(input_shape=(28, 28)),\n tf.keras.layers.Dense(params['dense_units'], activation=params['activation_type']),\n tf.keras.layers.Dropout(params['dropout_rate']),\n tf.keras.layers.Dense(10)\n])\n\nadam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate'])\nloss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\nmodel.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy'])"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## (Optional) Report intermediate results\nThe callback reports per-epoch accuracy to show learning curve in NNI web portal.\nAnd in :doc:`/hpo/assessors`, you will see how to leverage the metrics for early stopping.\n\nYou can safely skip this and the experiment will work fine.\n\n" "## (Optional) Report intermediate results\nThe callback reports per-epoch accuracy to show learning curve in the web portal.\nYou can also leverage the metrics for early stopping with :doc:`NNI assessors </hpo/assessors>`.\n\nThis part can be safely skipped and the experiment will work fine.\n\n"
] ]
}, },
{ {
...@@ -141,7 +141,7 @@ ...@@ -141,7 +141,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Report final result\nReport final accuracy to NNI so the tuning algorithm can predict best hyperparameters.\n\n" "## Report final result\nReport final accuracy to NNI so the tuning algorithm can suggest better hyperparameters.\n\n"
] ]
}, },
{ {
...@@ -172,7 +172,7 @@ ...@@ -172,7 +172,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.10.2" "version": "3.10.3"
} }
}, },
"nbformat": 4, "nbformat": 4,
......
docs/source/tutorials/hpo_quickstart_tensorflow/model.py
View file @ fbee0df1
...@@ -5,15 +5,15 @@ This is a modified version of `TensorFlow quickstart`_. ...@@ -5,15 +5,15 @@ This is a modified version of `TensorFlow quickstart`_.
It can be run directly and will have the exact same result as original version. It can be run directly and will have the exact same result as original version.
Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later. Furthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.
For now, we recommend to run this script directly to verify the environment. It is recommended to run this script directly first to verify the environment.
There are only 3 key differences from the original version: There are 3 key differences from the original version:
1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters. 1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.
2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy for visualization. 2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy metrics.
3. In `Report final result`_ part, it reports final accuracy for tuner to generate next hyperparameter set. 3. In `Report final result`_ part, it reports final accuracy.
.. _TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner .. _TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner
""" """
...@@ -25,6 +25,7 @@ import tensorflow as tf ...@@ -25,6 +25,7 @@ import tensorflow as tf
# %% # %%
# Hyperparameters to be tuned # Hyperparameters to be tuned
# --------------------------- # ---------------------------
# These are the hyperparameters that will be tuned later.
params = { params = {
'dense_units': 128, 'dense_units': 128,
'activation_type': 'relu', 'activation_type': 'relu',
...@@ -35,10 +36,11 @@ params = { ...@@ -35,10 +36,11 @@ params = {
# %% # %%
# Get optimized hyperparameters # Get optimized hyperparameters
# ----------------------------- # -----------------------------
# If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict. # If run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.
# But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm. # But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
optimized_params = nni.get_next_parameter() optimized_params = nni.get_next_parameter()
params.update(optimized_params) params.update(optimized_params)
print(params)
# %% # %%
# Load dataset # Load dataset
...@@ -59,18 +61,16 @@ model = tf.keras.models.Sequential([ ...@@ -59,18 +61,16 @@ model = tf.keras.models.Sequential([
]) ])
adam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate']) adam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate'])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy']) model.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy'])
# %% # %%
# (Optional) Report intermediate results # (Optional) Report intermediate results
# -------------------------------------- # --------------------------------------
# The callback reports per-epoch accuracy to show learning curve in NNI web portal. # The callback reports per-epoch accuracy to show learning curve in the web portal.
# And in :doc:`/hpo/assessors`, you will see how to leverage the metrics for early stopping. # You can also leverage the metrics for early stopping with :doc:`NNI assessors </hpo/assessors>`.
# #
# You can safely skip this and the experiment will work fine. # This part can be safely skipped and the experiment will work fine.
callback = tf.keras.callbacks.LambdaCallback( callback = tf.keras.callbacks.LambdaCallback(
on_epoch_end = lambda epoch, logs: nni.report_intermediate_result(logs['accuracy']) on_epoch_end = lambda epoch, logs: nni.report_intermediate_result(logs['accuracy'])
) )
...@@ -84,5 +84,5 @@ loss, accuracy = model.evaluate(x_test, y_test, verbose=2) ...@@ -84,5 +84,5 @@ loss, accuracy = model.evaluate(x_test, y_test, verbose=2)
# %% # %%
# Report final result # Report final result
# ------------------- # -------------------
# Report final accuracy to NNI so the tuning algorithm can predict best hyperparameters. # Report final accuracy to NNI so the tuning algorithm can suggest better hyperparameters.
nni.report_final_result(accuracy) nni.report_final_result(accuracy)
docs/source/tutorials/hpo_quickstart_tensorflow/model.py.md5
View file @ fbee0df1
1d29b3ef885b5725c4a3a2c8121ee8df d0c869d9d7c7f9208abc10357de52056
\ No newline at end of file \ No newline at end of file
docs/source/tutorials/hpo_quickstart_tensorflow/model.rst
View file @ fbee0df1
...@@ -25,15 +25,15 @@ This is a modified version of `TensorFlow quickstart`_. ...@@ -25,15 +25,15 @@ This is a modified version of `TensorFlow quickstart`_.
It can be run directly and will have the exact same result as original version. It can be run directly and will have the exact same result as original version.
Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later. Furthermore, it enables the ability of auto tuning with an NNI *experiment*, which will be detailed later.
For now, we recommend to run this script directly to verify the environment. It is recommended to run this script directly first to verify the environment.
There are only 3 key differences from the original version: There are 3 key differences from the original version:
1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters. 1. In `Get optimized hyperparameters`_ part, it receives generated hyperparameters.
2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy for visualization. 2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy metrics.
3. In `Report final result`_ part, it reports final accuracy for tuner to generate next hyperparameter set. 3. In `Report final result`_ part, it reports final accuracy.
.. _TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner .. _TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner
...@@ -51,12 +51,13 @@ There are only 3 key differences from the original version: ...@@ -51,12 +51,13 @@ There are only 3 key differences from the original version:
.. GENERATED FROM PYTHON SOURCE LINES 26-28 .. GENERATED FROM PYTHON SOURCE LINES 26-29
Hyperparameters to be tuned Hyperparameters to be tuned
--------------------------- ---------------------------
These are the hyperparameters that will be tuned later.
.. GENERATED FROM PYTHON SOURCE LINES 28-35 .. GENERATED FROM PYTHON SOURCE LINES 29-36
.. code-block:: default .. code-block:: default
...@@ -74,19 +75,20 @@ Hyperparameters to be tuned ...@@ -74,19 +75,20 @@ Hyperparameters to be tuned
.. GENERATED FROM PYTHON SOURCE LINES 36-40 .. GENERATED FROM PYTHON SOURCE LINES 37-41
Get optimized hyperparameters Get optimized hyperparameters
----------------------------- -----------------------------
If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict. If run directly, :func:`nni.get_next_parameter` is a no-op and returns an empty dict.
But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm. But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
.. GENERATED FROM PYTHON SOURCE LINES 40-43 .. GENERATED FROM PYTHON SOURCE LINES 41-45
.. code-block:: default .. code-block:: default
optimized_params = nni.get_next_parameter() optimized_params = nni.get_next_parameter()
params.update(optimized_params) params.update(optimized_params)
print(params)
...@@ -98,18 +100,17 @@ But with an NNI *experiment*, it will receive optimized hyperparameters from tun ...@@ -98,18 +100,17 @@ But with an NNI *experiment*, it will receive optimized hyperparameters from tun
.. code-block:: none .. code-block:: none
/home/lz/code/nnisrc/nni/runtime/platform/standalone.py:32: RuntimeWarning: Running NNI code without runtime. Check the following tutorial if you are new to NNI: https://nni.readthedocs.io/en/stable/Tutorial/QuickStart.html#id1 {'dense_units': 128, 'activation_type': 'relu', 'dropout_rate': 0.2, 'learning_rate': 0.001}
warnings.warn(warning_message, RuntimeWarning)
.. GENERATED FROM PYTHON SOURCE LINES 44-46 .. GENERATED FROM PYTHON SOURCE LINES 46-48
Load dataset Load dataset
------------ ------------
.. GENERATED FROM PYTHON SOURCE LINES 46-51 .. GENERATED FROM PYTHON SOURCE LINES 48-53
.. code-block:: default .. code-block:: default
...@@ -125,12 +126,12 @@ Load dataset ...@@ -125,12 +126,12 @@ Load dataset
.. GENERATED FROM PYTHON SOURCE LINES 52-54 .. GENERATED FROM PYTHON SOURCE LINES 54-56
Build model with hyperparameters Build model with hyperparameters
-------------------------------- --------------------------------
.. GENERATED FROM PYTHON SOURCE LINES 54-67 .. GENERATED FROM PYTHON SOURCE LINES 56-67
.. code-block:: default .. code-block:: default
...@@ -142,9 +143,7 @@ Build model with hyperparameters ...@@ -142,9 +143,7 @@ Build model with hyperparameters
]) ])
adam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate']) adam = tf.keras.optimizers.Adam(learning_rate=params['learning_rate'])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy']) model.compile(optimizer=adam, loss=loss_fn, metrics=['accuracy'])
...@@ -158,10 +157,10 @@ Build model with hyperparameters ...@@ -158,10 +157,10 @@ Build model with hyperparameters
(Optional) Report intermediate results (Optional) Report intermediate results
-------------------------------------- --------------------------------------
The callback reports per-epoch accuracy to show learning curve in NNI web portal. The callback reports per-epoch accuracy to show learning curve in the web portal.
And in :doc:`/hpo/assessors`, you will see how to leverage the metrics for early stopping. You can also leverage the metrics for early stopping with :doc:`NNI assessors </hpo/assessors>`.
You can safely skip this and the experiment will work fine. This part can be safely skipped and the experiment will work fine.
.. GENERATED FROM PYTHON SOURCE LINES 74-78 .. GENERATED FROM PYTHON SOURCE LINES 74-78
...@@ -201,21 +200,21 @@ Train and evluate the model ...@@ -201,21 +200,21 @@ Train and evluate the model
.. code-block:: none .. code-block:: none
Epoch 1/5 Epoch 1/5
[2022-03-07 02:37:35] INFO (nni/MainThread) Intermediate result: 0.9145833253860474 (Index 0) [2022-03-21 01:25:00] INFO (nni/MainThread) Intermediate result: 0.9153500199317932 (Index 0)
1875/1875 - 12s - loss: 0.2940 - accuracy: 0.9146 - 12s/epoch - 6ms/step 1875/1875 - 17s - loss: 0.2914 - accuracy: 0.9154 - 17s/epoch - 9ms/step
Epoch 2/5 Epoch 2/5
[2022-03-07 02:37:41] INFO (nni/MainThread) Intermediate result: 0.9573833346366882 (Index 1) [2022-03-21 01:25:18] INFO (nni/MainThread) Intermediate result: 0.9588666558265686 (Index 1)
1875/1875 - 5s - loss: 0.1422 - accuracy: 0.9574 - 5s/epoch - 3ms/step 1875/1875 - 18s - loss: 0.1387 - accuracy: 0.9589 - 18s/epoch - 10ms/step
Epoch 3/5 Epoch 3/5
[2022-03-07 02:37:49] INFO (nni/MainThread) Intermediate result: 0.967283308506012 (Index 2) [2022-03-21 01:25:38] INFO (nni/MainThread) Intermediate result: 0.9677000045776367 (Index 2)
1875/1875 - 8s - loss: 0.1075 - accuracy: 0.9673 - 8s/epoch - 4ms/step 1875/1875 - 20s - loss: 0.1073 - accuracy: 0.9677 - 20s/epoch - 11ms/step
Epoch 4/5 Epoch 4/5
[2022-03-07 02:37:57] INFO (nni/MainThread) Intermediate result: 0.9723333120346069 (Index 3) [2022-03-21 01:25:56] INFO (nni/MainThread) Intermediate result: 0.9738666415214539 (Index 3)
1875/1875 - 8s - loss: 0.0885 - accuracy: 0.9723 - 8s/epoch - 4ms/step 1875/1875 - 18s - loss: 0.0866 - accuracy: 0.9739 - 18s/epoch - 10ms/step
Epoch 5/5 Epoch 5/5
[2022-03-07 02:38:06] INFO (nni/MainThread) Intermediate result: 0.9762333035469055 (Index 4) [2022-03-21 01:26:16] INFO (nni/MainThread) Intermediate result: 0.977483332157135 (Index 4)
1875/1875 - 9s - loss: 0.0747 - accuracy: 0.9762 - 9s/epoch - 5ms/step 1875/1875 - 21s - loss: 0.0728 - accuracy: 0.9775 - 21s/epoch - 11ms/step
313/313 - 1s - loss: 0.0766 - accuracy: 0.9772 - 647ms/epoch - 2ms/step 313/313 - 2s - loss: 0.0702 - accuracy: 0.9776 - 2s/epoch - 6ms/step
...@@ -224,7 +223,7 @@ Train and evluate the model ...@@ -224,7 +223,7 @@ Train and evluate the model
Report final result Report final result
------------------- -------------------
Report final accuracy to NNI so the tuning algorithm can predict best hyperparameters. Report final accuracy to NNI so the tuning algorithm can suggest better hyperparameters.
.. GENERATED FROM PYTHON SOURCE LINES 88-89 .. GENERATED FROM PYTHON SOURCE LINES 88-89
...@@ -241,7 +240,7 @@ Report final accuracy to NNI so the tuning algorithm can predict best hyperparam ...@@ -241,7 +240,7 @@ Report final accuracy to NNI so the tuning algorithm can predict best hyperparam
.. code-block:: none .. code-block:: none
[2022-03-07 02:38:06] INFO (nni/MainThread) Final result: 0.9771999716758728 [2022-03-21 01:27:08] INFO (nni/MainThread) Final result: 0.9775999784469604
...@@ -249,7 +248,7 @@ Report final accuracy to NNI so the tuning algorithm can predict best hyperparam ...@@ -249,7 +248,7 @@ Report final accuracy to NNI so the tuning algorithm can predict best hyperparam
.. rst-class:: sphx-glr-timing .. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 44.370 seconds) **Total running time of the script:** ( 2 minutes 27.156 seconds)
.. _sphx_glr_download_tutorials_hpo_quickstart_tensorflow_model.py: .. _sphx_glr_download_tutorials_hpo_quickstart_tensorflow_model.py:
......
docs/source/tutorials/hpo_quickstart_tensorflow/sg_execution_times.rst
View file @ fbee0df1
...@@ -5,10 +5,10 @@ ...@@ -5,10 +5,10 @@
Computation times Computation times
================= =================
**12:45.612** total execution time for **tutorials_hpo_quickstart_tensorflow** files: **02:27.156** total execution time for **tutorials_hpo_quickstart_tensorflow** files:
+-----------------------------------------------------------------------------+-----------+--------+ +-----------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_main.py` (``main.py``) | 12:45.612 | 0.0 MB | | :ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_model.py` (``model.py``) | 02:27.156 | 0.0 MB |
+-----------------------------------------------------------------------------+-----------+--------+ +-----------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_model.py` (``model.py``) | 00:00.000 | 0.0 MB | | :ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_main.py` (``main.py``) | 00:00.000 | 0.0 MB |
+-----------------------------------------------------------------------------+-----------+--------+ +-----------------------------------------------------------------------------+-----------+--------+
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment