:orphan:

.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/hpo_quickstart_pytorch/main.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_hpo_quickstart_pytorch_main.py:


NNI HPO Quickstart with PyTorch
===============================
This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.

The tutorial consists of 4 steps: 

 1. Modify the model for auto-tuning.
 2. Define hyperparameters' search space.
 3. Configure the experiment.
 4. Run the experiment.

.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html

.. GENERATED FROM PYTHON SOURCE LINES 17-28

Step 1: Prepare the model
-------------------------
In first step, you need to prepare the model to be tuned.

The model should be put in a separate script.
It will be evaluated many times concurrently,
and possibly will be trained on distributed platforms.

In this tutorial, the model is defined in :doc:`model.py <model>`.

Please understand the model code before continue to next step.

.. GENERATED FROM PYTHON SOURCE LINES 30-51

Step 2: Define search space
---------------------------
In model code, we have prepared 3 hyperparameters to be tuned:
*features*, *lr*, and *momentum*.

Here we need to define their *search space* so the tuning algorithm can sample them in desired range.

Assuming we have following prior knowledge for these hyperparameters:

 1. *features* should be one of 128, 256, 512, 1024.
 2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
 3. *momentum* should be a float between 0 and 1.

In NNI, the space of *features* is called ``choice``;
the space of *lr* is called ``loguniform``;
and the space of *momentum* is called ``uniform``.
You may have noticed, these names are derived from ``numpy.random``.

For full specification of search space, check :doc:`the reference </hpo/search_space>`.

Now we can define the search space as follow:

.. GENERATED FROM PYTHON SOURCE LINES 51-58

.. code-block:: default


    search_space = {
        'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
        'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
        'momentum': {'_type': 'uniform', '_value': [0, 1]},
    }








.. GENERATED FROM PYTHON SOURCE LINES 59-66

Step 3: Configure the experiment
--------------------------------
NNI uses an *experiment* to manage the HPO process.
The *experiment config* defines how to train the models and how to explore the search space.

In this tutorial we use a *local* mode experiment,
which means models will be trained on local machine, without using any special training platform.

.. GENERATED FROM PYTHON SOURCE LINES 66-69

.. code-block:: default

    from nni.experiment import Experiment
    experiment = Experiment('local')








.. GENERATED FROM PYTHON SOURCE LINES 70-80

Now we start to configure the experiment.

Firstly, specify the model code.
In NNI evaluation of each hyperparameter set is called a *trial*.
So the model script is called *trial code*.

If you are using Linux system without Conda, you many need to change ``python`` to ``python3``.

When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.

.. GENERATED FROM PYTHON SOURCE LINES 80-83

.. code-block:: default

    experiment.config.trial_command = 'python model.py'
    experiment.config.trial_code_directory = '.'








.. GENERATED FROM PYTHON SOURCE LINES 84-85

Then specify the search space we defined above:

.. GENERATED FROM PYTHON SOURCE LINES 85-87

.. code-block:: default

    experiment.config.search_space = search_space








.. GENERATED FROM PYTHON SOURCE LINES 88-90

Choose a tuning algorithm.
Here we use :doc:`TPE tuner </hpo/tuners>`.

.. GENERATED FROM PYTHON SOURCE LINES 90-93

.. code-block:: default

    experiment.config.tuner.name = 'TPE'
    experiment.config.tuner.class_args['optimize_mode'] = 'maximize'








.. GENERATED FROM PYTHON SOURCE LINES 94-105

Specify how many trials to run.
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.

Please note that ``max_trial_number`` here is merely for a quick example.
With default config TPE tuner requires 20 trials to warm up.
In real world max trial number is commonly set to 100+.

You can also set ``max_experiment_duration = '1h'`` to limit running time.

And alternatively, you can skip this part and set no limit at all.
The experiment will run forever until you press Ctrl-C.

.. GENERATED FROM PYTHON SOURCE LINES 105-108

.. code-block:: default

    experiment.config.max_trial_number = 10
    experiment.config.trial_concurrency = 4








.. GENERATED FROM PYTHON SOURCE LINES 109-114

Step 4: Run the experiment
--------------------------
Now the experiment is ready. Choose a port and launch it.

You can use the web portal to view experiment status: http://localhost:8080.

.. GENERATED FROM PYTHON SOURCE LINES 114-115

.. code-block:: default

    experiment.run(8080)




.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [2022-03-18 12:58:19] Creating experiment, Experiment ID: 2ijcdvau
    [2022-03-18 12:58:19] Starting web server...
    [2022-03-18 12:58:21] Setting up...
    [2022-03-18 12:58:21] Web UI URLs: http://127.0.0.1:8080 http://172.22.104.196:8080
    [2022-03-18 13:57:44] Stopping experiment, please wait...
    [2022-03-18 13:57:47] Experiment stopped

    True




.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 59 minutes  28.055 seconds)


.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: main.py <main.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: main.ipynb <main.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
