
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/hpo_quickstart_pytorch/main.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_hpo_quickstart_pytorch_main.py:


HPO Quickstart with PyTorch
===========================
This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.

The tutorial consists of 4 steps: 

1. Modify the model for auto-tuning.
2. Define hyperparameters' search space.
3. Configure the experiment.
4. Run the experiment.

.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html

.. GENERATED FROM PYTHON SOURCE LINES 17-34

Step 1: Prepare the model
-------------------------
In first step, we need to prepare the model to be tuned.

The model should be put in a separate script.
It will be evaluated many times concurrently,
and possibly will be trained on distributed platforms.

In this tutorial, the model is defined in :doc:`model.py <model>`.

In short, it is a PyTorch model with 3 additional API calls:

1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.
2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.
3. Use :func:`nni.report_final_result` to report final accuracy.

Please understand the model code before continue to next step.

.. GENERATED FROM PYTHON SOURCE LINES 36-57

Step 2: Define search space
---------------------------
In model code, we have prepared 3 hyperparameters to be tuned:
*features*, *lr*, and *momentum*.

Here we need to define their *search space* so the tuning algorithm can sample them in desired range.

Assuming we have following prior knowledge for these hyperparameters:

1. *features* should be one of 128, 256, 512, 1024.
2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
3. *momentum* should be a float between 0 and 1.

In NNI, the space of *features* is called ``choice``;
the space of *lr* is called ``loguniform``;
and the space of *momentum* is called ``uniform``.
You may have noticed, these names are derived from ``numpy.random``.

For full specification of search space, check :doc:`the reference </hpo/search_space>`.

Now we can define the search space as follow:

.. GENERATED FROM PYTHON SOURCE LINES 57-64

.. code-block:: default


    search_space = {
        'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
        'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
        'momentum': {'_type': 'uniform', '_value': [0, 1]},
    }








.. GENERATED FROM PYTHON SOURCE LINES 65-72

Step 3: Configure the experiment
--------------------------------
NNI uses an *experiment* to manage the HPO process.
The *experiment config* defines how to train the models and how to explore the search space.

In this tutorial we use a *local* mode experiment,
which means models will be trained on local machine, without using any special training platform.

.. GENERATED FROM PYTHON SOURCE LINES 72-75

.. code-block:: default

    from nni.experiment import Experiment
    experiment = Experiment('local')








.. GENERATED FROM PYTHON SOURCE LINES 76-82

Now we start to configure the experiment.

Configure trial code
^^^^^^^^^^^^^^^^^^^^
In NNI evaluation of each hyperparameter set is called a *trial*.
So the model script is called *trial code*.

.. GENERATED FROM PYTHON SOURCE LINES 82-84

.. code-block:: default

    experiment.config.trial_command = 'python model.py'
    experiment.config.trial_code_directory = '.'







.. GENERATED FROM PYTHON SOURCE LINES 85-94

When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.
(`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__
is only available in standard Python, not in Jupyter Notebook.)

.. attention::

    If you are using Linux system without Conda,
    you may need to change ``"python model.py"`` to ``"python3 model.py"``.

.. GENERATED FROM PYTHON SOURCE LINES 96-98

Configure search space
^^^^^^^^^^^^^^^^^^^^^^

.. GENERATED FROM PYTHON SOURCE LINES 98-100

.. code-block:: default

    experiment.config.search_space = search_space








.. GENERATED FROM PYTHON SOURCE LINES 101-104

Configure tuning algorithm
^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we use :doc:`TPE tuner </hpo/tuners>`.

.. GENERATED FROM PYTHON SOURCE LINES 104-107

.. code-block:: default

    experiment.config.tuner.name = 'TPE'
    experiment.config.tuner.class_args['optimize_mode'] = 'maximize'








.. GENERATED FROM PYTHON SOURCE LINES 108-111

Configure how many trials to run
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.

.. GENERATED FROM PYTHON SOURCE LINES 111-113

.. code-block:: default

    experiment.config.max_trial_number = 10
    experiment.config.trial_concurrency = 2







.. GENERATED FROM PYTHON SOURCE LINES 114-124

You may also set ``max_experiment_duration = '1h'`` to limit running time.

If neither ``max_trial_number`` nor ``max_experiment_duration`` are set,
the experiment will run forever until you press Ctrl-C.

.. note::

    ``max_trial_number`` is set to 10 here for a fast example.
    In real world it should be set to a larger number.
    With default config TPE tuner requires 20 trials to warm up.

.. GENERATED FROM PYTHON SOURCE LINES 126-131

Step 4: Run the experiment
--------------------------
Now the experiment is ready. Choose a port and launch it. (Here we use port 8080.)

You can use the web portal to view experiment status: http://localhost:8080.

.. GENERATED FROM PYTHON SOURCE LINES 131-133

.. code-block:: default

    experiment.run(8080)





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [2022-04-13 12:07:29] Creating experiment, Experiment ID: hgkju3iq
    [2022-04-13 12:07:29] Starting web server...
    [2022-04-13 12:07:30] Setting up...
    [2022-04-13 12:07:30] Web portal URLs: http://127.0.0.1:8080 http://192.168.100.103:8080

    True



.. GENERATED FROM PYTHON SOURCE LINES 134-141

After the experiment is done
----------------------------
Everything is done and it is safe to exit now. The following are optional.

If you are using standard Python instead of Jupyter Notebook,
you can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,
allowing you to view the web portal after the experiment is done.

.. GENERATED FROM PYTHON SOURCE LINES 141-145

.. code-block:: default


    # input('Press enter to quit')
    experiment.stop()





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [2022-04-13 12:08:50] Stopping experiment, please wait...
    [2022-04-13 12:08:53] Experiment stopped




.. GENERATED FROM PYTHON SOURCE LINES 146-156

:meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,
so it can be omitted in your code.

After the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.

.. tip::

    This example uses :doc:`Python API </reference/experiment>` to create experiment.

    You can also create and manage experiments with :doc:`command line tool <../hpo_nnictl/nnictl>`.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 1 minutes  24.367 seconds)


.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: main.py <main.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: main.ipynb <main.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
