
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/hpo_quickstart_tensorflow/main.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_tutorials_hpo_quickstart_tensorflow_main.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_hpo_quickstart_tensorflow_main.py:


HPO Quickstart with TensorFlow
==============================
This tutorial optimizes the model in `official TensorFlow quickstart`_ with auto-tuning.

The tutorial consists of 4 steps: 

1. Modify the model for auto-tuning.
2. Define hyperparameters' search space.
3. Configure the experiment.
4. Run the experiment.

.. _official TensorFlow quickstart: https://www.tensorflow.org/tutorials/quickstart/beginner

.. GENERATED FROM PYTHON SOURCE LINES 17-34

Step 1: Prepare the model
-------------------------
In first step, we need to prepare the model to be tuned.

The model should be put in a separate script.
It will be evaluated many times concurrently,
and possibly will be trained on distributed platforms.

In this tutorial, the model is defined in :doc:`model.py <model>`.

In short, it is a TensorFlow model with 3 additional API calls:

1. Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be evalutated.
2. Use :func:`nni.report_intermediate_result` to report per-epoch accuracy metrics.
3. Use :func:`nni.report_final_result` to report final accuracy.

Please understand the model code before continue to next step.

.. GENERATED FROM PYTHON SOURCE LINES 36-58

Step 2: Define search space
---------------------------
In model code, we have prepared 4 hyperparameters to be tuned:
*dense_units*, *activation_type*, *dropout_rate*, and *learning_rate*.

Here we need to define their *search space* so the tuning algorithm can sample them in desired range.

Assuming we have following prior knowledge for these hyperparameters:

1. *dense_units* should be one of 64, 128, 256.
2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.
3. *dropout_rate* should be a float between 0.5 and 0.9.
4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.

In NNI, the space of *dense_units* and *activation_type* is called ``choice``;
the space of *dropout_rate* is called ``uniform``;
and the space of *learning_rate* is called ``loguniform``.
You may have noticed, these names are derived from ``numpy.random``.

For full specification of search space, check :doc:`the reference </hpo/search_space>`.

Now we can define the search space as follow:

.. GENERATED FROM PYTHON SOURCE LINES 58-66

.. code-block:: default


    search_space = {
        'dense_units': {'_type': 'choice', '_value': [64, 128, 256]},
        'activation_type': {'_type': 'choice', '_value': ['relu', 'tanh', 'swish', None]},
        'dropout_rate': {'_type': 'uniform', '_value': [0.5, 0.9]},
        'learning_rate': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
    }








.. GENERATED FROM PYTHON SOURCE LINES 67-74

Step 3: Configure the experiment
--------------------------------
NNI uses an *experiment* to manage the HPO process.
The *experiment config* defines how to train the models and how to explore the search space.

In this tutorial we use a *local* mode experiment,
which means models will be trained on local machine, without using any special training platform.

.. GENERATED FROM PYTHON SOURCE LINES 74-77

.. code-block:: default

    from nni.experiment import Experiment
    experiment = Experiment('local')








.. GENERATED FROM PYTHON SOURCE LINES 78-84

Now we start to configure the experiment.

Configure trial code
^^^^^^^^^^^^^^^^^^^^
In NNI evaluation of each hyperparameter set is called a *trial*.
So the model script is called *trial code*.

.. GENERATED FROM PYTHON SOURCE LINES 84-86

.. code-block:: default

    experiment.config.trial_command = 'python model.py'
    experiment.config.trial_code_directory = '.'







.. GENERATED FROM PYTHON SOURCE LINES 87-96

When ``trial_code_directory`` is a relative path, it relates to current working directory.
To run ``main.py`` in a different path, you can set trial code directory to ``Path(__file__).parent``.
(`__file__ <https://docs.python.org/3.10/reference/datamodel.html#index-43>`__
is only available in standard Python, not in Jupyter Notebook.)

.. attention::

    If you are using Linux system without Conda,
    you may need to change ``"python model.py"`` to ``"python3 model.py"``.

.. GENERATED FROM PYTHON SOURCE LINES 98-100

Configure search space
^^^^^^^^^^^^^^^^^^^^^^

.. GENERATED FROM PYTHON SOURCE LINES 100-102

.. code-block:: default

    experiment.config.search_space = search_space








.. GENERATED FROM PYTHON SOURCE LINES 103-106

Configure tuning algorithm
^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we use :doc:`TPE tuner </hpo/tuners>`.

.. GENERATED FROM PYTHON SOURCE LINES 106-109

.. code-block:: default

    experiment.config.tuner.name = 'TPE'
    experiment.config.tuner.class_args['optimize_mode'] = 'maximize'








.. GENERATED FROM PYTHON SOURCE LINES 110-113

Configure how many trials to run
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 2 sets at a time.

.. GENERATED FROM PYTHON SOURCE LINES 113-115

.. code-block:: default

    experiment.config.max_trial_number = 10
    experiment.config.trial_concurrency = 2







.. GENERATED FROM PYTHON SOURCE LINES 116-126

You may also set ``max_experiment_duration = '1h'`` to limit running time.

If neither ``max_trial_number`` nor ``max_experiment_duration`` are set,
the experiment will run forever until you press Ctrl-C.

.. note::

    ``max_trial_number`` is set to 10 here for a fast example.
    In real world it should be set to a larger number.
    With default config TPE tuner requires 20 trials to warm up.

.. GENERATED FROM PYTHON SOURCE LINES 128-133

Step 4: Run the experiment
--------------------------
Now the experiment is ready. Choose a port and launch it. (Here we use port 8080.)

You can use the web portal to view experiment status: http://localhost:8080.

.. GENERATED FROM PYTHON SOURCE LINES 133-135

.. code-block:: default

    experiment.run(8080)





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [2022-04-13 12:11:34] Creating experiment, Experiment ID: enw27qxj
    [2022-04-13 12:11:34] Starting web server...
    [2022-04-13 12:11:35] Setting up...
    [2022-04-13 12:11:35] Web portal URLs: http://127.0.0.1:8080 http://192.168.100.103:8080

    True



.. GENERATED FROM PYTHON SOURCE LINES 136-143

After the experiment is done
----------------------------
Everything is done and it is safe to exit now. The following are optional.

If you are using standard Python instead of Jupyter Notebook,
you can add ``input()`` or ``signal.pause()`` to prevent Python from exiting,
allowing you to view the web portal after the experiment is done.

.. GENERATED FROM PYTHON SOURCE LINES 143-147

.. code-block:: default


    # input('Press enter to quit')
    experiment.stop()





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    [2022-04-13 12:12:55] Stopping experiment, please wait...
    [2022-04-13 12:12:58] Experiment stopped




.. GENERATED FROM PYTHON SOURCE LINES 148-158

:meth:`nni.experiment.Experiment.stop` is automatically invoked when Python exits,
so it can be omitted in your code.

After the experiment is stopped, you can run :meth:`nni.experiment.Experiment.view` to restart web portal.

.. tip::

    This example uses :doc:`Python API </reference/experiment>` to create experiment.

    You can also create and manage experiments with :doc:`command line tool <../hpo_nnictl/nnictl>`.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 1 minutes  24.384 seconds)


.. _sphx_glr_download_tutorials_hpo_quickstart_tensorflow_main.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: main.py <main.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: main.ipynb <main.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
