"\n# NNI HPO Quickstart with TensorFlow\nThis tutorial optimizes the model in `official TensorFlow quickstart`_ with auto-tuning.\n\nThe tutorial consists of 4 steps: \n\n 1. Modify the model for auto-tuning.\n 2. Define hyperparameters' search space.\n 3. Configure the experiment.\n 4. Run the experiment.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Prepare the model\nIn first step, you need to prepare the model to be tuned.\n\nThe model should be put in a separate script.\nIt will be evaluated many times concurrently,\nand possibly will be trained on distributed platforms.\n\nIn this tutorial, the model is defined in :doc:`model.py <model>`.\n\nPlease understand the model code before continue to next step.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Define search space\nIn model code, we have prepared 4 hyperparameters to be tuned:\n*dense_units*, *activation_type*, *dropout_rate*, and *learning_rate*.\n\nHere we need to define their *search space* so the tuning algorithm can sample them in desired range.\n\nAssuming we have following prior knowledge for these hyperparameters:\n\n 1. *dense_units* should be one of 64, 128, 256.\n 2. *activation_type* should be one of 'relu', 'tanh', 'swish', or None.\n 3. *dropout_rate* should be a float between 0.5 and 0.9.\n 4. *learning_rate* should be a float between 0.0001 and 0.1, and it follows exponential distribution.\n\nIn NNI, the space of *dense_units* and *activation_type* is called ``choice``;\nthe space of *dropout_rate* is called ``uniform``;\nand the space of *learning_rate* is called ``loguniform``.\nYou may have noticed, these names are derived from ``numpy.random``.\n\nFor full specification of search space, check :doc:`the reference </hpo/search_space>`.\n\nNow we can define the search space as follow:\n\n"
"## Step 3: Configure the experiment\nNNI uses an *experiment* to manage the HPO process.\nThe *experiment config* defines how to train the models and how to explore the search space.\n\nIn this tutorial we use a *local* mode experiment,\nwhich means models will be trained on local machine, without using any special training platform.\n\n"
"Now we start to configure the experiment.\n\nFirstly, specify the model code.\nIn NNI evaluation of each hyperparameter set is called a *trial*.\nSo the model script is called *trial code*.\n\nIf you are using Linux system without Conda, you many need to change ``python`` to ``python3``.\n\nWhen ``trial_code_directory`` is a relative path, it relates to current working directory.\nTo run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.\n\n"
"Specify how many trials to run.\nHere we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.\n\nPlease note that ``max_trial_number`` here is merely for a quick example.\nWith default config TPE tuner requires 20 trials to warm up.\nIn real world max trial number is commonly set to 100+.\n\nYou can also set ``max_experiment_duration = '1h'`` to limit running time.\n\nAnd alternatively, you can skip this part and set no limit at all.\nThe experiment will run forever until you press Ctrl-C.\n\n"
"## Step 4: Run the experiment\nNow the experiment is ready. Choose a port and launch it.\n\nYou can use the web portal to view experiment status: http://localhost:8080.\n\n"
"\n# Port TensorFlow Quickstart to NNI\nThis is a modified version of `TensorFlow quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.\n\nFor now, we recommend to run this script directly to verify the environment.\n\nThere are only 3 key differences from the original version:\n\n 1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.\n 2. In `(Optional) Report intermediate results`_ part, it reports per-epoch accuracy for visualization.\n 3. In `Report final result`_ part, it reports final accuracy for tuner to generate next hyperparameter set.\n\n"
"## Get optimized hyperparameters\nIf run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n"
"## (Optional) Report intermediate results\nThe callback reports per-epoch accuracy to show learning curve in NNI web portal.\nAnd in :doc:`/hpo/assessors`, you will see how to leverage the metrics for early stopping.\n\nYou can safely skip this and the experiment will work fine.\n\n"
