"[View in Colaboratory](https://colab.research.google.com/github/marksandler2/models/blob/master/research/slim/nets/mobilenet/mobilenet_example.ipynb)"
]
},
{
"metadata": {
"id": "aUVxY7xOGD1G",
"colab_type": "toc"
},
"cell_type": "markdown",
"source": [
">[Prerequisites (downloading tensorflow_models and checkpoints)](#scrollTo=T_cETKXHDTXu)\n",
"\n",
">[Checkpoint based inference](#scrollTo=fxMe7_pkk_Vo)\n",
"\n",
">[Frozen inference](#scrollTo=PlwvpK3ElBk6)\n",
"\n"
]
},
{
"metadata": {
"id": "T_cETKXHDTXu",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Prerequisites (downloading tensorflow_models and checkpoints)"
"print('Successfully downloaded checkpoint from ', url,\n",
" '. It is available as', checkpoint)\n"
],
"execution_count": 2,
"outputs": [
{
"output_type": "stream",
"text": [
"Successfully downloaded checkpoint from https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_224.tgz . It is available as mobilenet_v2_1.0_224.ckpt\n"
For Mobilenet V2 see this file [mobilenet/README.md]
# MobileNet_v1
[MobileNets](https://arxiv.org/abs/1704.04861) are small, low-latency, low-power models parameterized to meet the resource constraints of a variety of use cases. They can be built upon for classification, detection, embeddings and segmentation similar to how other popular large scale models, such as Inception, are used. MobileNets can be run efficiently on mobile devices with [TensorFlow Mobile](https://www.tensorflow.org/mobile/).
3. We use the image processing functions from the [Official version of ResNet](/official/resnet/imagenet_preprocessing.py). Please checkout the Models repository if you haven't
already, and add the Official Models to your Python path:
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "JtEZ1pCPn--z",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Get Started with Eager Execution\n",
"\n",
"Note: you can run **[this notebook, live in Google Colab](https://colab.research.google.com/github/tensorflow/models/blob/master/samples/core/get_started/eager.ipynb)** with zero setup.\n",
"\n",
"This tutorial describes how to use machine learning to *categorize* Iris flowers by species. It uses [TensorFlow](https://www.tensorflow.org)'s eager execution to 1. build a *model*, 2. *train* the model on example data, and 3. use the model to make *predictions* on unknown data. Machine Learning experience isn't required to follow this guide, but you'll need to read some Python code.\n",
"\n",
"## TensorFlow programming\n",
"\n",
"There many [TensorFlow APIs](https://www.tensorflow.org/api_docs/python/) available, but we recommend starting with these high-level TensorFlow concepts:\n",
"\n",
"* Enable an [eager execution](https://www.tensorflow.org/programmers_guide/eager) development environment,\n",
"* Import data with the [Datasets API](https://www.tensorflow.org/programmers_guide/datasets),\n",
"* Build models and layers with TensorFlow's [Keras API](https://keras.io/getting-started/sequential-model-guide/).\n",
"\n",
"This tutorial shows these APIs and is structured like many other TensorFlow programs:\n",
"\n",
"1. Import and parse the data sets.\n",
"2. Select the type of model.\n",
"3. Train the model.\n",
"4. Evaluate the model's effectiveness.\n",
"5. Use the trained model to make predictions.\n",
"\n",
"To learn more about using TensorFlow, see the [Getting Started guide](https://www.tensorflow.org/get_started/) and the [example tutorials](https://www.tensorflow.org/tutorials/). If you'd like to learn about the basics of machine learning, consider taking the [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/).\n",
"\n",
"## Run the notebook\n",
"\n",
"This tutorial is available as an interactive [Colab notebook](https://colab.research.google.com) for you to run and change the Python code directly in the browser. The notebook handles setup and dependencies while you \"play\" cells to execute the code blocks. This is a fun way to explore the program and test ideas. If you are unfamiliar with Python notebook environments, there are a couple of things to keep in mind:\n",
"\n",
"1. Executing code requires connecting to a runtime environment. In the Colab notebook menu, select *Runtime > Connect to runtime...*\n",
"2. Notebook cells are arranged sequentially to gradually build the program. Typically, later code cells depend on prior code cells, though you can always rerun a code block. To execute the entire notebook in order, select *Runtime > Run all*. To rerun a code cell, select the cell and click the *play icon* on the left."
]
},
{
"metadata": {
"id": "yNr7H-AIoLOR",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Setup program"
]
},
{
"metadata": {
"id": "6qoYFqQ89aV3",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Install the latest version of TensorFlow\n",
"\n",
"This tutorial uses eager execution features available in [TensorFlow 1.7](https://www.tensorflow.org/install/). (You may need to restart the runtime after upgrading.)"
]
},
{
"metadata": {
"id": "jBmKxLVy9Uhg",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"!pip install --upgrade tensorflow"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "1J3AuPBT9gyR",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Configure imports and eager execution\n",
"\n",
"Import the required Python modules, including TensorFlow, and enable eager execution for this program. Eager execution makes TensorFlow evaluate operations immediately, returning concrete values instead of creating a [computational graph](https://www.tensorflow.org/programmers_guide/graphs) that is executed later. If you are used to a REPL or the `python` interactive console, you'll feel at home.\n",
"\n",
"Once eager execution is enabled, it *cannot* be disabled within the same program. See the [eager execution guide](https://www.tensorflow.org/programmers_guide/eager) for more details."
"Imagine you are a botanist seeking an automated way to categorize each Iris flower you find. Machine learning provides many algorithms to statistically classify flowers. For instance, a sophisticated machine learning program could classify flowers based on photographs. Our ambitions are more modest—we're going to classify Iris flowers based on the length and width measurements of their [sepals](https://en.wikipedia.org/wiki/Sepal) and [petals](https://en.wikipedia.org/wiki/Petal).\n",
"\n",
"The Iris genus entails about 300 species, but our program will classify only the following three:\n",
" alt=\"Petal geometry compared for three iris species: Iris setosa, Iris virginica, and Iris versicolor\">\n",
" </td></tr>\n",
" <tr><td align=\"center\">\n",
" <b>Figure 1.</b> <a href=\"https://commons.wikimedia.org/w/index.php?curid=170298\">Iris setosa</a> (by <a href=\"https://commons.wikimedia.org/wiki/User:Radomil\">Radomil</a>, CC BY-SA 3.0), <a href=\"https://commons.wikimedia.org/w/index.php?curid=248095\">Iris versicolor</a>, (by <a href=\"https://commons.wikimedia.org/wiki/User:Dlanglois\">Dlanglois</a>, CC BY-SA 3.0), and <a href=\"https://www.flickr.com/photos/33397993@N05/3352169862\">Iris virginica</a> (by <a href=\"https://www.flickr.com/photos/33397993@N05\">Frank Mayfield</a>, CC BY-SA 2.0).<br/> \n",
" </td></tr>\n",
"</table>\n",
"\n",
"Fortunately, someone has already created a [data set of 120 Iris flowers](https://en.wikipedia.org/wiki/Iris_flower_data_set) with the sepal and petal measurements. This is a classic dataset that is popular for beginner machine learning classification problems."
]
},
{
"metadata": {
"id": "3Px6KAg0Jowz",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Import and parse the training dataset\n",
"\n",
"We need to download the dataset file and convert it to a structure that can be used by this Python program.\n",
"\n",
"### Download the dataset\n",
"\n",
"Download the training dataset file using the [tf.keras.utils.get_file](https://www.tensorflow.org/api_docs/python/tf/keras/utils/get_file) function. This returns the file path of the downloaded file."
"print(\"Local copy of the dataset file: {}\".format(train_dataset_fp))"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "qnX1-aLors4S",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Inspect the data\n",
"\n",
"This dataset, `iris_training.csv`, is a plain text file that stores tabular data formatted as comma-separated values (CSV). Use the `head -n5` command to take a peak at the first five entries:"
]
},
{
"metadata": {
"id": "FQvb_JYdrpPm",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"!head -n5 {train_dataset_fp}"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "kQhzD6P-uBoq",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"From this view of the dataset, we see the following:\n",
"\n",
"1. The first line is a header containing information about the dataset:\n",
" * There are 120 total examples. Each example has four features and one of three possible label names. \n",
"2. Subsequent rows are data records, one *[example](https://developers.google.com/machine-learning/glossary/#example)* per line, where:\n",
" * The first four fields are *[features](https://developers.google.com/machine-learning/glossary/#feature)*: these are characteristics of an example. Here, the fields hold float numbers representing flower measurements.\n",
" * The last column is the *[label](https://developers.google.com/machine-learning/glossary/#label)*: this is the value we want to predict. For this dataset, it's an integer value of 0, 1, or 2 that corresponds to a flower name.\n",
"\n",
"Each label is associated with string name (for example, \"setosa\"), but machine learning typically relies on numeric values. The label numbers are mapped to a named representation, such as:\n",
"\n",
"* `0`: Iris setosa\n",
"* `1`: Iris versicolor\n",
"* `2`: Iris virginica\n",
"\n",
"For more information about features and labels, see the [ML Terminology section of the Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/framing/ml-terminology)."
]
},
{
"metadata": {
"id": "dqPkQExM2Pwt",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Parse the dataset\n",
"\n",
"Since our dataset is a CSV-formatted text file, we'll parse the feature and label values into a format our Python model can use. Each line—or row—in the file is passed to the `parse_csv` function which grabs the first four feature fields and combines them into a single tensor. Then, the last field is parsed as the label. The function returns *both* the `features` and `label` tensors:"
"TensorFlow's [Dataset API](https://www.tensorflow.org/programmers_guide/datasets) handles many common cases for feeding data into a model. This is a high-level API for reading data and transforming it into a form used for training. See the [Datasets Quick Start guide](https://www.tensorflow.org/get_started/datasets_quickstart) for more information.\n",
"\n",
"This program uses [tf.data.TextLineDataset](https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset) to load a CSV-formatted text file and is parsed with our `parse_csv` function. A [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) represents an input pipeline as a collection of elements and a series of transformations that act on those elements. Transformation methods are chained together or called sequentially—just make sure to keep a reference to the returned `Dataset` object.\n",
"\n",
"Training works best if the examples are in random order. Use `tf.data.Dataset.shuffle` to randomize entries, setting `buffer_size` to a value larger than the number of examples (120 in this case). To train the model faster, the dataset's *[batch size](https://developers.google.com/machine-learning/glossary/#batch_size)* is set to `32` examples to train at once."
"A *[model](https://developers.google.com/machine-learning/crash-course/glossary#model)* is the relationship between features and the label. For the Iris classification problem, the model defines the relationship between the sepal and petal measurements and the predicted Iris species. Some simple models can be described with a few lines of algebra, but complex machine learning models have a large number of parameters that are difficult to summarize.\n",
"\n",
"Could you determine the relationship between the four features and the Iris species *without* using machine learning? That is, could you use traditional programming techniques (for example, a lot of conditional statements) to create a model? Perhaps—if you analyzed the dataset long enough to determine the relationships between petal and sepal measurements to a particular species. And this becomes difficult—maybe impossible—on more complicated datasets. A good machine learning approach *determines the model for you*. If you feed enough representative examples into the right machine learning model type, the program will figure out the relationships for you.\n",
"\n",
"### Select the model\n",
"\n",
"We need to select the kind of model to train. There are many types of models and picking a good one takes experience. This tutorial uses a neural network to solve the Iris classification problem. *[Neural networks](https://developers.google.com/machine-learning/glossary/#neural_network)* can find complex relationships between features and the label. It is a highly-structured graph, organized into one or more *[hidden layers](https://developers.google.com/machine-learning/glossary/#hidden_layer)*. Each hidden layer consists of one or more *[neurons](https://developers.google.com/machine-learning/glossary/#neuron)*. There are several categories of neural networks and this program uses a dense, or *[fully-connected neural network](https://developers.google.com/machine-learning/glossary/#fully_connected_layer)*: the neurons in one layer receive input connections from *every* neuron in the previous layer. For example, Figure 2 illustrates a dense neural network consisting of an input layer, two hidden layers, and an output layer:\n",
" alt=\"A diagram of the network architecture: Inputs, 2 hidden layers, and outputs\">\n",
" </td></tr>\n",
" <tr><td align=\"center\">\n",
" <b>Figure 2.</b> A neural network with features, hidden layers, and predictions.<br/> \n",
" </td></tr>\n",
"</table>\n",
"\n",
"When the model from Figure 2 is trained and fed an unlabeled example, it yields three predictions: the likelihood that this flower is the given Iris species. This prediction is called *[inference](https://developers.google.com/machine-learning/crash-course/glossary#inference)*. For this example, the sum of the output predictions are 1.0. In Figure 2, this prediction breaks down as: `0.03` for *Iris setosa*, `0.95` for *Iris versicolor*, and `0.02` for *Iris virginica*. This means that the model predicts—with 95% probability—that an unlabeled example flower is an *Iris versicolor*."
]
},
{
"metadata": {
"id": "W23DIMVPQEBt",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Create a model using Keras\n",
"\n",
"The TensorFlow [tf.keras](https://www.tensorflow.org/api_docs/python/tf/keras) API is the preferred way to create models and layers. This makes it easy to build models and experiment while Keras handles the complexity of connecting everything together. See the [Keras documentation](https://keras.io/) for details.\n",
"\n",
"The [tf.keras.Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) model is a linear stack of layers. Its constructor takes a list of layer instances, in this case, two [Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layers with 10 nodes each, and an output layer with 3 nodes representing our label predictions. The first layer's `input_shape` parameter corresponds to the amount of features from the dataset, and is required."
"The *[activation function](https://developers.google.com/machine-learning/crash-course/glossary#activation_function)* determines the output of a single neuron to the next layer. This is loosely based on how brain neurons are connected. There are many [available activations](https://www.tensorflow.org/api_docs/python/tf/keras/activations), but [ReLU](https://developers.google.com/machine-learning/crash-course/glossary#ReLU) is common for hidden layers.\n",
"\n",
"The ideal number of hidden layers and neurons depends on the problem and the dataset. Like many aspects of machine learning, picking the best shape of the neural network requires a mixture of knowledge and experimentation. As a rule of thumb, increasing the number of hidden layers and neurons typically creates a more powerful model, which requires more data to train effectively."
]
},
{
"metadata": {
"id": "Vzq2E5J2QMtw",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Train the model\n",
"\n",
"*[Training](https://developers.google.com/machine-learning/crash-course/glossary#training)* is the stage of machine learning when the model is gradually optimized, or the model *learns* the dataset. The goal is to learn enough about the structure of the training dataset to make predictions about unseen data. If you learn *too much* about the training dataset, then the predictions only work for the data it has seen and will not be generalizable. This problem is called *[overfitting](https://developers.google.com/machine-learning/crash-course/glossary#overfitting)*—it's like memorizing the answers instead of understanding how to solve a problem.\n",
"\n",
"The Iris classification problem is an example of *[supervised machine learning](https://developers.google.com/machine-learning/glossary/#supervised_machine_learning)*: the model is trained from examples that contain labels. In *[unsupervised machine learning](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning)*, the examples don't contain labels. Instead, the model typically finds patterns among the features."
]
},
{
"metadata": {
"id": "RaKp8aEjKX6B",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Define the loss and gradient function\n",
"\n",
"Both training and evaluation stages need to calculate the model's *[loss](https://developers.google.com/machine-learning/crash-course/glossary#loss)*. This measures how off a model's predictions are from the desired label, in other words, how bad the model is performing. We want to minimize, or optimize, this value.\n",
"\n",
"Our model will calculate its loss using the [tf.losses.sparse_softmax_cross_entropy](https://www.tensorflow.org/api_docs/python/tf/losses/sparse_softmax_cross_entropy) function which takes the model's prediction and the desired label. The returned loss value is progressively larger as the prediction gets worse."
"The `grad` function uses the `loss` function and the [tfe.GradientTape](https://www.tensorflow.org/api_docs/python/tf/contrib/eager/GradientTape) to record operations that compute the *[gradients](https://developers.google.com/machine-learning/crash-course/glossary#gradient)* used to optimize our model. For more examples of this, see the [eager execution guide](https://www.tensorflow.org/programmers_guide/eager)."
]
},
{
"metadata": {
"id": "lOxFimtlKruu",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Create an optimizer\n",
"\n",
"An *[optimizer](https://developers.google.com/machine-learning/crash-course/glossary#optimizer)* applies the computed gradients to the model's variables to minimize the `loss` function. You can think of a curved surface (see Figure 3) and we want to find its lowest point by walking around. The gradients point in the direction of steepest the ascent—so we'll travel the opposite way and move down the hill. By iteratively calculating the loss and gradients for each *step* (or *[learning rate](https://developers.google.com/machine-learning/crash-course/glossary#learning_rate)*), we'll adjust the model during training. Gradually, the model will find the best combination of weights and bias to minimize loss. And the lower the loss, the better the model's predictions.\n",
" alt=\"Optimization algorthims visualized over time in 3D space.\">\n",
" </td></tr>\n",
" <tr><td align=\"center\">\n",
" <b>Figure 3.</b> Optimization algorthims visualized over time in 3D space. (Source: <a href=\"http://cs231n.github.io/neural-networks-3/\">Stanford class CS231n</a>, MIT License)<br/> \n",
" </td></tr>\n",
"</table>\n",
"\n",
"TensorFlow has many [optimization algorithms](https://www.tensorflow.org/api_guides/python/train) available for training. This model uses the [tf.train.GradientDescentOptimizer](https://www.tensorflow.org/api_docs/python/tf/train/GradientDescentOptimizer) that implements the *[standard gradient descent](https://developers.google.com/machine-learning/crash-course/glossary#gradient_descent)* (SGD) algorithm. The `learning_rate` sets the step size to take for each iteration down the hill. This is a *hyperparameter* that you'll commonly adjust to achieve better results."
"With all the pieces in place, the model is ready for training! A training loop feeds the dataset examples into the model to help it make better predictions. The following code block sets up these training steps:\n",
"\n",
"1. Iterate each epoch. An epoch is one pass through the dataset.\n",
"2. Within an epoch, iterate over each example in the training `Dataset` grabbing its *features* (`x`) and *label* (`y`).\n",
"3. Using the example's features, make a prediction and compare it with the label. Measure the inaccuracy of the prediction and use that to calculate the model's loss and gradients.\n",
"4. Use an `optimizer` to update the model's variables.\n",
"5. Keep track of some stats for visualization.\n",
"6. Repeat for each epoch.\n",
"\n",
"The `num_epochs` variable is the amount of times to loop over the dataset collection. Counter-intuitively, training a model longer does not guarantee a better model. `num_epochs` is a *[hyperparameter](https://developers.google.com/machine-learning/glossary/#hyperparameter)* that you can tune. Choosing the right number usually requires both experience and experimentation."
]
},
{
"metadata": {
"id": "AIgulGRUhpto",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"## Note: Rerunning this cell uses the same model variables\n",
"While it's helpful to print out the model's training progress, it's often *more helpful* to see this progress. [TensorBoard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard) is a nice visualization tool that is packaged with TensorFlow, but we can create basic charts using the `mathplotlib` module.\n",
"\n",
"Interpreting these charts takes some experience, but you really want to see the *loss* go down and the *accuracy* go up."
"Now that the model is trained, we can get some statistics on its performance.\n",
"\n",
"*Evaluating* means determining how effectively the model makes predictions. To determine the model's effectiveness at Iris classification, pass some sepal and petal measurements to the model and ask the model to predict what Iris species they represent. Then compare the model's prediction against the actual label. For example, a model that picked the correct species on half the input examples has an *[accuracy](https://developers.google.com/machine-learning/glossary/#accuracy)* of `0.5`. Figure 4 shows a slightly more effective model, getting 4 out of 5 predictions correct at 80% accuracy:\n",
" <b>Figure 4.</b> An Iris classifier that is 80% accurate.<br/> \n",
" </td></tr>\n",
"</table>"
]
},
{
"metadata": {
"id": "z-EvK7hGL0d8",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Setup the test dataset\n",
"\n",
"Evaluating the model is similar to training the model. The biggest difference is the examples come from a separate *[test set](https://developers.google.com/machine-learning/crash-course/glossary#test_set)* rather than the training set. To fairly assess a model's effectiveness, the examples used to evaluate a model must be different from the examples used to train the model.\n",
"\n",
"The setup for the test `Dataset` is similar to the setup for training `Dataset`. Download the CSV text file and parse that values, then give it a little shuffle:"
"test_dataset = test_dataset.batch(32) # use the same batch size as the training set"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "HFuOKXJdMAdm",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Evaluate the model on the test dataset\n",
"\n",
"Unlike the training stage, the model only evaluates a single [epoch](https://developers.google.com/machine-learning/glossary/#epoch) of the test data. In the following code cell, we iterate over each example in the test set and compare the model's prediction against the actual label. This is used to measure the model's accuracy across the entire test set."
"print(\"Test set accuracy: {:.3%}\".format(test_accuracy.result()))"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "7Li2r1tYvW7S",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Use the trained model to make predictions\n",
"\n",
"We've trained a model and \"proven\" that it's good—but not perfect—at classifying Iris species. Now let's use the trained model to make some predictions on [unlabeled examples](https://developers.google.com/machine-learning/glossary/#unlabeled_example); that is, on examples that contain features but not a label.\n",
"\n",
"In real-life, the unlabeled examples could come from lots of different sources including apps, CSV files, and data feeds. For now, we're going to manually provide three unlabeled examples to predict their labels. Recall, the label numbers are mapped to a named representation as:\n",
"To dig deeper into machine learning models, take a look at the TensorFlow [Programmer's Guide](https://www.tensorflow.org/programmers_guide/) and check out the [community](https://www.tensorflow.org/community/)."
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "g7nGs4mzVUHP",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Eager execution\n",
"\n",
"Note: you can run **[this notebook, live in Google Colab](https://colab.research.google.com/github/tensorflow/models/blob/master/samples/outreach/demos/eager_execution.ipynb)** with zero setup. \n",
"2. _A NumPy-like library for numerical computation and machine learning. Case study: Fitting a huber regression_.\n",
"3. _Neural networks. Case study: Training a multi-layer RNN._\n",
"4. _Exercises: Batching; debugging._\n",
"5. _Further reading_"
]
},
{
"metadata": {
"id": "ZVKfj5ttVkqz",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 1. Enabling eager execution!\n",
"\n",
"A single function call is all you need to enable eager execution: `tf.enable_eager_execution()`. You should invoke this function before calling into any other TensorFlow APIs --- the simplest way to satisfy this requirement is to make `tf.enable_eager_execution()` the first line of your `main` function.\n"
]
},
{
"metadata": {
"id": "C783D4QKVlK1",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"!pip install -q -U tf-nightly\n",
"\n",
"import tensorflow as tf\n",
"\n",
"tf.enable_eager_execution()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "trrHQBM1VnD0",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 2. A NumPy-like library for numerical computation and machine learning\n",
"Enabling eager execution transforms TensorFlow into an **imperative** library for numerical computation, automatic differentiation, and machine learning. When executing eagerly, _TensorFlow no longer behaves like a dataflow graph engine_: Tensors are backed by NumPy arrays (goodbye, placeholders!), and TensorFlow operations execute *immediately* via Python (goodbye, sessions!)."
]
},
{
"metadata": {
"id": "MLUSuZuccgmF",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Numpy-like usage\n",
"\n",
"Tensors are backed by numpy arrays, which are accessible via their `.numpy()`\n",
"method."
]
},
{
"metadata": {
"id": "lzrktlC0cPi1",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"A = tf.constant([[2.0, 0.0], [0.0, 3.0]])"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "F5oDeGhYcX6c",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import numpy as np\n",
"\n",
"print(\"Tensors are backed by NumPy arrays, which are accessible through their \"\n",
"Create variables with `tf.contrib.eager.Variable`, and use `tf.GradientTape`\n",
"to compute gradients with respect to them."
]
},
{
"metadata": {
"id": "PGAqOzqzccwd",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import tensorflow.contrib.eager as tfe\n",
"w = tfe.Variable(3.0)\n",
"with tf.GradientTape() as tape:\n",
" loss = w ** 2\n",
"dw, = tape.gradient(loss, [w])\n",
"print(\"\\nYou can use `tf.GradientTape` to compute the gradient of a \"\n",
" \"computation with respect to a list of `tf.contrib.eager.Variable`s;\\n\"\n",
" \"for example, `tape.gradient(loss, [w])`, where `loss` = w ** 2 and \"\n",
" \"`w` == 3.0, yields`\", dw,\"`.\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "gZFXrVTKdFnl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### GPU usage\n",
"Eager execution lets you offload computation to hardware accelerators like\n",
"GPUs, if you have any available."
]
},
{
"metadata": {
"id": "ER-Hsk3RVmX9",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "both"
},
"cell_type": "code",
"source": [
"if tf.test.is_gpu_available() > 0:\n",
" with tf.device(tf.test.gpu_device_name()):\n",
" print(tf.matmul(A, A))"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "JQ8kQT99VqDk",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Fitting a Huber regression\n",
"\n",
"If you come from a scientific or numerical computing background, eager execution should feel natural to you. Not only does it stand on its own as an accelerator-compatible library for numerical computation, it also interoperates with popular Python packages like NumPy and Matplotlib. To demonstrate this fact, in this section, we fit and evaluate a regression using a [Huber regression](https://en.wikipedia.org/wiki/Huber_loss), writing our code in a NumPy-like way and making use of Python control flow."
]
},
{
"metadata": {
"id": "6dXt0WfBK9-7",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Data generation\n",
"\n",
"Our dataset for this example has many outliers — least-squares would be a poor choice."
" # You can freely mix Tensors and NumPy arrays in your computations:\n",
" # `sign` is a NumPy array, but the other symbols below are Tensors.\n",
" Y = sign * (w_star * X + b_star + noise) \n",
" return X, Y\n",
"\n",
"X, Y = gen_regression_data()\n",
"plt.plot(X, Y, \"go\") # You can plot Tensors!\n",
"plt.title(\"Observed data\")\n",
"plt.show()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "sYumjOrdMRFM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Huber loss\n",
"The Huber loss function is piecewise function that is quadratic for small inputs and linear otherwise; for that reason, using a Huber loss gives considerably less weight to outliers than least-squares does. When eager execution is enabled, we can implement the Huber function in the natural way, using **Python control flow**."
]
},
{
"metadata": {
"id": "anflUCeaVtK8",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"def huber_loss(y, y_hat, m=1.0):\n",
" # Enabling eager execution lets you use Python control flow.\n",
" delta = tf.abs(y - y_hat)\n",
" return delta ** 2 if delta <= m else m * (2 * delta - m)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "0_OALYGwM7ma",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### A simple class for regressions\n",
"\n",
"The next cell encapsulates a linear regression model in a Python class and defines a\n",
"function that fits the model using a stochastic optimizer."
]
},
{
"metadata": {
"id": "-90due2RVuDF",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"import time\n",
"\n",
"from google.colab import widgets\n",
"import tensorflow.contrib.eager as tfe # Needed to create tfe.Variable objects.\n",
"### Enabling eager execution lets you debug your code on-the-fly; use `pdb` and print statements to your heart's content.\n",
"\n",
"Check out exercise 2 towards the bottom of this notebook for a hands-on look at how eager simplifies model debugging."
]
},
{
"metadata": {
"id": "DNHJpCyNVwA9",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import pdb\n",
"\n",
"def buggy_loss(y, y_hat):\n",
" pdb.set_trace()\n",
" huber_loss(y, y_hat)\n",
" \n",
"print(\"Type 'exit' to stop the debugger, or 's' to step into `huber_loss` and \"\n",
" \"'n' to step through it.\")\n",
"try:\n",
" buggy_loss(1.0, 2.0)\n",
"except:\n",
" pass"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "mvI3ljk-vJ_h",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Leverage the Python profiler to dig into the relative costs of training your model.\n",
"\n",
"If you run the below cell, you'll see that most of the time is spent computing gradients and binary operations, which is sensible considering our loss function."
"print(\"Most of the time is spent during backpropagation and binary operations.\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "5AeTwwPobkaJ",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 3. Neural networks\n",
"\n",
"While eager execution can certainly be used as a library for numerical computation, it shines as a library for deep learning: TensorFlow provides a suite of tools for deep learning research and development, most of which are compatible with eager execution. In this section, we put some of these tools to use to build _RNNColorbot_, an RNN that takes as input names of colors and predicts their corresponding RGB tuples. "
]
},
{
"metadata": {
"id": "6IcmEQ-jpTMO",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Constructing a data pipeline\n",
"\n",
"**[`tf.data`](https://www.tensorflow.org/api_guides/python/reading_data#_tf_data_API) is TensorFlow's canonical API for constructing input pipelines.** `tf.data` lets you easily construct multi-stage pipelines that supply data to your networks during training and inference. The following cells defines methods that download and format the data needed for RNNColorbot; the details aren't important (read them in the privacy of your own home if you so wish), but make sure to run the cells before proceeding."
]
},
{
"metadata": {
"id": "dcUC3Ma8bjgY",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"import os\n",
"import six\n",
"from six.moves import urllib\n",
"\n",
"\n",
"def parse(line):\n",
" \"\"\"Parse a line from the colors dataset.\"\"\"\n",
"TensorFlow packages several APIs for creating neural networks in a modular fashion. **The canonical way to define neural networks in TensorFlow is to encapsulate your model in a class that inherits from `tf.keras.Model`**. You should think of `tf.keras.Model` as a container of **[object-oriented layers](https://www.tensorflow.org/api_docs/python/tf/layers)**, TensorFlow's building blocks for constructing neural networks (*e.g.*, `tf.layers.Dense`, `tf.layers.Conv2D`). Every `Layer` object that is set as an attribute of a `Model` is automatically tracked by the latter, letting you access `Layer`-contained variables by invoking `Model`'s `.variables()` method. Most important, **inheriting from `tf.keras.Model` makes it easy to checkpoint your model and to subsequently restore it** --- more on that later. \n",
"\n",
"The following cell exemplifies our high-level neural network APIs. Note that `RNNColorbot` encapsulates only the model definition and prediction generation logic. The loss, training, and evaluation functions exist outside the class definition: conceptually, the model doesn't need know how to train and benchmark itself."
]
},
{
"metadata": {
"id": "NlKcdvT9leQ2",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"class RNNColorbot(tf.keras.Model):\n",
" \"\"\"Multi-layer RNN that predicts RGB tuples given color names.\n",
"The next cell **trains** our `RNNColorbot`, **restoring and saving checkpoints** of the learned variables along the way. Thanks to checkpointing, every run of the below cell will resume training from wherever the previous run left off. For more on checkpointing, take a look at our [user guide](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/g3doc/guide.md#checkpointing-trained-variables)."
"print(\"Colorbot is ready to generate colors!\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "1HdJk37R1xz9",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Paint me a color, Colorbot!\n",
"\n",
"We can interact with RNNColorbot in a natural way; no need to thread NumPy arrays into placeholders through feed dicts.\n",
"So go ahead and ask RNNColorbot to paint you some colors. If they're not to your liking, re-run the previous cell to resume training from where we left off, and then re-run the next one for updated results."
]
},
{
"metadata": {
"id": "LXAYjopasyWr",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"tb = widgets.TabBar([\"RNN Colorbot\"])\n",
"while True:\n",
" with tb.output_to(0):\n",
" try:\n",
" color_name = six.moves.input(\n",
" \"Give me a color name (or press 'enter' to exit): \")\n",
" clipped_preds = tuple(min(float(p), 1.0) for p in preds)\n",
" rgb = tuple(int(p * 255) for p in clipped_preds)\n",
" with tb.output_to(0):\n",
" tb.clear_tab()\n",
" print(\"Predicted RGB tuple:\", rgb)\n",
" plt.imshow([[clipped_preds]])\n",
" plt.title(color_name)\n",
" plt.show()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "aJopbdYiXXQM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 4. Exercises"
]
},
{
"metadata": {
"id": "Nt2bZ3SNq0bl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Exercise 1: Batching\n",
"\n",
"Executing operations eagerly incurs small overheads; these overheads become neglible when amortized over batched operations. In this exercise, we explore the relationship between batching and performance by revisiting our Huber regression example."
]
},
{
"metadata": {
"id": "U5NR8vOY-4Xx",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"# Our original implementation of `huber_loss` is not compatible with non-scalar\n",
"# data. Your task is to fix that. For your convenience, the original\n",
"# implementation is reproduced below.\n",
"#\n",
"# def huber_loss(y, y_hat, m=1.0):\n",
"# delta = tf.abs(y - y_hat)\n",
"# return delta ** 2 if delta <= m else m * (2 * delta - m)\n",
"#\n",
"def batched_huber_loss(y, y_hat, m=1.0):\n",
" # TODO: Uncomment out the below code and replace `...` with your solution.\n",
"We've heard you loud and clear: TensorFlow programs that construct and execute graphs are difficult to debug. By design, enabling eager execution vastly simplifies the process of debugging TensorFlow programs. Once eager execution is enabled, you can step through your models using `pdb` and bisect them with `print` statements. The best way to understand the extent to which eager execution simplifies debugging is to debug a model yourself. `BuggyModel` below has two bugs lurking in it. Execute the following cell, read the error message, and go hunt some bugs!\n",
"\n",
"*Hint: As is often the case with TensorFlow programs, both bugs are related to the shapes of Tensors.*\n",
"\n",
"*Hint: You might find `tf.layers.flatten` useful.*"
"* our [collection of example models](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples), which includes a convolutional model for [MNIST](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/mnist) classification, a [GAN](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/gan), a [recursive neural network](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/spinn), and more;\n",
"* [this advanced notebook](https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/contrib/autograph/examples/notebooks/dev_summit_2018_demo.ipynb), which explains how to build and execute graphs while eager execution is enabled and how to call into eager execution while constructing a graph, and which also introduces Autograph, a source-code translation tool that automatically generates graph-construction code from dynamic eager code.\n",
