"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "g7nGs4mzVUHP",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Eager execution\n",
"\n",
"Note: you can run [**this notebook, live in Google Colab**](https://colab.research.google.com/github/tensorflow/models/blob/master/samples/outreach/demos/eager_execution.ipynb) with zero setup. \n",
"2. _A NumPy-like library for numerical computation and machine learning. Case study: Fitting a huber regression_.\n",
"3. _Neural networks. Case study: Training a multi-layer RNN._\n",
"4. _Exercises: Batching; debugging._\n",
"5. _Further reading_"
]
},
{
"metadata": {
"id": "ZVKfj5ttVkqz",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 1. Enabling eager execution!\n",
"\n",
"A single function call is all you need to enable eager execution: `tf.enable_eager_execution()`. You should invoke this function before calling into any other TensorFlow APIs --- the simplest way to satisfy this requirement is to make `tf.enable_eager_execution()` the first line of your `main` function.\n"
]
},
{
"metadata": {
"id": "C783D4QKVlK1",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"!pip install -q -U tf-nightly\n",
"\n",
"import tensorflow as tf\n",
"\n",
"tf.enable_eager_execution()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "trrHQBM1VnD0",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 2. A NumPy-like library for numerical computation and machine learning\n",
"Enabling eager execution transforms TensorFlow into an **imperative** library for numerical computation, automatic differentiation, and machine learning. When executing eagerly, _TensorFlow no longer behaves like a dataflow graph engine_: Tensors are backed by NumPy arrays (goodbye, placeholders!), and TensorFlow operations execute *immediately* via Python (goodbye, sessions!)."
]
},
{
"metadata": {
"id": "MLUSuZuccgmF",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Numpy-like usage\n",
"\n",
"Tensors are backed by numpy arrays, which are accessible via their `.numpy()`\n",
"method."
]
},
{
"metadata": {
"id": "lzrktlC0cPi1",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"A = tf.constant([[2.0, 0.0], [0.0, 3.0]])"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "F5oDeGhYcX6c",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import numpy as np\n",
"\n",
"print(\"Tensors are backed by NumPy arrays, which are accessible through their \"\n",
"Create variables with `tf.contrib.eager.Variable`, and use `tf.GradientTape`\n",
"to compute gradients with respect to them."
]
},
{
"metadata": {
"id": "PGAqOzqzccwd",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import tensorflow.contrib.eager as tfe\n",
"w = tfe.Variable(3.0)\n",
"with tf.GradientTape() as tape:\n",
" loss = w ** 2\n",
"dw, = tape.gradient(loss, [w])\n",
"print(\"\\nYou can use `tf.GradientTape` to compute the gradient of a \"\n",
" \"computation with respect to a list of `tf.contrib.eager.Variable`s;\\n\"\n",
" \"for example, `tape.gradient(loss, [w])`, where `loss` = w ** 2 and \"\n",
" \"`w` == 3.0, yields`\", dw,\"`.\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "gZFXrVTKdFnl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### GPU usage\n",
"Eager execution lets you offload computation to hardware accelerators like\n",
"GPUs, if you have any available."
]
},
{
"metadata": {
"id": "ER-Hsk3RVmX9",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "both"
},
"cell_type": "code",
"source": [
"if tf.test.is_gpu_available() > 0:\n",
" with tf.device(tf.test.gpu_device_name()):\n",
" print(tf.matmul(A, A))"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "JQ8kQT99VqDk",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Fitting a Huber regression\n",
"\n",
"If you come from a scientific or numerical computing background, eager execution should feel natural to you. Not only does it stand on its own as an accelerator-compatible library for numerical computation, it also interoperates with popular Python packages like NumPy and Matplotlib. To demonstrate this fact, in this section, we fit and evaluate a regression using a [Huber regression](https://en.wikipedia.org/wiki/Huber_loss), writing our code in a NumPy-like way and making use of Python control flow."
]
},
{
"metadata": {
"id": "6dXt0WfBK9-7",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Data generation\n",
"\n",
"Our dataset for this example has many outliers — least-squares would be a poor choice."
" # You can freely mix Tensors and NumPy arrays in your computations:\n",
" # `sign` is a NumPy array, but the other symbols below are Tensors.\n",
" Y = sign * (w_star * X + b_star + noise) \n",
" return X, Y\n",
"\n",
"X, Y = gen_regression_data()\n",
"plt.plot(X, Y, \"go\") # You can plot Tensors!\n",
"plt.title(\"Observed data\")\n",
"plt.show()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "sYumjOrdMRFM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Huber loss\n",
"The Huber loss function is piecewise function that is quadratic for small inputs and linear otherwise; for that reason, using a Huber loss gives considerably less weight to outliers than least-squares does. When eager execution is enabled, we can implement the Huber function in the natural way, using **Python control flow**."
]
},
{
"metadata": {
"id": "anflUCeaVtK8",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"def huber_loss(y, y_hat, m=1.0):\n",
" # Enabling eager execution lets you use Python control flow.\n",
" delta = tf.abs(y - y_hat)\n",
" return delta ** 2 if delta <= m else m * (2 * delta - m)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "0_OALYGwM7ma",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### A simple class for regressions\n",
"\n",
"The next cell encapsulates a linear regression model in a Python class and defines a\n",
"function that fits the model using a stochastic optimizer."
]
},
{
"metadata": {
"id": "-90due2RVuDF",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"import time\n",
"\n",
"from google.colab import widgets\n",
"import tensorflow.contrib.eager as tfe # Needed to create tfe.Variable objects.\n",
"### Enabling eager execution lets you debug your code on-the-fly; use `pdb` and print statements to your heart's content.\n",
"\n",
"Check out exercise 2 towards the bottom of this notebook for a hands-on look at how eager simplifies model debugging."
]
},
{
"metadata": {
"id": "DNHJpCyNVwA9",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"import pdb\n",
"\n",
"def buggy_loss(y, y_hat):\n",
" pdb.set_trace()\n",
" huber_loss(y, y_hat)\n",
" \n",
"print(\"Type 'exit' to stop the debugger, or 's' to step into `huber_loss` and \"\n",
" \"'n' to step through it.\")\n",
"try:\n",
" buggy_loss(1.0, 2.0)\n",
"except:\n",
" pass"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "mvI3ljk-vJ_h",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Leverage the Python profiler to dig into the relative costs of training your model.\n",
"\n",
"If you run the below cell, you'll see that most of the time is spent computing gradients and binary operations, which is sensible considering our loss function."
"print(\"Most of the time is spent during backpropagation and binary operations.\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "5AeTwwPobkaJ",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 3. Neural networks\n",
"\n",
"While eager execution can certainly be used as a library for numerical computation, it shines as a library for deep learning: TensorFlow provides a suite of tools for deep learning research and development, most of which are compatible with eager execution. In this section, we put some of these tools to use to build _RNNColorbot_, an RNN that takes as input names of colors and predicts their corresponding RGB tuples. "
]
},
{
"metadata": {
"id": "6IcmEQ-jpTMO",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Constructing a data pipeline\n",
"\n",
"**[`tf.data`](https://www.tensorflow.org/api_guides/python/reading_data#_tf_data_API) is TensorFlow's canonical API for constructing input pipelines.** `tf.data` lets you easily construct multi-stage pipelines that supply data to your networks during training and inference. The following cells defines methods that download and format the data needed for RNNColorbot; the details aren't important (read them in the privacy of your own home if you so wish), but make sure to run the cells before proceeding."
]
},
{
"metadata": {
"id": "dcUC3Ma8bjgY",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"import os\n",
"import six\n",
"from six.moves import urllib\n",
"\n",
"\n",
"def parse(line):\n",
" \"\"\"Parse a line from the colors dataset.\"\"\"\n",
"TensorFlow packages several APIs for creating neural networks in a modular fashion. **The canonical way to define neural networks in TensorFlow is to encapsulate your model in a class that inherits from `tf.keras.Model`**. You should think of `tf.keras.Model` as a container of **[object-oriented layers](https://www.tensorflow.org/api_docs/python/tf/layers)**, TensorFlow's building blocks for constructing neural networks (*e.g.*, `tf.layers.Dense`, `tf.layers.Conv2D`). Every `Layer` object that is set as an attribute of a `Model` is automatically tracked by the latter, letting you access `Layer`-contained variables by invoking `Model`'s `.variables()` method. Most important, **inheriting from `tf.keras.Model` makes it easy to checkpoint your model and to subsequently restore it** --- more on that later. \n",
"\n",
"The following cell exemplifies our high-level neural network APIs. Note that `RNNColorbot` encapsulates only the model definition and prediction generation logic. The loss, training, and evaluation functions exist outside the class definition: conceptually, the model doesn't need know how to train and benchmark itself."
]
},
{
"metadata": {
"id": "NlKcdvT9leQ2",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"cellView": "code"
},
"cell_type": "code",
"source": [
"class RNNColorbot(tf.keras.Model):\n",
" \"\"\"Multi-layer RNN that predicts RGB tuples given color names.\n",
"The next cell **trains** our `RNNColorbot`, **restoring and saving checkpoints** of the learned variables along the way. Thanks to checkpointing, every run of the below cell will resume training from wherever the previous run left off. For more on checkpointing, take a look at our [user guide](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/g3doc/guide.md#checkpointing-trained-variables)."
"print(\"Colorbot is ready to generate colors!\")"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "1HdJk37R1xz9",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Paint me a color, Colorbot!\n",
"\n",
"We can interact with RNNColorbot in a natural way; no need to thread NumPy arrays into placeholders through feed dicts.\n",
"So go ahead and ask RNNColorbot to paint you some colors. If they're not to your liking, re-run the previous cell to resume training from where we left off, and then re-run the next one for updated results."
]
},
{
"metadata": {
"id": "LXAYjopasyWr",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"tb = widgets.TabBar([\"RNN Colorbot\"])\n",
"while True:\n",
" with tb.output_to(0):\n",
" try:\n",
" color_name = six.moves.input(\n",
" \"Give me a color name (or press 'enter' to exit): \")\n",
" clipped_preds = tuple(min(float(p), 1.0) for p in preds)\n",
" rgb = tuple(int(p * 255) for p in clipped_preds)\n",
" with tb.output_to(0):\n",
" tb.clear_tab()\n",
" print(\"Predicted RGB tuple:\", rgb)\n",
" plt.imshow([[clipped_preds]])\n",
" plt.title(color_name)\n",
" plt.show()"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "aJopbdYiXXQM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 4. Exercises"
]
},
{
"metadata": {
"id": "Nt2bZ3SNq0bl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Exercise 1: Batching\n",
"\n",
"Executing operations eagerly incurs small overheads; these overheads become neglible when amortized over batched operations. In this exercise, we explore the relationship between batching and performance by revisiting our Huber regression example."
]
},
{
"metadata": {
"id": "U5NR8vOY-4Xx",
"colab_type": "code",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
}
},
"cell_type": "code",
"source": [
"# Our original implementation of `huber_loss` is not compatible with non-scalar\n",
"# data. Your task is to fix that. For your convenience, the original\n",
"# implementation is reproduced below.\n",
"#\n",
"# def huber_loss(y, y_hat, m=1.0):\n",
"# delta = tf.abs(y - y_hat)\n",
"# return delta ** 2 if delta <= m else m * (2 * delta - m)\n",
"#\n",
"def batched_huber_loss(y, y_hat, m=1.0):\n",
" # TODO: Uncomment out the below code and replace `...` with your solution.\n",
"We've heard you loud and clear: TensorFlow programs that construct and execute graphs are difficult to debug. By design, enabling eager execution vastly simplifies the process of debugging TensorFlow programs. Once eager execution is enabled, you can step through your models using `pdb` and bisect them with `print` statements. The best way to understand the extent to which eager execution simplifies debugging is to debug a model yourself. `BuggyModel` below has two bugs lurking in it. Execute the following cell, read the error message, and go hunt some bugs!\n",
"\n",
"*Hint: As is often the case with TensorFlow programs, both bugs are related to the shapes of Tensors.*\n",
"\n",
"*Hint: You might find `tf.layers.flatten` useful.*"
"* our [collection of example models](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples), which includes a convolutional model for [MNIST](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/mnist) classification, a [GAN](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/gan), a [recursive neural network](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/spinn), and more;\n",
"* [this advanced notebook](https://colab.research.com/github/tensorflow/tensorflow/blob/master/tensorflow/contrib/autograph/examples/notebooks/dev_summit_2018_demo.ipynb), which explains how to build and execute graphs while eager execution is enabled and how to call into eager execution while constructing a graph, and which also introduces Autograph, a source-code translation tool that automatically generates graph-construction code from dynamic eager code.\n",
