Skip to content

GitLab

"examples/vscode:/vscode.git/clone" did not exist on "0452cc3c0c81b6033365a6112768c2a8a807062c"
Commit eaf31375 authored by Mark Daoust's avatar Mark Daoust
Browse files

Move buttons to the top, keep image aspect ratio, add plots during optimization, fix headings.

parent e14651bb
Hide whitespace changes
Inline Side-by-side
Showing with 165 additions and 97 deletions
research/nst_blogpost/4_Neural_Style_Transfer_with_Eager_Execution.ipynb
View file @ eaf31375
...@@ -3,11 +3,12 @@ ...@@ -3,11 +3,12 @@
"nbformat_minor": 0, "nbformat_minor": 0,
"metadata": { "metadata": {
"colab": { "colab": {
"name": "Copy of 4. Neural Style Transfer with Eager Execution", "name": "Neural Style Transfer with Eager Execution",
"version": "0.3.2", "version": "0.3.2",
"provenance": [], "provenance": [],
"private_outputs": true, "private_outputs": true,
"collapsed_sections": [] "collapsed_sections": [],
"toc_visible": true
}, },
"kernelspec": { "kernelspec": {
"name": "python3", "name": "python3",
...@@ -18,12 +19,32 @@ ...@@ -18,12 +19,32 @@
"cells": [ "cells": [
{ {
"metadata": { "metadata": {
"id": "aDyGj8DmXCJI", "id": "jo5PziEC4hWs",
"colab_type": "text" "colab_type": "text"
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Neural Style Transfer with tf.keras\n", "# Neural Style Transfer with tf.keras\n",
"\n",
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/models/blob/master/research/nst_blogpost/4_Neural_Style_Transfer_with_Eager_Execution.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/4_Neural_Style_Transfer_with_Eager_Execution.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
"</table>"
]
},
{
"metadata": {
"id": "aDyGj8DmXCJI",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Overview\n",
"\n",
"In this tutorial, we will learn how to use deep learning to compose images in the style of another image (ever wish you could paint like Picasso or Van Gogh?). This is known as **neural style transfer**! This is a technique outlined in [Leon A. Gatys' paper, A Neural Algorithm of Artistic Style](https://arxiv.org/abs/1508.06576), which is a great read, and you should definitely check it out. \n", "In this tutorial, we will learn how to use deep learning to compose images in the style of another image (ever wish you could paint like Picasso or Van Gogh?). This is known as **neural style transfer**! This is a technique outlined in [Leon A. Gatys' paper, A Neural Algorithm of Artistic Style](https://arxiv.org/abs/1508.06576), which is a great read, and you should definitely check it out. \n",
"\n", "\n",
"But, what is neural style transfer?\n", "But, what is neural style transfer?\n",
...@@ -35,22 +56,23 @@ ...@@ -35,22 +56,23 @@
"\n", "\n",
"<img src=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/Green_Sea_Turtle_grazing_seagrass.jpg?raw=1\" alt=\"Drawing\" style=\"width: 200px;\"/>\n", "<img src=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/Green_Sea_Turtle_grazing_seagrass.jpg?raw=1\" alt=\"Drawing\" style=\"width: 200px;\"/>\n",
"<img src=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/The_Great_Wave_off_Kanagawa.jpg?raw=1\" alt=\"Drawing\" style=\"width: 200px;\"/>\n", "<img src=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/The_Great_Wave_off_Kanagawa.jpg?raw=1\" alt=\"Drawing\" style=\"width: 200px;\"/>\n",
"Image of Green Sea Turtle: \n", "\n",
"https://commons.wikimedia.org/wiki/File:Green_Sea_Turtle_grazing_seagrass.jpg\n", "[Image of Green Sea Turtle](https://commons.wikimedia.org/wiki/File:Green_Sea_Turtle_grazing_seagrass.jpg)\n",
"-By P.Lindgren [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Common\n", "-By P.Lindgren [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Common\n",
"\n", "\n",
"\n", "\n",
"Now how would it look like if Hokusai decided to paint the picture of this Turtle exclusively with this style? Something like this?\n", "Now how would it look like if Hokusai decided to paint the picture of this Turtle exclusively with this style? Something like this?\n",
"\n", "\n",
"<img src=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/wave_turtle.png?raw=1\" alt=\"Drawing\" style=\"width: 500px;\"/>\n", "<img src=\"https://github.com/MarkDaoust/models/blob/nst_fixes/research/nst_blogpost/wave_turtle.png?raw=1\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
"\n", "\n",
"Is this magic or just deep learning? Fortunately, this doesn’t involve any witchcraft: style transfer is a fun and interesting technique that showcases the capabilities and internal representations of neural networks. \n", "Is this magic or just deep learning? Fortunately, this doesn’t involve any witchcraft: style transfer is a fun and interesting technique that showcases the capabilities and internal representations of neural networks. \n",
"\n", "\n",
"The principle of neural style transfer is to define two distance functions, one that describes how different the content of two images are , $L_{content}$, and one that describes the difference between two images in terms of their style, $L_{style}$. Then, given three images, a desired style image, a desired content image, and the input image (initialized with the content image), we try to transform the input image to minimize the content distance with the content image and its style distance with the style image. \n", "The principle of neural style transfer is to define two distance functions, one that describes how different the content of two images are , $L_{content}$, and one that describes the difference between two images in terms of their style, $L_{style}$. Then, given three images, a desired style image, a desired content image, and the input image (initialized with the content image), we try to transform the input image to minimize the content distance with the content image and its style distance with the style image. \n",
"In summary, we’ll take the base input image, a content image that we want to match, and the style image that we want to match. We’ll transform the base input image by minimizing the content and style distances (losses) with backpropagation, creating an image that matches the content of the content image and the style of the style image. \n", "In summary, we’ll take the base input image, a content image that we want to match, and the style image that we want to match. We’ll transform the base input image by minimizing the content and style distances (losses) with backpropagation, creating an image that matches the content of the content image and the style of the style image. \n",
"\n", "\n",
"## Specific concepts that will be covered:\n", "### Specific concepts that will be covered:\n",
"In the process, we will build practical experience and develop intuition around the following concepts\n", "In the process, we will build practical experience and develop intuition around the following concepts\n",
"\n",
"* **Eager Execution** - use TensorFlow's imperative programming environment that evaluates operations immediately \n", "* **Eager Execution** - use TensorFlow's imperative programming environment that evaluates operations immediately \n",
" * [Learn more about eager execution](https://www.tensorflow.org/programmers_guide/eager)\n", " * [Learn more about eager execution](https://www.tensorflow.org/programmers_guide/eager)\n",
" * [See it in action](https://www.tensorflow.org/get_started/eager)\n", " * [See it in action](https://www.tensorflow.org/get_started/eager)\n",
...@@ -59,6 +81,7 @@ ...@@ -59,6 +81,7 @@
"* **Create custom training loops** - we'll examine how to set up an optimizer to minimize a given loss with respect to input parameters\n", "* **Create custom training loops** - we'll examine how to set up an optimizer to minimize a given loss with respect to input parameters\n",
"\n", "\n",
"### We will follow the general steps to perform style transfer:\n", "### We will follow the general steps to perform style transfer:\n",
"\n",
"1. Visualize data\n", "1. Visualize data\n",
"2. Basic Preprocessing/preparing our data\n", "2. Basic Preprocessing/preparing our data\n",
"3. Set up loss functions \n", "3. Set up loss functions \n",
...@@ -74,34 +97,19 @@ ...@@ -74,34 +97,19 @@
}, },
{ {
"metadata": { "metadata": {
"id": "43RpeUuwNF1w", "id": "U8ajP_u73s6m",
"colab_type": "text" "colab_type": "text"
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n", "## Setup\n",
" <td>\n", "\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/models/blob/master/research/nst_blogpost/4_Neural_Style_Transfer_with_Eager_Execution.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n", "### Download Images"
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/models/blob/master/research/nst_blogpost/4_Neural_Style_Transfer_with_Eager_Execution.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
"</table>"
]
},
{
"metadata": {
"id": "ALy6RWBANF1x",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Let's get all of our images that we want"
] ]
}, },
{ {
"metadata": { "metadata": {
"id": "JnUvVHTkNF1x", "id": "riWE_b8k3s6o",
"colab_type": "code", "colab_type": "code",
"colab": {} "colab": {}
}, },
...@@ -111,16 +119,26 @@ ...@@ -111,16 +119,26 @@
"img_dir = '/tmp/nst'\n", "img_dir = '/tmp/nst'\n",
"if not os.path.exists(img_dir):\n", "if not os.path.exists(img_dir):\n",
" os.makedirs(img_dir)\n", " os.makedirs(img_dir)\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/d/d7/Green_Sea_Turtle_grazing_seagrass.jpg\n", "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/d/d7/Green_Sea_Turtle_grazing_seagrass.jpg\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/0/0a/The_Great_Wave_off_Kanagawa.jpg\n", "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/0/0a/The_Great_Wave_off_Kanagawa.jpg\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/b/b4/Vassily_Kandinsky%2C_1913_-_Composition_7.jpg\n", "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/b/b4/Vassily_Kandinsky%2C_1913_-_Composition_7.jpg\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/0/00/Tuebingen_Neckarfront.jpg\n", "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/0/00/Tuebingen_Neckarfront.jpg\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/6/68/Pillars_of_creation_2014_HST_WFC3-UVIS_full-res_denoised.jpg\n", "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/6/68/Pillars_of_creation_2014_HST_WFC3-UVIS_full-res_denoised.jpg\n",
"!wget -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg/1024px-Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg" "!wget --quiet -P /tmp/nst/ https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg/1024px-Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg"
], ],
"execution_count": 0, "execution_count": 0,
"outputs": [] "outputs": []
}, },
{
"metadata": {
"id": "eqxUicSPUOP6",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Import and configure modules"
]
},
{ {
"metadata": { "metadata": {
"id": "sc1OLbOWhPCO", "id": "sc1OLbOWhPCO",
...@@ -130,6 +148,10 @@ ...@@ -130,6 +148,10 @@
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"import matplotlib.pyplot as plt\n", "import matplotlib.pyplot as plt\n",
"import matplotlib as mpl\n",
"mpl.rcParams['figure.figsize'] = (10,10)\n",
"mpl.rcParams['axes.grid'] = False\n",
"\n",
"import numpy as np\n", "import numpy as np\n",
"from PIL import Image\n", "from PIL import Image\n",
"import time\n", "import time\n",
...@@ -140,7 +162,7 @@ ...@@ -140,7 +162,7 @@
}, },
{ {
"metadata": { "metadata": {
"id": "FzroKP5SNF15", "id": "RYEjlrYk3s6w",
"colab_type": "code", "colab_type": "code",
"colab": {} "colab": {}
}, },
...@@ -149,7 +171,7 @@ ...@@ -149,7 +171,7 @@
"import tensorflow as tf\n", "import tensorflow as tf\n",
"import tensorflow.contrib.eager as tfe\n", "import tensorflow.contrib.eager as tfe\n",
"\n", "\n",
"from tensorflow.python.keras.preprocessing import image\n", "from tensorflow.python.keras.preprocessing import image as kp_image\n",
"from tensorflow.python.keras import models \n", "from tensorflow.python.keras import models \n",
"from tensorflow.python.keras import losses\n", "from tensorflow.python.keras import losses\n",
"from tensorflow.python.keras import layers\n", "from tensorflow.python.keras import layers\n",
...@@ -191,7 +213,6 @@ ...@@ -191,7 +213,6 @@
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"# Set up some global values here\n", "# Set up some global values here\n",
"img_shape = (512, 512, 3)\n",
"content_path = '/tmp/nst/Green_Sea_Turtle_grazing_seagrass.jpg'\n", "content_path = '/tmp/nst/Green_Sea_Turtle_grazing_seagrass.jpg'\n",
"style_path = '/tmp/nst/The_Great_Wave_off_Kanagawa.jpg'" "style_path = '/tmp/nst/The_Great_Wave_off_Kanagawa.jpg'"
], ],
...@@ -205,7 +226,7 @@ ...@@ -205,7 +226,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Let's begin by visualizing our input" "## Visualize the input"
] ]
}, },
{ {
...@@ -217,8 +238,14 @@ ...@@ -217,8 +238,14 @@
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"def load_img(path_to_img):\n", "def load_img(path_to_img):\n",
" img = image.load_img(path_to_img, target_size=(img_shape[0], img_shape[1]))\n", " max_dim = 512\n",
" img = image.img_to_array(img)\n", " img = Image.open(path_to_img)\n",
" long = max(img.size)\n",
" scale = max_dim/long\n",
" img = img.resize((round(img.size[0]*scale), round(img.size[1]*scale)), Image.ANTIALIAS)\n",
" \n",
" img = kp_image.img_to_array(img)\n",
" \n",
" # We need to broadcast the image array such that it has a batch dimension \n", " # We need to broadcast the image array such that it has a batch dimension \n",
" img = np.expand_dims(img, axis=0)\n", " img = np.expand_dims(img, axis=0)\n",
" return img" " return img"
...@@ -268,7 +295,7 @@ ...@@ -268,7 +295,7 @@
"plt.figure(figsize=(10,10))\n", "plt.figure(figsize=(10,10))\n",
"\n", "\n",
"content = load_img(content_path).astype('uint8')\n", "content = load_img(content_path).astype('uint8')\n",
"style = load_img(style_path)\n", "style = load_img(style_path).astype('uint8')\n",
"\n", "\n",
"plt.subplot(1, 2, 1)\n", "plt.subplot(1, 2, 1)\n",
"imshow(content, 'Content Image')\n", "imshow(content, 'Content Image')\n",
...@@ -287,7 +314,7 @@ ...@@ -287,7 +314,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Prepare the data\n", "## Prepare the data\n",
"Let's create methods that will allow us to load and preprocess our images easily. We perform the same preprocessing process as are expected according to the VGG training process. VGG networks are trained on image with each channel normalized by `mean = [103.939, 116.779, 123.68]`and with channels BGR." "Let's create methods that will allow us to load and preprocess our images easily. We perform the same preprocessing process as are expected according to the VGG training process. VGG networks are trained on image with each channel normalized by `mean = [103.939, 116.779, 123.68]`and with channels BGR."
] ]
}, },
...@@ -328,9 +355,9 @@ ...@@ -328,9 +355,9 @@
"def deprocess_img(processed_img):\n", "def deprocess_img(processed_img):\n",
" x = processed_img.copy()\n", " x = processed_img.copy()\n",
" if len(x.shape) == 4:\n", " if len(x.shape) == 4:\n",
" x = x.reshape(img_shape)\n", " x = np.squeeze(x, 0)\n",
" assert len(x.shape) == 3, (\"Input to deprocess image must be an image of \"\n", " assert len(x.shape) == 3, (\"Input to deprocess image must be an image of \"\n",
" \"dimension [batch, height, width, channel] or [height_width_channel]\")\n", " \"dimension [1, height, width, channel] or [height, width, channel]\")\n",
" if len(x.shape) != 3:\n", " if len(x.shape) != 3:\n",
" raise ValueError(\"Invalid input to deprocessing image\")\n", " raise ValueError(\"Invalid input to deprocessing image\")\n",
" \n", " \n",
...@@ -353,10 +380,10 @@ ...@@ -353,10 +380,10 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Define content and style representationst\n", "### Define content and style representationst\n",
"In order to get both the content and style representations of our image, we will look at some intermediate layers within our model. As we go deeper into the model, these intermediate layers represent higher and higher order features. In this case, we are using the network architecture VGG19, a pretrained image classification network. These intermediate layers are necessary to define the representation of content and style from our images. For an input image, we will try to match the corresponding style and content target representations at these intermediate layers. \n", "In order to get both the content and style representations of our image, we will look at some intermediate layers within our model. As we go deeper into the model, these intermediate layers represent higher and higher order features. In this case, we are using the network architecture VGG19, a pretrained image classification network. These intermediate layers are necessary to define the representation of content and style from our images. For an input image, we will try to match the corresponding style and content target representations at these intermediate layers. \n",
"\n", "\n",
"### Why intermediate layers?\n", "#### Why intermediate layers?\n",
"\n", "\n",
"You may be wondering why these intermediate outputs within our pretrained image classification network allow us to define style and content representations. At a high level, this phenomenon can be explained by the fact that in order for a network to perform image classification (which our network has been trained to do), it must understand the image. This involves taking the raw image as input pixels and building an internal representation through transformations that turn the raw image pixels into a complex understanding of the features present within the image. This is also partly why convolutional neural networks are able to generalize well: they’re able to capture the invariances and defining features within classes (e.g., cats vs. dogs) that are agnostic to background noise and other nuisances. Thus, somewhere between where the raw image is fed in and the classification label is output, the model serves as a complex feature extractor; hence by accessing intermediate layers, we’re able to describe the content and style of input images. \n", "You may be wondering why these intermediate outputs within our pretrained image classification network allow us to define style and content representations. At a high level, this phenomenon can be explained by the fact that in order for a network to perform image classification (which our network has been trained to do), it must understand the image. This involves taking the raw image as input pixels and building an internal representation through transformations that turn the raw image pixels into a complex understanding of the features present within the image. This is also partly why convolutional neural networks are able to generalize well: they’re able to capture the invariances and defining features within classes (e.g., cats vs. dogs) that are agnostic to background noise and other nuisances. Thus, somewhere between where the raw image is fed in and the classification label is output, the model serves as a complex feature extractor; hence by accessing intermediate layers, we’re able to describe the content and style of input images. \n",
"\n", "\n",
...@@ -396,7 +423,7 @@ ...@@ -396,7 +423,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Model \n", "## Build the Model \n",
"In this case, we load [VGG19](https://keras.io/applications/#vgg19), and feed in our input tensor to the model. This will allow us to extract the feature maps (and subsequently the content and style representations) of the content, style, and generated images.\n", "In this case, we load [VGG19](https://keras.io/applications/#vgg19), and feed in our input tensor to the model. This will allow us to extract the feature maps (and subsequently the content and style representations) of the content, style, and generated images.\n",
"\n", "\n",
"We use VGG19, as suggested in the paper. In addition, since VGG19 is a relatively simple model (compared with ResNet, Inception, etc) the feature maps actually work better for style transfer. " "We use VGG19, as suggested in the paper. In addition, since VGG19 is a relatively simple model (compared with ResNet, Inception, etc) the feature maps actually work better for style transfer. "
...@@ -465,7 +492,7 @@ ...@@ -465,7 +492,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Define and create our loss functions (content and style distances)" "## Define and create our loss functions (content and style distances)"
] ]
}, },
{ {
...@@ -475,7 +502,7 @@ ...@@ -475,7 +502,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Content Loss" "### Content Loss"
] ]
}, },
{ {
...@@ -502,7 +529,7 @@ ...@@ -502,7 +529,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Computing content loss\n", "### Computing content loss\n",
"We will actually add our content losses at each desired layer. This way, each iteration when we feed our input image through the model (which in eager is simply `model(input_image)`!) all the content losses through the model will be properly compute and because we are executing eagerly, all the gradients will be computed. " "We will actually add our content losses at each desired layer. This way, each iteration when we feed our input image through the model (which in eager is simply `model(input_image)`!) all the content losses through the model will be properly compute and because we are executing eagerly, all the gradients will be computed. "
] ]
}, },
...@@ -527,7 +554,7 @@ ...@@ -527,7 +554,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Style Loss" "## Style Loss"
] ]
}, },
{ {
...@@ -556,7 +583,7 @@ ...@@ -556,7 +583,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Computing style loss\n", "### Computing style loss\n",
"Again, we implement our loss as a distance metric . " "Again, we implement our loss as a distance metric . "
] ]
}, },
...@@ -595,7 +622,7 @@ ...@@ -595,7 +622,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Apply style transfer to our images\n" "## Apply style transfer to our images\n"
] ]
}, },
{ {
...@@ -605,7 +632,7 @@ ...@@ -605,7 +632,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Run Gradient Descent \n", "### Run Gradient Descent \n",
"If you aren't familiar with gradient descent/backpropagation or need a refresher, you should definitely check out this [awesome resource](https://developers.google.com/machine-learning/crash-course/reducing-loss/gradient-descent).\n", "If you aren't familiar with gradient descent/backpropagation or need a refresher, you should definitely check out this [awesome resource](https://developers.google.com/machine-learning/crash-course/reducing-loss/gradient-descent).\n",
"\n", "\n",
"In this case, we use the [Adam](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam)* optimizer in order to minimize our loss. We iteratively update our output image such that it minimizes our loss: we don't update the weights associated with our network, but instead we train our input image to minimize loss. In order to do this, we must know how we calculate our loss and gradients. \n", "In this case, we use the [Adam](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam)* optimizer in order to minimize our loss. We iteratively update our output image such that it minimizes our loss: we don't update the weights associated with our network, but instead we train our input image to minimize loss. In order to do this, we must know how we calculate our loss and gradients. \n",
...@@ -651,12 +678,13 @@ ...@@ -651,12 +678,13 @@
" style_image = load_and_process_img(style_path)\n", " style_image = load_and_process_img(style_path)\n",
" \n", " \n",
" # batch compute content and style features\n", " # batch compute content and style features\n",
" stack_images = np.concatenate([style_image, content_image], axis=0)\n", " style_outputs = model(style_image)\n",
" model_outputs = model(stack_images)\n", " content_outputs = model(content_image)\n",
" \n",
" \n", " \n",
" # Get the style and content feature representations from our model \n", " # Get the style and content feature representations from our model \n",
" style_features = [style_layer[0] for style_layer in model_outputs[:num_style_layers]]\n", " style_features = [style_layer[0] for style_layer in style_outputs[:num_style_layers]]\n",
" content_features = [content_layer[1] for content_layer in model_outputs[num_style_layers:]]\n", " content_features = [content_layer[0] for content_layer in content_outputs[num_style_layers:]]\n",
" return style_features, content_features" " return style_features, content_features"
], ],
"execution_count": 0, "execution_count": 0,
...@@ -669,7 +697,7 @@ ...@@ -669,7 +697,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Computing the loss and gradients\n", "### Computing the loss and gradients\n",
"Here we use [**tf.GradientTape**](https://www.tensorflow.org/programmers_guide/eager#computing_gradients) to compute the gradient. It allows us to take advantage of the automatic differentiation available by tracing operations for computing the gradient later. It records the operations during the forward pass and then is able to compute the gradient of our loss function with respect to our input image for the backwards pass." "Here we use [**tf.GradientTape**](https://www.tensorflow.org/programmers_guide/eager#computing_gradients) to compute the gradient. It allows us to take advantage of the automatic differentiation available by tracing operations for computing the gradient later. It records the operations during the forward pass and then is able to compute the gradient of our loss function with respect to our input image for the backwards pass."
] ]
}, },
...@@ -768,7 +796,7 @@ ...@@ -768,7 +796,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Apply and run the style transfer process" "### Optimization loop"
] ]
}, },
{ {
...@@ -779,12 +807,13 @@ ...@@ -779,12 +807,13 @@
}, },
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"import IPython.display\n",
"\n",
"def run_style_transfer(content_path, \n", "def run_style_transfer(content_path, \n",
" style_path,\n", " style_path,\n",
" num_iterations=1000,\n", " num_iterations=1000,\n",
" content_weight=1e3, \n", " content_weight=1e3, \n",
" style_weight=1e-2): \n", " style_weight=1e-2): \n",
" display_num = 100\n",
" # We don't need to (or want to) train any layers of our model, so we set their\n", " # We don't need to (or want to) train any layers of our model, so we set their\n",
" # trainable to false. \n", " # trainable to false. \n",
" model = get_model() \n", " model = get_model() \n",
...@@ -818,18 +847,20 @@ ...@@ -818,18 +847,20 @@
" }\n", " }\n",
" \n", " \n",
" # For displaying\n", " # For displaying\n",
" plt.figure(figsize=(14, 7))\n", " num_rows = 2\n",
" num_rows = (num_iterations / display_num) // 5\n", " num_cols = 5\n",
" display_interval = num_iterations/(num_rows*num_cols)\n",
" start_time = time.time()\n", " start_time = time.time()\n",
" global_start = time.time()\n", " global_start = time.time()\n",
" \n", " \n",
" norm_means = np.array([103.939, 116.779, 123.68])\n", " norm_means = np.array([103.939, 116.779, 123.68])\n",
" min_vals = -norm_means\n", " min_vals = -norm_means\n",
" max_vals = 255 - norm_means \n", " max_vals = 255 - norm_means \n",
" \n",
" imgs = []\n",
" for i in range(num_iterations):\n", " for i in range(num_iterations):\n",
" grads, all_loss = compute_grads(cfg)\n", " grads, all_loss = compute_grads(cfg)\n",
" loss, style_score, content_score = all_loss\n", " loss, style_score, content_score = all_loss\n",
" # grads, _ = tf.clip_by_global_norm(grads, 5.0)\n",
" opt.apply_gradients([(grads, init_image)])\n", " opt.apply_gradients([(grads, init_image)])\n",
" clipped = tf.clip_by_value(init_image, min_vals, max_vals)\n", " clipped = tf.clip_by_value(init_image, min_vals, max_vals)\n",
" init_image.assign(clipped)\n", " init_image.assign(clipped)\n",
...@@ -838,26 +869,30 @@ ...@@ -838,26 +869,30 @@
" if loss < best_loss:\n", " if loss < best_loss:\n",
" # Update best loss and best image from total loss. \n", " # Update best loss and best image from total loss. \n",
" best_loss = loss\n", " best_loss = loss\n",
" best_img = init_image.numpy()\n", " best_img = deprocess_img(init_image.numpy())\n",
"\n", "\n",
" if i % display_num == 0:\n", " if i % display_interval== 0:\n",
" print('Iteration: {}'.format(i)) \n",
" print('Total loss: {:.4e}, ' \n",
" 'style loss: {:.4e}, '\n",
" 'content loss: {:.4e}, '\n",
" 'time: {:.4f}s'.format(loss, style_score, content_score, time.time() - start_time))\n",
" start_time = time.time()\n", " start_time = time.time()\n",
" \n", " \n",
" # Display intermediate images\n",
" if iter_count > num_rows * 5: continue \n",
" plt.subplot(num_rows, 5, iter_count)\n",
" # Use the .numpy() method to get the concrete numpy array\n", " # Use the .numpy() method to get the concrete numpy array\n",
" plot_img = init_image.numpy()\n", " plot_img = init_image.numpy()\n",
" plot_img = deprocess_img(plot_img)\n", " plot_img = deprocess_img(plot_img)\n",
" plt.imshow(plot_img)\n", " imgs.append(plot_img)\n",
" plt.title('Iteration {}'.format(i + 1))\n", " IPython.display.clear_output(wait=True)\n",
" iter_count += 1\n", " IPython.display.display_png(Image.fromarray(plot_img))\n",
" print('Iteration: {}'.format(i)) \n",
" print('Total loss: {:.4e}, ' \n",
" 'style loss: {:.4e}, '\n",
" 'content loss: {:.4e}, '\n",
" 'time: {:.4f}s'.format(loss, style_score, content_score, time.time() - start_time))\n",
" print('Total time: {:.4f}s'.format(time.time() - global_start))\n", " print('Total time: {:.4f}s'.format(time.time() - global_start))\n",
" IPython.display.clear_output(wait=True)\n",
" plt.figure(figsize=(14,4))\n",
" for i,img in enumerate(imgs):\n",
" plt.subplot(num_rows,num_cols,i+1)\n",
" plt.imshow(img)\n",
" plt.xticks([])\n",
" plt.yticks([])\n",
" \n", " \n",
" return best_img, best_loss " " return best_img, best_loss "
], ],
...@@ -873,7 +908,44 @@ ...@@ -873,7 +908,44 @@
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"best, best_loss = run_style_transfer(content_path, \n", "best, best_loss = run_style_transfer(content_path, \n",
" style_path)" " style_path, num_iterations=1000)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "dzJTObpsO3TZ",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"Image.fromarray(best)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "dCXQ9vSnQbDy",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"To download the image from Colab uncomment the following code:"
]
},
{
"metadata": {
"id": "SSH6OpyyQn7w",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"#from google.colab import files\n",
"#files.download('wave_turtle.png')"
], ],
"execution_count": 0, "execution_count": 0,
"outputs": [] "outputs": []
...@@ -885,7 +957,7 @@ ...@@ -885,7 +957,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Visualize outputs\n", "## Visualize outputs\n",
"We \"deprocess\" the output image in order to remove the processing that was applied to it. " "We \"deprocess\" the output image in order to remove the processing that was applied to it. "
] ]
}, },
...@@ -898,26 +970,20 @@ ...@@ -898,26 +970,20 @@
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"def show_results(best_img, content_path, style_path, show_large_final=True):\n", "def show_results(best_img, content_path, style_path, show_large_final=True):\n",
" plt.figure(figsize=(15, 15))\n", " plt.figure(figsize=(10, 5))\n",
" x = deprocess_img(best_img)\n",
" content = load_img(content_path) \n", " content = load_img(content_path) \n",
" style = load_img(style_path)\n", " style = load_img(style_path)\n",
"\n", "\n",
" plt.subplot(1, 3, 1)\n", " plt.subplot(1, 2, 1)\n",
" imshow(content, 'Content Image')\n", " imshow(content, 'Content Image')\n",
"\n", "\n",
" plt.subplot(1, 3, 2)\n", " plt.subplot(1, 2, 2)\n",
" imshow(style, 'Style Image')\n", " imshow(style, 'Style Image')\n",
"\n", "\n",
" plt.subplot(1, 3, 3)\n",
" plt.imshow(x)\n",
" plt.title('Output Image')\n",
" plt.show()\n",
"\n",
" if show_large_final: \n", " if show_large_final: \n",
" plt.figure(figsize=(10, 10))\n", " plt.figure(figsize=(10, 10))\n",
"\n", "\n",
" plt.imshow(x)\n", " plt.imshow(best_img)\n",
" plt.title('Output Image')\n", " plt.title('Output Image')\n",
" plt.show()" " plt.show()"
], ],
...@@ -944,7 +1010,7 @@ ...@@ -944,7 +1010,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Let's see some other applications!\n", "## Try it on other images\n",
"Image of Tuebingen \n", "Image of Tuebingen \n",
"\n", "\n",
"Photo By: Andreas Praefcke [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY 3.0 (https://creativecommons.org/licenses/by/3.0)], from Wikimedia Commons" "Photo By: Andreas Praefcke [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY 3.0 (https://creativecommons.org/licenses/by/3.0)], from Wikimedia Commons"
...@@ -957,7 +1023,7 @@ ...@@ -957,7 +1023,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Starry night + Tuebingen" "### Starry night + Tuebingen"
] ]
}, },
{ {
...@@ -995,7 +1061,7 @@ ...@@ -995,7 +1061,7 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Pillars of Creation + Tuebingen" "### Pillars of Creation + Tuebingen"
] ]
}, },
{ {
...@@ -1029,12 +1095,12 @@ ...@@ -1029,12 +1095,12 @@
}, },
{ {
"metadata": { "metadata": {
"id": "Eteg3glPNF3O", "id": "bTZdTOdW3s8H",
"colab_type": "text" "colab_type": "text"
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Kandinsky Composition 7 + Tuebingen" "### Kandinsky Composition 7 + Tuebingen"
] ]
}, },
{ {
...@@ -1068,12 +1134,12 @@ ...@@ -1068,12 +1134,12 @@
}, },
{ {
"metadata": { "metadata": {
"id": "ACA8-74rNF3U", "id": "cg68lW2A3s8N",
"colab_type": "text" "colab_type": "text"
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"## Pillars of Creation + Sea Turtle" "### Pillars of Creation + Sea Turtle"
] ]
}, },
{ {
...@@ -1112,8 +1178,10 @@ ...@@ -1112,8 +1178,10 @@
}, },
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Key Takeaways\n", "## Key Takeaways\n",
"## What we covered:\n", "\n",
"### What we covered:\n",
"\n",
"* We built several different loss functions and used backpropagation to transform our input image in order to minimize these losses\n", "* We built several different loss functions and used backpropagation to transform our input image in order to minimize these losses\n",
" * In order to do this we had to load in an a **pretrained model** and used its learned feature maps to describe the content and style representation of our images.\n", " * In order to do this we had to load in an a **pretrained model** and used its learned feature maps to describe the content and style representation of our images.\n",
" * Our main loss functions were primarily computing the distance in terms of these different representations\n", " * Our main loss functions were primarily computing the distance in terms of these different representations\n",
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment