"In this tutorial we will learn how to segment images. **Segmentation** is the process of generating pixel-wise segmentations giving the class of the object visible at each pixel. For example, we could be identifying the location and boundaries of people within an image or identifying cell nuclei from an image. Formally, image segmentation refers to the process of partitioning an image into a set of pixels that we desire to identify (our target) and the background. \n",
"\n",
...
...
@@ -79,25 +63,27 @@
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "bJcQiA3OdCY6",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "bJcQiA3OdCY6"
},
"cell_type": "code",
"outputs": [],
"source": [
"!pip install kaggle"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "ODNLPGHKKgr-",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "ODNLPGHKKgr-"
},
"cell_type": "code",
"outputs": [],
"source": [
"import os\n",
"import glob\n",
...
...
@@ -114,17 +100,17 @@
"import matplotlib.image as mpimg\n",
"import pandas as pd\n",
"from PIL import Image\n"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "YQ9VRReUQxXi",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "YQ9VRReUQxXi"
},
"cell_type": "code",
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import tensorflow.contrib as tfcontrib\n",
...
...
@@ -132,28 +118,28 @@
"from tensorflow.python.keras import losses\n",
"from tensorflow.python.keras import models\n",
"from tensorflow.python.keras import backend as K "
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RW9gk331S0KA",
"colab_type": "text"
"colab_type": "text",
"id": "RW9gk331S0KA"
},
"cell_type": "markdown",
"source": [
"# Get all the files \n",
"Since this tutorial will be using a dataset from Kaggle, it requires [creating an API Token](https://github.com/Kaggle/kaggle-api#api-credentials) for your Kaggle acccount, and uploading it. "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "sAVM1ZTmdAMR",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "sAVM1ZTmdAMR"
},
"cell_type": "code",
"outputs": [],
"source": [
"import os\n",
"\n",
...
...
@@ -182,64 +168,64 @@
" os.chmod(token_file, 600)\n",
"\n",
"get_kaggle_credentials()\n"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gh6jkMp8dN5B",
"colab_type": "text"
"colab_type": "text",
"id": "gh6jkMp8dN5B"
},
"cell_type": "markdown",
"source": [
"Only import kaggle after adding the credentials."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "EoWJ1hb9dOV_",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "EoWJ1hb9dOV_"
},
"cell_type": "code",
"outputs": [],
"source": [
"import kaggle"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wC-byMdadAMT",
"colab_type": "text"
"colab_type": "text",
"id": "wC-byMdadAMT"
},
"cell_type": "markdown",
"source": [
"### We'll download the data from Kaggle\n",
"Caution, large download ahead - downloading all files will require 14GB of diskspace. "
"print(\"Number of training examples: {}\".format(num_train_examples))\n",
"print(\"Number of validation examples: {}\".format(num_val_examples))"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Nhda5fkPS3JD",
"colab_type": "text"
"colab_type": "text",
"id": "Nhda5fkPS3JD"
},
"cell_type": "markdown",
"source": [
"### Here's what the paths look like "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "Di1N83ArilzR",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "Di1N83ArilzR"
},
"cell_type": "code",
"outputs": [],
"source": [
"x_train_filenames[:10]"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "Gc-BDv1Zio1z",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "Gc-BDv1Zio1z"
},
"cell_type": "code",
"outputs": [],
"source": [
"y_train_filenames[:10]"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mhvDoZkbcUa1",
"colab_type": "text"
"colab_type": "text",
"id": "mhvDoZkbcUa1"
},
"cell_type": "markdown",
"source": [
"# Visualize\n",
"Let's take a look at some of the examples of different images in our dataset. "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "qUA6SDLhozjj",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "qUA6SDLhozjj"
},
"cell_type": "code",
"outputs": [],
"source": [
"display_num = 5\n",
"\n",
...
...
@@ -434,51 +420,49 @@
" \n",
"plt.suptitle(\"Examples of Images and their Masks\")\n",
"plt.show()"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d4CPgvPiToB_",
"colab_type": "text"
"colab_type": "text",
"id": "d4CPgvPiToB_"
},
"cell_type": "markdown",
"source": [
"# Set up "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HfeMRgyoa2n6",
"colab_type": "text"
"colab_type": "text",
"id": "HfeMRgyoa2n6"
},
"cell_type": "markdown",
"source": [
"Let’s begin by setting up some parameters. We’ll standardize and resize all the shapes of the images. We’ll also set up some training parameters: "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "oeDoiSFlothe",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "oeDoiSFlothe"
},
"cell_type": "code",
"outputs": [],
"source": [
"img_shape = (256, 256, 3)\n",
"batch_size = 3\n",
"epochs = 5"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8_d5ATP21npW",
"colab_type": "text"
"colab_type": "text",
"id": "8_d5ATP21npW"
},
"cell_type": "markdown",
"source": [
"Using these exact same parameters may be too computationally intensive for your hardware, so tweak the parameters accordingly. Also, it is important to note that due to the architecture of our UNet version, the size of the image must be evenly divisible by a factor of 32, as we down sample the spatial resolution by a factor of 2 with each `MaxPooling2Dlayer`.\n",
"\n",
...
...
@@ -490,11 +474,11 @@
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_HONB9JbXxDM",
"colab_type": "text"
"colab_type": "text",
"id": "_HONB9JbXxDM"
},
"cell_type": "markdown",
"source": [
"# Build our input pipeline with `tf.data`\n",
"Since we begin with filenames, we will need to build a robust and scalable data pipeline that will play nicely with our model. If you are unfamiliar with **tf.data** you should check out my other tutorial introducing the concept! \n",
...
...
@@ -516,33 +500,35 @@
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EtRA8vILbx2_",
"colab_type": "text"
"colab_type": "text",
"id": "EtRA8vILbx2_"
},
"cell_type": "markdown",
"source": [
"#### Why do we do these image transformations?\n",
"This is known as **data augmentation**. Data augmentation \"increases\" the amount of training data by augmenting them via a number of random transformations. During training time, our model would never see twice the exact same picture. This helps prevent [overfitting](https://developers.google.com/machine-learning/glossary/#overfitting) and helps the model generalize better to unseen data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3aGi28u8Cq9M",
"colab_type": "text"
"colab_type": "text",
"id": "3aGi28u8Cq9M"
},
"cell_type": "markdown",
"source": [
"## Processing each pathname"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "Fb_psznAggwr",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "Fb_psznAggwr"
},
"cell_type": "code",
"outputs": [],
"source": [
"def _process_pathnames(fname, label_path):\n",
" # We map this function onto each pathname pair \n",
"We'll build the U-Net model. U-Net is especially good with segmentation tasks because it can localize well to provide high resolution segmentation masks. In addition, it works well with small datasets and is relatively robust against overfitting as the training data is in terms of the number of patches within an image, which is much larger than the number of training images itself. Unlike the original model, we will add batch normalization to each of our blocks. \n",
"Defining loss and metric functions are simple with Keras. Simply define a function that takes both the True labels for a given example and the Predicted labels for the same given example. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sfuBVut0fogM",
"colab_type": "text"
"colab_type": "text",
"id": "sfuBVut0fogM"
},
"cell_type": "markdown",
"source": [
"Dice loss is a metric that measures overlap. More info on optimizing for Dice coefficient (our dice loss) can be found in the [paper](http://campar.in.tum.de/pub/milletari2016Vnet/milletari2016Vnet.pdf), where it was introduced. \n",
"Here, we'll use a specialized loss function that combines binary cross entropy and our dice loss. This is based on [individuals who competed within this competition obtaining better results empirically](https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/40199). "
"Here, we'll use a specialized loss function that combines binary cross entropy and our dice loss. This is based on [individuals who competed within this competition obtaining better results empirically](https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/40199). Try out your own custom losses to measure performance (e.g. bce + log(dice_loss), only bce, etc.)!"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"id": "udrfi9JGB-bL",
"colab": {},
"colab_type": "code",
"colab": {}
"id": "udrfi9JGB-bL"
},
"cell_type": "code",
"outputs": [],
"source": [
"def bce_dice_loss(y_true, y_pred):\n",
" loss = losses.binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n",
" return loss"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LifmpjXNc9Gz",
"colab_type": "text"
"colab_type": "text",
"id": "LifmpjXNc9Gz"
},
"cell_type": "markdown",
"source": [
"## Compile your model\n",
"We use our custom loss function to minimize. In addition, we specify what metrics we want to keep track of as we train. Note that metrics are not actually used during the training process to tune the parameters, but are instead used to measure performance of the training process. "
"Training your model with `tf.data` involves simply providing the model's `fit` function with your training/validation dataset, the number of steps, and epochs. \n",
"plt.suptitle(\"Examples of Input Image, Label, and Prediction\")\n",
"plt.show()"
],
"execution_count": 0,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iPV7RMA9TjPC",
"colab_type": "text"
"colab_type": "text",
"id": "iPV7RMA9TjPC"
},
"cell_type": "markdown",
"source": [
"# Key Takeaways\n",
"In this tutorial we learned how to train a network to automatically detect and create cutouts of cars from images! \n",
...
...
@@ -1292,5 +1249,34 @@
"* **Save and load our model** - We saved our best model that we encountered according to our specified metric. When we wanted to perform inference with out best model, we loaded it from disk. Note that saving the model capture more than just the weights of the model: by default, it saves the model architecture, weights, as well as information about the training process such as the state of the optimizer, etc. "
