{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fast Frame Interpolation with FLAVR.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Fast Frame Interpolation with FLAVR\n",
        "FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction. It uses a customized encoder decoder architecture with spatio-temporal convolutions and channel gating to capture and interpolate complex motion trajectories between frames to generate realistic high frame rate videos. This notebook is to apply slow-motion filtering on your own videos.  \n",
        "A GPU runtime is suggested to execute the code in this notebook.  \n",
        "  \n",
        "Credits for the original FLAVR work:\n",
        "\n",
        "\n",
        "```\n",
        "@article{kalluri2021flavr,\n",
        "  title={FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation},\n",
        "  author={Kalluri, Tarun and Pathak, Deepak and Chandraker, Manmohan and Tran, Du},\n",
        "  booktitle={arxiv},\n",
        "  year={2021}\n",
        "}\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "GtNm2bt5m__t"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Settings"
      ],
      "metadata": {
        "id": "Cer3xI_vC8AX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Clone the official GitHub repository."
      ],
      "metadata": {
        "id": "L25AZqD1aqYy"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5geYqIv5ah8_"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/tarun005/FLAVR.git\n",
        "%cd FLAVR"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install the missing requirements. Almost all the required Python packages for the code in this notebook are available by default in a Colab runtime. Only *PyAV*, a Pythonic binding for the FFmpeg libraries, to be installed really."
      ],
      "metadata": {
        "id": "VZ69AA375uby"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install av"
      ],
      "metadata": {
        "id": "L1Bd6U5H5x8X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download a pretrained model. The Colab GPU runtime specs allow full completion only for 2X interpolation."
      ],
      "metadata": {
        "id": "3idcRJmwa0ss"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown --id 1XFk9YZP9llTeporF-t_Au1dI-VhDQppG"
      ],
      "metadata": {
        "id": "eAjOsOhCbCXB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the code cell above doesn't work, please copy the pre-trained model manually to your Google Drive space and then follow the instructions for the next 3 code cells."
      ],
      "metadata": {
        "id": "nspERdHKiilc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mount your Google Drive. After executing the code in the cell below, a URL will be shown in the cell output. Click on it and follow the instructions that would appear online."
      ],
      "metadata": {
        "id": "t1W1cafV0RRB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/gdrive')"
      ],
      "metadata": {
        "id": "DPiPftbD0SWC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Copy the pre-trained model to this runtime filesystem."
      ],
      "metadata": {
        "id": "-PibNYIlpu4K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cp -av '/content/gdrive/My Drive/FLAVR_2x.pth' './FLAVR_2x.pth'"
      ],
      "metadata": {
        "id": "cEcpbDyW0axe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Unmount your Google Drive when done with the pre-trained model copy."
      ],
      "metadata": {
        "id": "ZLD3eO790bLP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "drive.flush_and_unmount()"
      ],
      "metadata": {
        "id": "8Meb4kd90eFF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Define a function to upload videos."
      ],
      "metadata": {
        "id": "_-ll5UukbWE_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import shutil\n",
        "from google.colab import files\n",
        "\n",
        "def upload_files(upload_path):\n",
        "  uploaded = files.upload()\n",
        "  for filename, content in uploaded.items():\n",
        "    dst_path = os.path.join(upload_path, filename)\n",
        "    shutil.move(filename, dst_path)\n",
        "  return list(uploaded.keys())"
      ],
      "metadata": {
        "id": "1R442tTXbcT9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a directory for uploaded videos."
      ],
      "metadata": {
        "id": "4edydmfoceHc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!mkdir ./test_videos\n",
        "image_input_dir = '/content/FLAVR/test_videos/'"
      ],
      "metadata": {
        "id": "Zl3_EauGcjqE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Slow-Motion Filtering"
      ],
      "metadata": {
        "id": "zJieST7OoVEV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Upload your own video."
      ],
      "metadata": {
        "id": "5tko37fpczcH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "uploaded_videos = upload_files(image_input_dir)"
      ],
      "metadata": {
        "id": "btKmYZb3ciJt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ['UPLOADED_VIDEO_FILENAME'] = os.path.join(image_input_dir, uploaded_videos[0])"
      ],
      "metadata": {
        "id": "KtbfV4g24LI7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Execute interpolation on the uploaded video."
      ],
      "metadata": {
        "id": "B8oX93NIc60B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python ./interpolate.py --input_video $UPLOADED_VIDEO_FILENAME --factor 2 --load_model ./FLAVR_2x.pth"
      ],
      "metadata": {
        "id": "I94Xy1e8dEW_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Display the result."
      ],
      "metadata": {
        "id": "LsmXsAVI-c3d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from moviepy.editor import VideoFileClip\n",
        "\n",
        "uploaded_video_filename_tokens = uploaded_videos[0].split('.')\n",
        "result_video_path = uploaded_video_filename_tokens[0] + '_2x.' + uploaded_video_filename_tokens[1]\n",
        "\n",
        "clip = VideoFileClip(result_video_path)\n",
        "clip.ipython_display(width=280)"
      ],
      "metadata": {
        "id": "Z9CeJL-Dd-Ul"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}