FastPitch_voice_modification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastPitch: Voice Modification with Pre-defined Pitch Transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [FastPitch](https://arxiv.org/abs/2006.06873) model is based on the [FastSpeech](https://arxiv.org/abs/1905.09263) model. Similarly to [FastSpeech2](https://arxiv.org/abs/2006.04558), which has been developed concurrently, it learns to predict the pitch contour and conditions the generation on such contour.\n",
    "\n",
    "The simple mechanism of predicting the pitch on grapheme-level (rather than frame-level, as FastSpeech2 does) allows to easily alter the pitch during synthesis. FastPitch can thus change the perceived emotional state of the speaker, or slightly emphasise certain lexical units."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the notebook inside the container. By default the container forwards port `8888`.\n",
    "```\n",
    "bash scripts/docker/interactive.sh\n",
    "\n",
    "# inside the container\n",
    "cd notebooks\n",
    "jupyter notebook --ip='*' --port=8888\n",
    "```\n",
    "Please refer the Requirement section in `README.md` for more details and running outside the container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "assert os.getcwd().split('/')[-1] == 'notebooks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate audio samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training a FastPitch model from scrath takes 3 to 27 hours depending on the type and number of GPUs, performance numbers can be found in Section \"Training performance results\" in `README.md`. Therefore, to save the time of running this notebook, we recommend to download the pretrained FastPitch checkpoints on NGC for inference.\n",
    "\n",
    "You can find FP32 checkpoint at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:fastpitch_pyt_fp32_ckpt_v1/files) , and AMP (Automatic Mixed Precision) checkpoint at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:fastpitch_pyt_amp_ckpt_v1/files).\n",
    "\n",
    "To synthesize audio, you will need a WaveGlow model, which generates waveforms based on mel-spectrograms generated by FastPitch.You can download a pre-trained WaveGlow AMP model at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:waveglow256pyt_fp16)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p output\n",
    "! MODEL_DIR='../pretrained_models' ../scripts/download_fastpitch.sh\n",
    "! MODEL_DIR='../pretrained_models' ../scripts/download_waveglow.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform inference using the respective checkpoints that are passed as `--fastpitch` and `--waveglow` arguments. Next, you will use FastPitch model to generate audio samples for input text, including the basic version and the variations i npace, fade out, and pitch transforms, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython\n",
    "\n",
    "# store paths in aux variables\n",
    "fastp = '../pretrained_models/fastpitch/nvidia_fastpitch_200518.pt'\n",
    "waveg = '../pretrained_models/waveglow/waveglow_1076430_14000_amp.pt'\n",
    "flags = f'--cuda --fastpitch {fastp} --waveglow {waveg} --wn-channels 256'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Basic speech synthesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to create an input file with some text, or just input the text in the below cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile text.txt\n",
    "The forms of printed letters should be beautiful, and that their arrangement on the page should be reasonable and a help to the shapeliness of the letters themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the script below to generate audio from the input text file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# basic systhesis\n",
    "!python ../inference.py {flags} -i text.txt -o output/original > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/original/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add variations to the generated speech"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastPitch allows us to exert additional control over the synthesized utterances, the key parameters are the pace, fade out, and pitch transforms in particular."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Pace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastPitch allows you to linearly adjust the pace of synthesized speech, similar to [FastSpeech](https://arxiv.org/abs/1905.09263) model. For instance, pass --pace 0.5 for a twofold decrease in speed, --pace 1.0 = unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the pace of speech to double with --pace 0.5\n",
    "# (1.0 = unchanged)\n",
    "!python ../inference.py {flags} -i text.txt -o output/pace --pace 0.5 > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/pace/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Raise or lower the pitch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For every input character, the model predicts a pitch cue - an average pitch over a character in Hz. Pitch can be adjusted by transforming those pitch cues. A few simple examples are provided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Raise/lower pitch by --pitch-transform-shift <Hz>\n",
    "# Synthesize with a -50 Hz shift\n",
    "!python ../inference.py {flags} -i text.txt -o output/riselowpitch --pitch-transform-shift -50 > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/riselowpitch/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Flatten the pitch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten the pitch to a constant value with --pitch-transform-flatten\n",
    "!python ../inference.py {flags} -i text.txt -o output/flattenpitch --pitch-transform-flatten > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/flattenpitch/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Invert the pitch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Invert pitch wrt. to the mean pitch with --pitch-transform-invert\n",
    "!python ../inference.py {flags} -i text.txt -o output/invertpitch --pitch-transform-invert > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/invertpitch/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Amplify the pitch "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Amplify pitch wrt. to the mean pitch with --pitch-transform-amplify 2.0\n",
    "# values in the (1.0, 3.0) range work the best\n",
    "!python ../inference.py {flags} -i text.txt -o output/amplifypitch --pitch-transform-amplify 2.0 > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/amplifypitch/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Combine the flags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The flags can be combined. You can find all the available options by calling python inference.py --help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ../inference.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below example shows how to generate an audio with a combination of the flags --pace --pitch-transform-flatten --pitch-transform-shift --pitch-transform-invert --pitch-transform-amplify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dobuble the speed and combine multiple transformations\n",
    "!python ../inference.py {flags} -i text.txt -o output/combine \\\n",
    "    --pace 2.0 --pitch-transform-flatten --pitch-transform-shift 50 \\\n",
    "    --pitch-transform-invert --pitch-transform-amplify 1.5 > /dev/null\n",
    "\n",
    "IPython.display.Audio(\"output/combine/audio_0.wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Inference performance benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Benchmark inference using AMP\n",
    "!python ../inference.py {flags} \\\n",
    "    --include-warmup --batch-size 8 --repeats 100 --torchscript --amp \\\n",
    "    -i ../phrases/benchmark_8_128.tsv -o output/benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Next step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have learnt how to generate high quality audio from text using FastPitch, as well as add variations to the audio using the flags. You can experiment with more input texts, or change the hyperparameters of the models, such as pitch flags, batch size, different precisions, etc, to see if they could improve the inference results.\n",
    "\n",
    "If you are interested in learning more about FastPitch, please check more samples (trained with multi-speaker) presented at [samples page](https://fastpitch.github.io/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}