{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "450114d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "!pip install -Uqq nixtla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6ed6931",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "from nixtla.utils import in_colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f3add2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "IN_COLAB = in_colab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "156d2425",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    from nixtla.utils import colab_badge\n",
    "    from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6de758ee-a0d2-4b3f-acff-eed419dd17c5",
   "metadata": {},
   "source": [
    "# Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d267032-535b-4b7b-b7d3-d2db8f673af6",
   "metadata": {},
   "source": [
    "One of the primary challenges in time series forecasting is the inherent uncertainty and variability over time, making it crucial to validate the accuracy and reliability of the models employed. Cross-validation, a robust model validation technique, is particularly adapted for this task, as it provides insights into the expected performance of a model on unseen data, ensuring the forecasts are reliable and resilient before being deployed in real-world scenarios.\n",
    "\n",
    "`TimeGPT`, understanding the intricate needs of time series forecasting, incorporates the `cross_validation` method, designed to streamline the validation process for time series models. This functionality enables practitioners to rigorously test their forecasting models against historical data, assessing their effectiveness while tuning them for optimal performance. This tutorial will guide you through the nuanced process of conducting cross-validation within the `NixtlaClient` class, ensuring your time series forecasting models are not just well-constructed, but also validated for trustworthiness and precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30bcfdcd-2554-4d80-b2bd-7edf6e0b1056",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Nixtla/nixtla/blob/main/nbs/docs/tutorials/08_cross_validation.ipynb)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "if not IN_COLAB:\n",
    "    load_dotenv()    \n",
    "    colab_badge('docs/tutorials/08_cross_validation')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50299317",
   "metadata": {},
   "source": [
    "## 1. Import packages\n",
    "First, we install and import the required packages and initialize the Nixtla client."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca8110a6",
   "metadata": {},
   "source": [
    "We start off by initializing an instance of `NixtlaClient`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77c6d3be-dd0c-4590-b59f-0b3495e00839",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from nixtla import NixtlaClient\n",
    "\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "462c262f-172a-430e-9883-c6684a5c4760",
   "metadata": {},
   "outputs": [],
   "source": [
    "nixtla_client = NixtlaClient(\n",
    "    # defaults to os.environ.get(\"NIXTLA_API_KEY\")\n",
    "    api_key = 'my_api_key_provided_by_nixtla'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c0e22f",
   "metadata": {},
   "source": [
    "> 👍 Use an Azure AI endpoint\n",
    "> \n",
    "> To use an Azure AI endpoint, remember to set also the `base_url` argument:\n",
    "> \n",
    "> `nixtla_client = NixtlaClient(base_url=\"you azure ai endpoint\", api_key=\"your api_key\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a505e6e4-61bd-4762-943b-c55e4e31c193",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    nixtla_client = NixtlaClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c852d79b",
   "metadata": {},
   "source": [
    "## 2. Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf6415cb",
   "metadata": {},
   "source": [
    "Let's see an example, using the Peyton Manning dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cf2ef4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pm_df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/peyton-manning.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd57a883",
   "metadata": {},
   "source": [
    "## 3. Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937ccb60-8a1b-4a58-9111-d9fb9d8d727c",
   "metadata": {},
   "source": [
    "The `cross_validation` method within the `TimeGPT` class is an advanced functionality crafted to perform systematic validation on time series forecasting models. This method necessitates a dataframe comprising time-ordered data and employs a rolling-window scheme to meticulously evaluate the model's performance across different time periods, thereby ensuring the model's reliability and stability over time. The animation below shows how TimeGPT performs cross-validation.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/Nixtla/statsforecast/main/nbs/imgs/ChainedWindows.gif\" alt=\"Rolling-window cross-validation\" /> \n",
    "\n",
    "Key parameters include `freq`, which denotes the data's frequency and is automatically inferred if not specified. The `id_col`, `time_col`, and `target_col` parameters designate the respective columns for each series' identifier, time step, and target values. The method offers customization through parameters like `n_windows`, indicating the number of separate time windows on which the model is assessed, and `step_size`, determining the gap between these windows. If `step_size` is unspecified, it defaults to the forecast horizon `h`. \n",
    "\n",
    "The process also allows for model refinement via `finetune_steps`, specifying the number of iterations for model fine-tuning on new data. Data pre-processing is manageable through `clean_ex_first`, deciding whether to cleanse the exogenous signal prior to forecasting. Additionally, the method supports enhanced feature engineering from time data through the `date_features` parameter, which can automatically generate crucial date-related features or accept custom functions for bespoke feature creation. The `date_features_to_one_hot` parameter further enables the transformation of categorical date features into a format suitable for machine learning models.\n",
    "\n",
    "In execution, `cross_validation` assesses the model's forecasting accuracy in each window, providing a robust view of the model's performance variability over time and potential overfitting. This detailed evaluation ensures the forecasts generated are not only accurate but also consistent across diverse temporal contexts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c2f02bf-b13a-4d8d-aeb6-fd59bffe549e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Restricting input...\n",
      "INFO:nixtla.nixtla_client:Calling Cross Validation Endpoint...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>cutoff</th>\n",
       "      <th>y</th>\n",
       "      <th>TimeGPT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-17</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.591862</td>\n",
       "      <td>7.939553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-18</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.528869</td>\n",
       "      <td>7.887512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-19</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.171657</td>\n",
       "      <td>7.766617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-20</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.891331</td>\n",
       "      <td>7.931502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-21</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>8.360071</td>\n",
       "      <td>8.312632</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id         ds     cutoff         y   TimeGPT\n",
       "0          0 2015-12-17 2015-12-16  7.591862  7.939553\n",
       "1          0 2015-12-18 2015-12-16  7.528869  7.887512\n",
       "2          0 2015-12-19 2015-12-16  7.171657  7.766617\n",
       "3          0 2015-12-20 2015-12-16  7.891331  7.931502\n",
       "4          0 2015-12-21 2015-12-16  8.360071  8.312632"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timegpt_cv_df = nixtla_client.cross_validation(\n",
    "    pm_df, \n",
    "    h=7, \n",
    "    n_windows=5, \n",
    "    freq='D',\n",
    ")\n",
    "timegpt_cv_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c7e0749",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.cross_validation(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0793fd0d-c762-4f94-b704-1e5312886fbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 78424 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 79284 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 79336 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 79468 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 79616 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cutoffs = timegpt_cv_df['cutoff'].unique()\n",
    "for cutoff in cutoffs:\n",
    "    fig = nixtla_client.plot(\n",
    "        pm_df.tail(100), \n",
    "        timegpt_cv_df.query('cutoff == @cutoff').drop(columns=['cutoff', 'y']),\n",
    "    )\n",
    "    display(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be475644",
   "metadata": {},
   "source": [
    "## 4. Cross-validation with prediction intervals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84e9a89-8de1-462f-a8d8-e45347031d23",
   "metadata": {},
   "source": [
    "It is also possible to generate prediction intervals during cross-validation. To do so, we simply use the `level` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2e6a0db-1253-4fd3-973d-0c373911af13",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Restricting input...\n",
      "INFO:nixtla.nixtla_client:Calling Cross Validation Endpoint...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>cutoff</th>\n",
       "      <th>y</th>\n",
       "      <th>TimeGPT</th>\n",
       "      <th>TimeGPT-hi-80</th>\n",
       "      <th>TimeGPT-hi-90</th>\n",
       "      <th>TimeGPT-lo-80</th>\n",
       "      <th>TimeGPT-lo-90</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-17</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.591862</td>\n",
       "      <td>7.939553</td>\n",
       "      <td>8.201465</td>\n",
       "      <td>8.314956</td>\n",
       "      <td>7.677642</td>\n",
       "      <td>7.564151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-18</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.528869</td>\n",
       "      <td>7.887512</td>\n",
       "      <td>8.175414</td>\n",
       "      <td>8.207470</td>\n",
       "      <td>7.599609</td>\n",
       "      <td>7.567553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-19</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.171657</td>\n",
       "      <td>7.766617</td>\n",
       "      <td>8.267363</td>\n",
       "      <td>8.386674</td>\n",
       "      <td>7.265871</td>\n",
       "      <td>7.146560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-20</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.891331</td>\n",
       "      <td>7.931502</td>\n",
       "      <td>8.205929</td>\n",
       "      <td>8.369983</td>\n",
       "      <td>7.657075</td>\n",
       "      <td>7.493020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-21</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>8.360071</td>\n",
       "      <td>8.312632</td>\n",
       "      <td>9.184893</td>\n",
       "      <td>9.625794</td>\n",
       "      <td>7.440371</td>\n",
       "      <td>6.999469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id         ds     cutoff         y   TimeGPT  TimeGPT-hi-80  \\\n",
       "0          0 2015-12-17 2015-12-16  7.591862  7.939553       8.201465   \n",
       "1          0 2015-12-18 2015-12-16  7.528869  7.887512       8.175414   \n",
       "2          0 2015-12-19 2015-12-16  7.171657  7.766617       8.267363   \n",
       "3          0 2015-12-20 2015-12-16  7.891331  7.931502       8.205929   \n",
       "4          0 2015-12-21 2015-12-16  8.360071  8.312632       9.184893   \n",
       "\n",
       "   TimeGPT-hi-90  TimeGPT-lo-80  TimeGPT-lo-90  \n",
       "0       8.314956       7.677642       7.564151  \n",
       "1       8.207470       7.599609       7.567553  \n",
       "2       8.386674       7.265871       7.146560  \n",
       "3       8.369983       7.657075       7.493020  \n",
       "4       9.625794       7.440371       6.999469  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timegpt_cv_df = nixtla_client.cross_validation(\n",
    "    pm_df, \n",
    "    h=7, \n",
    "    n_windows=5, \n",
    "    freq='D',\n",
    "    level=[80, 90],\n",
    ")\n",
    "timegpt_cv_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d73b97d1",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.cross_validation(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa92c392",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 87872 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 88576 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 89760 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 90552 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 90488 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cutoffs = timegpt_cv_df['cutoff'].unique()\n",
    "for cutoff in cutoffs:\n",
    "    fig = nixtla_client.plot(\n",
    "        pm_df.tail(100), \n",
    "        timegpt_cv_df.query('cutoff == @cutoff').drop(columns=['cutoff', 'y']),\n",
    "        level=[80, 90],\n",
    "        models=['TimeGPT']\n",
    "    )\n",
    "    display(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b8f68b",
   "metadata": {},
   "source": [
    "## 5. Cross-validation with exogenous variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c27f048",
   "metadata": {},
   "source": [
    "### Time features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84388bb9-54c3-408e-bae2-46e39ffc3ee5",
   "metadata": {},
   "source": [
    "It is possible to include exogenous variables when performing cross-validation. Here we use the `date_features` parameter to create labels for each month. These features are then used by the model to make predictions during cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc956968-902a-4b35-a3e1-8db30a6b7448",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Using the following exogenous features: ['month_1.0', 'month_2.0', 'month_3.0', 'month_4.0', 'month_5.0', 'month_6.0', 'month_7.0', 'month_8.0', 'month_9.0', 'month_10.0', 'month_11.0', 'month_12.0']\n",
      "INFO:nixtla.nixtla_client:Calling Cross Validation Endpoint...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>cutoff</th>\n",
       "      <th>y</th>\n",
       "      <th>TimeGPT</th>\n",
       "      <th>TimeGPT-hi-80</th>\n",
       "      <th>TimeGPT-hi-90</th>\n",
       "      <th>TimeGPT-lo-80</th>\n",
       "      <th>TimeGPT-lo-90</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-12-17</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.591862</td>\n",
       "      <td>8.426320</td>\n",
       "      <td>8.721996</td>\n",
       "      <td>8.824101</td>\n",
       "      <td>8.130644</td>\n",
       "      <td>8.028540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-12-18</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.528869</td>\n",
       "      <td>8.049962</td>\n",
       "      <td>8.452083</td>\n",
       "      <td>8.658603</td>\n",
       "      <td>7.647842</td>\n",
       "      <td>7.441321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-12-19</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.171657</td>\n",
       "      <td>7.509098</td>\n",
       "      <td>7.984788</td>\n",
       "      <td>8.138017</td>\n",
       "      <td>7.033409</td>\n",
       "      <td>6.880180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-12-20</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>7.891331</td>\n",
       "      <td>7.739536</td>\n",
       "      <td>8.306914</td>\n",
       "      <td>8.641355</td>\n",
       "      <td>7.172158</td>\n",
       "      <td>6.837718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-12-21</td>\n",
       "      <td>2015-12-16</td>\n",
       "      <td>8.360071</td>\n",
       "      <td>8.027471</td>\n",
       "      <td>8.722828</td>\n",
       "      <td>9.152306</td>\n",
       "      <td>7.332113</td>\n",
       "      <td>6.902636</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id         ds     cutoff         y   TimeGPT  TimeGPT-hi-80  \\\n",
       "0        0.0 2015-12-17 2015-12-16  7.591862  8.426320       8.721996   \n",
       "1        0.0 2015-12-18 2015-12-16  7.528869  8.049962       8.452083   \n",
       "2        0.0 2015-12-19 2015-12-16  7.171657  7.509098       7.984788   \n",
       "3        0.0 2015-12-20 2015-12-16  7.891331  7.739536       8.306914   \n",
       "4        0.0 2015-12-21 2015-12-16  8.360071  8.027471       8.722828   \n",
       "\n",
       "   TimeGPT-hi-90  TimeGPT-lo-80  TimeGPT-lo-90  \n",
       "0       8.824101       8.130644       8.028540  \n",
       "1       8.658603       7.647842       7.441321  \n",
       "2       8.138017       7.033409       6.880180  \n",
       "3       8.641355       7.172158       6.837718  \n",
       "4       9.152306       7.332113       6.902636  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timegpt_cv_df = nixtla_client.cross_validation(\n",
    "    pm_df, \n",
    "    h=7, \n",
    "    n_windows=5,  \n",
    "    freq='D',\n",
    "    level=[80, 90],\n",
    "    date_features=['month'],\n",
    "    date_features_to_one_hot=True,\n",
    ")\n",
    "timegpt_cv_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b40b247",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 88740 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 88432 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 91764 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 91084 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 90380 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cutoffs = timegpt_cv_df['cutoff'].unique()\n",
    "for cutoff in cutoffs:\n",
    "    fig = nixtla_client.plot(\n",
    "        pm_df.tail(100), \n",
    "        timegpt_cv_df.query('cutoff == @cutoff').drop(columns=['cutoff', 'y']),\n",
    "        level=[80, 90],\n",
    "        models=['TimeGPT']\n",
    "    )\n",
    "    display(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ca2ffe2",
   "metadata": {},
   "source": [
    "### Dynamic features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95ea323-cd6d-43cb-aed1-f10cf23c5a61",
   "metadata": {},
   "source": [
    "Additionally you can pass dynamic exogenous variables to better inform `TimeGPT` about the data. You just simply have to add the exogenous regressors after the target column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85137aed-17d5-4d04-939a-5a83219b97a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_df = pd.read_csv('https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/electricity.csv')\n",
    "X_df = pd.read_csv('https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/exogenous-vars-electricity.csv')\n",
    "df = Y_df.merge(X_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2eb7f1a-2ea8-4c60-995a-808d2dcdd771",
   "metadata": {},
   "source": [
    "Now let's cross validate `TimeGPT` considering this information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab7d17a5-d344-414e-a4c5-9500cab26a9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    nixtla_client = NixtlaClient(max_retries=2, retry_interval=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6903c91e-38f3-4313-9e80-edb90421faf4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Inferred freq: h\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Using the following exogenous features: ['Exogenous1', 'Exogenous2', 'day_0', 'day_1', 'day_2', 'day_3', 'day_4', 'day_5', 'day_6']\n",
      "INFO:nixtla.nixtla_client:Calling Cross Validation Endpoint...\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 112444 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 113636 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "timegpt_cv_df_x = nixtla_client.cross_validation(\n",
    "    df.groupby('unique_id').tail(100 * 48), \n",
    "    h=48, \n",
    "    n_windows=2,\n",
    "    level=[80, 90]\n",
    ")\n",
    "cutoffs = timegpt_cv_df_x.query('unique_id == \"BE\"')['cutoff'].unique()\n",
    "for cutoff in cutoffs:\n",
    "    fig = nixtla_client.plot(\n",
    "        df.query('unique_id == \"BE\"').tail(24 * 7), \n",
    "        timegpt_cv_df_x.query('cutoff == @cutoff & unique_id == \"BE\"').drop(columns=['cutoff', 'y']),\n",
    "        models=['TimeGPT'],\n",
    "        level=[80, 90],\n",
    "    )\n",
    "    display(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ef9e40b",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.cross_validation(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c8c469-bbb5-45ef-bd49-07bfdbc51b6b",
   "metadata": {},
   "source": [
    "## 6. Cross-validation with different TimeGPT instances\n",
    "\n",
    "Also, you can generate cross validation for different instances of `TimeGPT` using the `model` argument. Here we use the base model and the model for long-horizon forecasting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bee1ec6a-3529-4e6c-be20-6cab3e032911",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Inferred freq: h\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Using the following exogenous features: ['Exogenous1', 'Exogenous2', 'day_0', 'day_1', 'day_2', 'day_3', 'day_4', 'day_5', 'day_6']\n",
      "INFO:nixtla.nixtla_client:Calling Cross Validation Endpoint...\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 139964 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 143312 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "timegpt_cv_df_x_long_horizon = nixtla_client.cross_validation(\n",
    "    df.groupby('unique_id').tail(100 * 48), \n",
    "    h=48, \n",
    "    n_windows=2,\n",
    "    level=[80, 90],\n",
    "    model='timegpt-1-long-horizon',\n",
    ")\n",
    "timegpt_cv_df_x_long_horizon.columns = timegpt_cv_df_x_long_horizon.columns.str.replace('TimeGPT', 'TimeGPT-LongHorizon')\n",
    "timegpt_cv_df_x_models = timegpt_cv_df_x_long_horizon.merge(timegpt_cv_df_x)\n",
    "cutoffs = timegpt_cv_df_x_models.query('unique_id == \"BE\"')['cutoff'].unique()\n",
    "for cutoff in cutoffs:\n",
    "    fig = nixtla_client.plot(\n",
    "        df.query('unique_id == \"BE\"').tail(24 * 7), \n",
    "        timegpt_cv_df_x_models.query('cutoff == @cutoff & unique_id == \"BE\"').drop(columns=['cutoff', 'y']),\n",
    "        models=['TimeGPT', 'TimeGPT-LongHorizon'],\n",
    "        level=[80, 90],\n",
    "    )\n",
    "    display(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6593cf18",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.cross_validation(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
