{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "!pip install -Uqq nixtla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "from nixtla.utils import in_colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "IN_COLAB = in_colab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    from nixtla.utils import colab_badge\n",
    "    from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why TimeGPT?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we compare the performance of TimeGPT against three forecasting models: the classical model (ARIMA), the machine learning model (LightGBM), and the deep learning model (N-HiTS), using a subset of data from the M5 Forecasting competition. We want to highlight three top-rated benefits our users love about TimeGPT:\n",
    "\n",
    "🎯 **Accuracy**: TimeGPT consistently outperforms traditional models by capturing complex patterns with precision.\n",
    "\n",
    "⚡ **Speed**: Generate forecasts faster without needing extensive training or tuning for each series.\n",
    "\n",
    "🚀 **Ease of Use**: Minimal setup and no complex preprocessing make TimeGPT accessible and ready to use right out of the box!\n",
    "\n",
    "Before diving into the notebook, please visit our [dashboard](https://dashboard.nixtla.io) to generate your TimeGPT `api_key` and give it a try yourself!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "1. [Data Introduction](#1-data-introduction)\n",
    "2. [Model Fitting](#2-model-fitting-timegpt-arima-lightgbm-n-hits)\n",
    "    1. [Fitting TimeGPT](#21-timegpt)\n",
    "    2. [Fitting ARIMA](#22-classical-models-arima)\n",
    "    3. [Fitting Light GBM](#23-machine-learning-models-lightgbm)\n",
    "    4. [Fitting NHITS](#24-n-hits)\n",
    "3. [Results and Evaluation](#3-performance-comparison-and-results)\n",
    "4. [Conclusion](#4-conclusion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Nixtla/nixtla/blob/main/nbs/docs/getting-started/7_why_timegpt.ipynb)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "if not IN_COLAB:\n",
    "    load_dotenv()\n",
    "    colab_badge('docs/getting-started/7_why_timegpt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from nixtla import NixtlaClient\n",
    "from utilsforecast.plotting import plot_series\n",
    "from utilsforecast.losses import mae, rmse, smape\n",
    "from utilsforecast.evaluation import evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nixtla_client = NixtlaClient(\n",
    "    # api_key = 'my_api_key_provided_by_nixtla'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we’re working with an aggregated dataset from the M5 Forecasting - Accuracy competition. This dataset includes **7 daily time series**, each with **1,941 data points**. The last **28 data points** of each series are set aside as the test set, allowing us to evaluate model performance on unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/demand_example.csv', parse_dates=['ds'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">ds</th>\n",
       "      <th colspan=\"4\" halign=\"left\">y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>min</th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>FOODS_1</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2674.085523</td>\n",
       "      <td>2665.0</td>\n",
       "      <td>5493.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FOODS_2</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4015.984029</td>\n",
       "      <td>3894.0</td>\n",
       "      <td>9069.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FOODS_3</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>10.0</td>\n",
       "      <td>16969.089129</td>\n",
       "      <td>16548.0</td>\n",
       "      <td>28663.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOBBIES_1</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2936.122617</td>\n",
       "      <td>2908.0</td>\n",
       "      <td>5009.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOBBIES_2</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>279.053065</td>\n",
       "      <td>248.0</td>\n",
       "      <td>871.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOUSEHOLD_1</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6039.594539</td>\n",
       "      <td>5984.0</td>\n",
       "      <td>11106.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HOUSEHOLD_2</th>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>2016-05-22</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1566.840289</td>\n",
       "      <td>1520.0</td>\n",
       "      <td>2926.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ds                      y                                \n",
       "                   min        max count   min          mean   median      max\n",
       "unique_id                                                                    \n",
       "FOODS_1     2011-01-29 2016-05-22  1941   0.0   2674.085523   2665.0   5493.0\n",
       "FOODS_2     2011-01-29 2016-05-22  1941   0.0   4015.984029   3894.0   9069.0\n",
       "FOODS_3     2011-01-29 2016-05-22  1941  10.0  16969.089129  16548.0  28663.0\n",
       "HOBBIES_1   2011-01-29 2016-05-22  1941   0.0   2936.122617   2908.0   5009.0\n",
       "HOBBIES_2   2011-01-29 2016-05-22  1941   0.0    279.053065    248.0    871.0\n",
       "HOUSEHOLD_1 2011-01-29 2016-05-22  1941   0.0   6039.594539   5984.0  11106.0\n",
       "HOUSEHOLD_2 2011-01-29 2016-05-22  1941   0.0   1566.840289   1520.0   2926.0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('unique_id').agg({\"ds\":[\"min\",\"max\",\"count\"],\\\n",
    "                             \"y\":[\"min\",\"mean\",\"median\",\"max\"]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(13391, 3) (196, 3)\n"
     ]
    }
   ],
   "source": [
    "df_train = df.query('ds <= \"2016-04-24\"')\n",
    "df_test = df.query('ds > \"2016-04-24\"')\n",
    "\n",
    "print(df_train.shape, df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Model Fitting (TimeGPT, ARIMA, LightGBM, N-HiTS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 TimeGPT\n",
    "TimeGPT offers a powerful, streamlined solution for time series forecasting, delivering state-of-the-art results with minimal effort. With TimeGPT, there's no need for data preprocessing or feature engineering -- simply initiate the Nixtla client and call `nixtla_client.forecast` to produce accurate, high-performance forecasts tailored to your unique time series.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Inferred freq: D\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Calling Forecast Endpoint...\n"
     ]
    }
   ],
   "source": [
    "# Forecast with TimeGPT\n",
    "fcst_timegpt = nixtla_client.forecast(df = df_train,\n",
    "                       target_col = 'y', \n",
    "                       h=28,                              # Forecast horizon, predicts the next 28 time steps\n",
    "                       model='timegpt-1-long-horizon',    # Use the model for long-horizon forecasting\n",
    "                       finetune_steps=10,                 # Number of finetuning steps\n",
    "                       level = [90])                      # Generate a 90% confidence interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "metric\n",
       "rmse     592.607378\n",
       "smape      0.049403\n",
       "Name: TimeGPT, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate performance and plot forecast\n",
    "fcst_timegpt['ds'] = pd.to_datetime(fcst_timegpt['ds'])\n",
    "test_df = pd.merge(df_test, fcst_timegpt, 'left', ['unique_id', 'ds'])\n",
    "evaluation_timegpt = evaluate(test_df, metrics=[rmse, smape], models=[\"TimeGPT\"])\n",
    "evaluation_timegpt.groupby(['metric'])['TimeGPT'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Classical Models (ARIMA):\n",
    "Next, we applied ARIMA, a traditional statistical model, to the same forecasting task. Classical models use historical trends and seasonality to make predictions by relying on linear assumptions. However, they struggled to capture the complex, non-linear patterns within the data, leading to lower accuracy compared to other approaches. Additionally, ARIMA was slower due to its iterative parameter estimation process, which becomes computationally intensive for larger datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 Why Use TimeGPT over Classical Models?\n",
    ">\n",
    "> * **Complex Patterns**: TimeGPT captures non-linear trends classical models miss.\n",
    ">\n",
    "> * **Minimal Preprocessing**: TimeGPT requires little to no data preparation.\n",
    ">\n",
    "> * **Scalability**: TimeGPT can efficiently scales across multiple series without retraining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "from statsforecast import StatsForecast\n",
    "from statsforecast.models import AutoARIMA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "#Initiate ARIMA model\n",
    "sf = StatsForecast(\n",
    "    models=[AutoARIMA(season_length=7)],\n",
    "    freq='D'\n",
    ")\n",
    "# Fit and forecast\n",
    "fcst_arima = sf.forecast(h=28, df=df_train) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "fcst_arima = pd.read_csv('../../assets/arima_rst.csv', parse_dates=['ds'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "metric\n",
       "rmse     724.957364\n",
       "smape      0.055018\n",
       "Name: AutoARIMA, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcst_arima.reset_index(inplace=True)\n",
    "test_df = pd.merge(df_test, fcst_arima, 'left', ['unique_id', 'ds'])\n",
    "evaluation_arima = evaluate(test_df, metrics=[rmse, smape], models=[\"AutoARIMA\"])\n",
    "evaluation_arima.groupby(['metric'])['AutoARIMA'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Machine Learning Models (LightGBM)\n",
    "\n",
    "Thirdly, we used a machine learning model, LightGBM, for the same forecasting task, implemented through the automated pipeline provided by our mlforecast library.\n",
    "While LightGBM can capture seasonality and patterns, achieving the best performance often requires detailed feature engineering, careful hyperparameter tuning, and domain knowledge. You can try our mlforecast library to simplify this process and get started quickly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 Why Use TimeGPT over Machine Learning Models?\n",
    ">\n",
    "> * **Automatic Pattern Recognition**: Captures complex patterns from raw data, bypassing the need for feature engineering.\n",
    ">\n",
    "> * **Minimal Tuning**: Works well without extensive tuning.\n",
    ">\n",
    "> * **Scalability**: Forecasts across multiple series without retraining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "import optuna\n",
    "from mlforecast.auto import AutoMLForecast, AutoLightGBM\n",
    "\n",
    "# Suppress Optuna's logging output\n",
    "optuna.logging.set_verbosity(optuna.logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "# Initialize an automated forecasting pipeline using AutoMLForecast.\n",
    "mlf = AutoMLForecast(\n",
    "    models=[AutoLightGBM()],\n",
    "    freq='D',\n",
    "    season_length=7,            \n",
    "    fit_config=lambda trial: {'static_features': ['unique_id']}\n",
    ")\n",
    "\n",
    "# Fit the model to the training dataset.\n",
    "mlf.fit(\n",
    "    df=df_train.astype({'unique_id': 'category'}),\n",
    "    n_windows=1,\n",
    "    h=28,\n",
    "    num_samples=10,\n",
    ")\n",
    "fcst_lgbm = mlf.predict(28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "fcst_lgbm = pd.read_csv('../../assets/lgbm_rst.csv', parse_dates=['ds'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "metric\n",
       "rmse     687.773744\n",
       "smape      0.051448\n",
       "Name: AutoLightGBM, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.merge(df_test, fcst_lgbm, 'left', ['unique_id', 'ds'])\n",
    "evaluation_lgbm = evaluate(test_df, metrics=[rmse, smape], models=[\"AutoLightGBM\"])\n",
    "evaluation_lgbm.groupby(['metric'])['AutoLightGBM'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 N-HiTS\n",
    "\n",
    "Lastly, we used N-HiTS, a state-of-the-art deep learning model designed for time series forecasting. The model produced accurate results, demonstrating its ability to capture complex, non-linear patterns within the data. However, setting up and tuning N-HiTS required significantly more time and computational resources compared to TimeGPT."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 Why Use TimeGPT Over Deep Learning Models?\n",
    ">\n",
    "> * **Faster Setup**: Quick setup and forecasting, unlike the lengthy configuration and training times of neural networks.\n",
    ">\n",
    "> * **Less Tuning**: Performs well with minimal tuning and preprocessing, while neural networks often need extensive adjustments.\n",
    ">\n",
    "> * **Ease of Use**: Simple deployment with high accuracy, making it accessible without deep technical expertise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "from neuralforecast.core import NeuralForecast\n",
    "from neuralforecast.models import NHITS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| eval: false\n",
    "# Initialize the N-HiTS model.\n",
    "models = [NHITS(h=28, \n",
    "                input_size=28, \n",
    "                max_steps=100)]\n",
    "\n",
    "# Fit the model using training data\n",
    "nf = NeuralForecast(models=models, freq='D')\n",
    "nf.fit(df=df_train)\n",
    "fcst_nhits = nf.predict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "fcst_nhits = pd.read_csv('../../assets/nhits_rst.csv', parse_dates=['ds'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "metric\n",
       "rmse     605.011948\n",
       "smape      0.053446\n",
       "Name: NHITS, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.merge(df_test,fcst_nhits, 'left', ['unique_id', 'ds'])\n",
    "evaluation_nhits = evaluate(test_df, metrics=[rmse, smape], models=[\"NHITS\"])\n",
    "evaluation_nhits.groupby(['metric'])['NHITS'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Performance Comparison and Results:\n",
    "The performance of each model is evaluated using RMSE (Root Mean Squared Error) and SMAPE (Symmetric Mean Absolute Percentage Error). While RMSE emphasizes the models’ ability to control significant errors, SMAPE provides a relative performance perspective by normalizing errors as percentages. Below, we present a snapshot of performance across all groups. The results demonstrate that TimeGPT outperforms other models on both metrics.\n",
    "\n",
    "🌟 For a deeper dive into benchmarking, check out our benchmark repository. The summarized results are displayed below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Overall Performance Metrics\n",
    "\n",
    "| **Model**        | **RMSE** | **SMAPE** |\n",
    "|------------------|----------|-----------|\n",
    "| ARIMA            | 724.9   | 5.50%     |\n",
    "| LightGBM     | 687.8    | 5.14%     |\n",
    "| N-HiTS           | 605.0    | 5.34%     |\n",
    "| **TimeGPT**      | **592.6**| **4.94%** |\n",
    " \n",
    "\n",
    "#### Breakdown for Each Time-series\n",
    "Followed below are the metrics for each individual time series groups. TimeGPT consistently delivers accurate forecasts across all time series groups. In many cases, it performs as well as or better than data-specific models, showing its versatility and reliability across different datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| echo: false\n",
    "evaluation_df = evaluation_arima.merge(evaluation_lgbm, on = ['unique_id','metric'], how = 'left')\\\n",
    "                 .merge(evaluation_nhits, on = ['unique_id','metric'], how = 'left')\\\n",
    "                 .merge(evaluation_timegpt, on = ['unique_id','metric'], how = 'left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56652 characters]",
      "text/plain": [
       "<Figure size 1400x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# | echo: false\n",
    "colors = [\n",
    "    (\"#A9B9C3\", 0.5),  # Grey-bluish color 1\n",
    "    (\"#7A8D9D\", 0.5),  # Grey-bluish color 2\n",
    "    (\"#5B6D79\", 0.5),  # Grey-bluish color 3\n",
    "    ('#F95D6A', 0.75)   # Green color for the last\n",
    "]\n",
    "\n",
    "\n",
    "# Filter evaluation data by metric and set unique_id as index\n",
    "rmse_df = evaluation_df[evaluation_df['metric'] == 'rmse'].set_index('unique_id')\n",
    "smape_df = evaluation_df[evaluation_df['metric'] == 'smape'].set_index('unique_id')\n",
    "\n",
    "# Plot function with custom colors and opacity\n",
    "def plot_metric(ax, df, title, ylabel):\n",
    "    x = np.arange(len(df))\n",
    "    bar_width = 0.2\n",
    "    for i, (col, (color, alpha)) in enumerate(zip(df.columns[1:], colors)):\n",
    "        ax.bar(x + i * bar_width, df[col], width=bar_width, label=col, color=color, alpha=alpha)\n",
    "    ax.set(title=title, ylabel=ylabel, xticks=x + bar_width * (len(df.columns[1:]) - 1) / 2, xticklabels=df.index)\n",
    "    ax.tick_params(axis='x', rotation=45)\n",
    "    ax.legend()\n",
    "\n",
    "# Generate side-by-side plots for RMSE and SMAPE\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
    "plot_metric(axes[0], rmse_df, \"RMSE Comparison Across Models\", \"RMSE\")\n",
    "plot_metric(axes[1], smape_df*100, \"%SMAPE Comparison Across Models\", \"SMAPE\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Benchmark Results\n",
    "For a more comprehensive dive into model accuracy and performance, explore our [Time Series Model Arena](https://github.com/Nixtla/nixtla/tree/main/experiments/foundation-time-series-arena)! TimeGPT continues to lead the pack with exceptional performance across benchmarks! 🌟\n",
    "\n",
    "<img src=\"https://github.com/Nixtla/nixtla/blob/main/nbs/img/timeseries_model_arena.png?raw=true\" alt=\"Benchmark\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Conclusion\n",
    "At the end of this notebook, we’ve put together a handy table to show you exactly where TimeGPT shines brightest compared to other forecasting models. ☀️ Think of it as your quick guide to choosing the best model for your unique project needs. We’re confident that TimeGPT will be a valuable tool in your forecasting journey. Don’t forget to visit our [dashboard](https://dashboard.nixtla.io) to generate your TimeGPT `api_key` and get started today! Happy forecasting, and enjoy the insights ahead! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Scenario                          | TimeGPT                                    | Classical Models (e.g., ARIMA)                         | Machine Learning Models (e.g., XGB, LGBM)               | Deep Learning Models (e.g., N-HITS)                   |\n",
    "|-----------------------------------|-------------------------------------------|-------------------------------------------------------|---------------------------------------------------------|-------------------------------------------------------|\n",
    "| **Seasonal Patterns**             | ✅ Performs well with minimal setup        | ✅ Handles seasonality with adjustments (e.g., SARIMA) | ✅ Performs well with feature engineering                | ✅ Captures seasonal patterns effectively              |\n",
    "| **Non-Linear Patterns**           | ✅ Excels, especially with complex non-linear patterns | ❌ Limited performance                                | ❌ Struggles without extensive feature engineering       | ✅ Performs well with non-linear relationships         |\n",
    "| **Large Dataset**                 | ✅ Highly scalable across many series      | ❌ Slow and resource-intensive                        | ✅ Scalable with optimized implementations               | ❌ Requires significant resources for large datasets   |\n",
    "| **Small Dataset**                 | ✅ Performs well; requires only one data point to start | ✅ Performs well; may struggle with very sparse data | ✅ Performs adequately if enough features are extracted  | ❌ May need a minimum data size to learn effectively   |\n",
    "| **Preprocessing Required**        | ✅ Minimal preprocessing needed            | ❌ Requires scaling, log-transform, etc., to meet model assumptions | ❌ Requires extensive feature engineering for complex patterns | ❌ Needs data normalization and preprocessing          |\n",
    "| **Accuracy Requirement**          | ✅ Achieves high accuracy with minimal tuning | ❌ May struggle with complex accuracy requirements   | ✅ Can achieve good accuracy with tuning                 | ✅ High accuracy possible but with significant resource use |\n",
    "| **Scalability**                   | ✅ Highly scalable with minimal task-specific configuration | ❌ Not easily scalable                               | ✅ Moderate scalability, with feature engineering and tuning per task | ❌ Limited scalability due to resource demands         |\n",
    "| **Computational Resources**       | ✅ Highly efficient, operates seamlessly on CPU, no GPU needed | ✅ Light to moderate, scales poorly with large datasets | ❌ Moderate, depends on feature complexity               | ❌ High resource consumption, often requires GPU       |\n",
    "| **Memory Requirement**            | ✅ Efficient memory usage for large datasets | ✅ Moderate memory requirements                      | ❌ High memory usage for larger datasets or many series cases | ❌ High memory consumption for larger datasets and multiple series |\n",
    "| **Technical Requirements & Domain Knowledge** | ✅ Low; minimal technical setup and no domain expertise needed | ✅ Low to moderate; needs understanding of stationarity | ❌ Moderate to high; requires feature engineering and tuning | ❌ High; complex architecture and tuning               |\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
