{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "!pip install -Uqq nixtla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "from nixtla.utils import in_colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "IN_COLAB = in_colab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    from nixtla.utils import colab_badge\n",
    "    from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anomaly detection\n",
    "\n",
    "Anomaly detection is task of detecting abnormal points, points that deviate from the normal behaviour of the general series. This is crucial in many application, such as cybersecurity or equipment monitoring.\n",
    "\n",
    "In this tutorial, we explore in detail the anomaly detection capability of TimeGPT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Nixtla/nixtla/blob/main/nbs/docs/tutorials/20_anomaly_detection.ipynb)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "if not IN_COLAB:\n",
    "    load_dotenv()\n",
    "    colab_badge('docs/tutorials/20_anomaly_detection')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages\n",
    "\n",
    "First, we import the required packages for this tutorial and create an instance of `NixtlaClient`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from nixtla import NixtlaClient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nixtla_client = NixtlaClient(\n",
    "    # defaults to os.environ.get(\"NIXTLA_API_KEY\")\n",
    "    api_key = 'my_api_key_provided_by_nixtla'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 👍 Use an Azure AI endpoint\n",
    ">\n",
    "> To use an Azure AI endpoint, set the `base_url` argument:\n",
    ">\n",
    "> `nixtla_client = NixtlaClient(base_url=\"you azure ai endpoint\", api_key=\"your api_key\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    nixtla_client = NixtlaClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "Now, let's load the dataset for this tutorial. We use the Peyton Manning dataset which tracks the visits to the Wikipedia page of Peyton Mannig."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2007-12-10</td>\n",
       "      <td>9.590761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2007-12-11</td>\n",
       "      <td>8.519590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2007-12-12</td>\n",
       "      <td>8.183677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>2007-12-13</td>\n",
       "      <td>8.072467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2007-12-14</td>\n",
       "      <td>7.893572</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id          ds         y\n",
       "0          0  2007-12-10  9.590761\n",
       "1          0  2007-12-11  8.519590\n",
       "2          0  2007-12-12  8.183677\n",
       "3          0  2007-12-13  8.072467\n",
       "4          0  2007-12-14  7.893572"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/peyton-manning.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 91212 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(df, max_insample_length=365)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Anomaly detection\n",
    "\n",
    "We now perform anomaly detection. By default, TimeGPT uses a 99% confidence interval. If a point falls outisde of that interval, it is considered to be an anomaly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Calling Anomaly Detector Endpoint...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>y</th>\n",
       "      <th>TimeGPT</th>\n",
       "      <th>TimeGPT-hi-99</th>\n",
       "      <th>TimeGPT-lo-99</th>\n",
       "      <th>anomaly</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2008-01-10</td>\n",
       "      <td>8.281724</td>\n",
       "      <td>8.224187</td>\n",
       "      <td>9.503586</td>\n",
       "      <td>6.944788</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2008-01-11</td>\n",
       "      <td>8.292799</td>\n",
       "      <td>8.151533</td>\n",
       "      <td>9.430932</td>\n",
       "      <td>6.872135</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2008-01-12</td>\n",
       "      <td>8.199189</td>\n",
       "      <td>8.127243</td>\n",
       "      <td>9.406642</td>\n",
       "      <td>6.847845</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>2008-01-13</td>\n",
       "      <td>9.996522</td>\n",
       "      <td>8.917259</td>\n",
       "      <td>10.196658</td>\n",
       "      <td>7.637861</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>2008-01-14</td>\n",
       "      <td>10.127071</td>\n",
       "      <td>9.002326</td>\n",
       "      <td>10.281725</td>\n",
       "      <td>7.722928</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id         ds          y   TimeGPT  TimeGPT-hi-99  TimeGPT-lo-99  \\\n",
       "0          0 2008-01-10   8.281724  8.224187       9.503586       6.944788   \n",
       "1          0 2008-01-11   8.292799  8.151533       9.430932       6.872135   \n",
       "2          0 2008-01-12   8.199189  8.127243       9.406642       6.847845   \n",
       "3          0 2008-01-13   9.996522  8.917259      10.196658       7.637861   \n",
       "4          0 2008-01-14  10.127071  9.002326      10.281725       7.722928   \n",
       "\n",
       "   anomaly  \n",
       "0    False  \n",
       "1    False  \n",
       "2    False  \n",
       "3    False  \n",
       "4    False  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anomalies_df = nixtla_client.detect_anomalies(df, freq='D')\n",
    "anomalies_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.detect_anomalies(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, `False` is assigned to \"normal\" values, as they fall inside the confidence interval. A label of `True` is then assigned to abnormal points.\n",
    "\n",
    "We can also plot the anomalies using `NixtlaClient`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 190780 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(df, anomalies_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Anomaly detection with exogenous features\n",
    "\n",
    "Previously, we performed anomaly detection without using any exogenous features. Now, it is possible to create features specifically for this scenario to inform the model in its task of anomaly detection.\n",
    "\n",
    "Here, we create date features that can be used by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is done using the `date_features` argument. We can set it to `True` and it will generate all possible features from the given dates and frequency of the data. Alternatively, we can specify a list of features that we want. In this case, we want only features at the *month* and *year* level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Using the following exogenous features: ['month_1.0', 'month_2.0', 'month_3.0', 'month_4.0', 'month_5.0', 'month_6.0', 'month_7.0', 'month_8.0', 'month_9.0', 'month_10.0', 'month_11.0', 'month_12.0', 'year_2007.0', 'year_2008.0', 'year_2009.0', 'year_2010.0', 'year_2011.0', 'year_2012.0', 'year_2013.0', 'year_2014.0', 'year_2015.0', 'year_2016.0']\n",
      "INFO:nixtla.nixtla_client:Calling Anomaly Detector Endpoint...\n"
     ]
    }
   ],
   "source": [
    "anomalies_df_x = nixtla_client.detect_anomalies(\n",
    "    df,\n",
    "    freq='D', \n",
    "    date_features=['month', 'year'],\n",
    "    date_features_to_one_hot=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can plot the detected anomalies where the model now used additional information from exogenous features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 167840 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(df, anomalies_df_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the confidence intervals\n",
    "\n",
    "We can tweak the confidence intervals using the `level` argument. This takes any values between 0 and 100, including decimal numbers.\n",
    "\n",
    "Reducing the confidence interval resutls in more anomalies being detected, while increasing it will reduce the number of anomalies.\n",
    "\n",
    "Here, for example, we reduce the interval to 70%, and we will notice more anomalies being plotted (red dots)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Calling Anomaly Detector Endpoint...\n"
     ]
    }
   ],
   "source": [
    "anomalies_df = nixtla_client.detect_anomalies(\n",
    "    df, \n",
    "    freq='D',\n",
    "    level=70\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 236752 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(df, anomalies_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
