{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "!pip install -Uqq nixtla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "from nixtla.utils import in_colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "IN_COLAB = in_colab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    from nixtla.utils import colab_badge\n",
    "    from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Online (Real-Time) Anomaly Detection\n",
    "In this notebook, we introduce the `detect_anomalies_online` method. You will learn how to quickly start using this new endpoint and understand its key differences from the historical anomaly detection endpoint. New features include:\n",
    "* More flexibility and control over the anomaly detection process\n",
    "* Perform univariate and multivariate anomaly detection\n",
    "* Detect anomalies on stream data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Nixtla/nixtla/blob/main/nbs/docs/capabilities/online-anomaly-detection/01_quickstart.ipynb)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "if not IN_COLAB:\n",
    "    load_dotenv()\n",
    "    colab_badge('docs/capabilities/online-anomaly-detection/01_quickstart')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from nixtla import NixtlaClient\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nixtla_client = NixtlaClient(\n",
    "    # defaults to os.environ.get(\"NIXTLA_API_KEY\")\n",
    "    api_key = 'my_api_key_provided_by_nixtla'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 👍 Use an Azure AI endpoint\n",
    "> \n",
    "> To use an Azure AI endpoint, set the `base_url` argument:\n",
    "> \n",
    "> `nixtla_client = NixtlaClient(base_url=\"you azure ai endpoint\", api_key=\"your api_key\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    nixtla_client = NixtlaClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Dataset\n",
    "In this notebook, we use a minute-level time series dataset that monitors server usage. This is a good example of a streaming data scenario, as the task is to detect server failures or downtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('https://datasets-nixtla.s3.us-east-1.amazonaws.com/machine-1-1.csv', parse_dates=['ts'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the time series remains stable during the initial period; however, a spike occurs in the last 20 steps, indicating an anomalous behavior. Our goal is to capture this abnormal jump as soon as it appears. Let's see how the real-time anomaly detection capability of TimeGPT performs in this scenario!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32216 characters]",
      "text/plain": [
       "<Figure size 1200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "ax = df.tail(300).plot(x='ts', y='y', color = 'navy', title='Time Series', figsize=(12, 2))\n",
    "plt.axvspan('2020-02-01 21:00:00', '2020-02-01 21:02:00', color='orchid', alpha=0.3, label='Anomalous Period')\n",
    "plt.axvspan('2020-02-01 21:47:00', '2020-02-01 22:11:00', color='orchid', alpha=0.3)\n",
    "ax.legend(['Time Series', 'Anomalous Period'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Detect anomalies in real time\n",
    "The `detect_anomalies_online` method detect anomalies in a time series leveraging TimeGPT's forecast power. It uses the forecast error in deciding the anomalous step so you can specify and tune the parameters like that of the `forecast` method. This function will return a dataframe that contains anomaly flags and anomaly score (its absolute value quantifies the abnormality of the value).\n",
    "\n",
    "To perfom real-time anomaly detection, set the following parameters:\n",
    "\n",
    "- `df`: A pandas DataFrame containing the time series data.\n",
    "- `time_col`: The column that identifies the datestamp.\n",
    "- `target_col`: The variable to forecast.\n",
    "- `h`: Horizon is the number of steps ahead to make forecast.\n",
    "- `freq`: The frequency of the time series in Pandas format.\n",
    "- `level`: Percentile of scores distribution at which the threshold is set, controlling how strictly anomalies are flagged. Default at 99%.\n",
    "- `detection_size`: The number of steps to analyze for anomaly at the end of time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Calling Online Anomaly Detector Endpoint...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ts</th>\n",
       "      <th>y</th>\n",
       "      <th>TimeGPT</th>\n",
       "      <th>anomaly</th>\n",
       "      <th>anomaly_score</th>\n",
       "      <th>TimeGPT-hi-99</th>\n",
       "      <th>TimeGPT-lo-99</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>machine-1-1_y_29</td>\n",
       "      <td>2020-02-01 22:11:00</td>\n",
       "      <td>0.606017</td>\n",
       "      <td>0.544625</td>\n",
       "      <td>True</td>\n",
       "      <td>18.463266</td>\n",
       "      <td>0.553161</td>\n",
       "      <td>0.536090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>machine-1-1_y_29</td>\n",
       "      <td>2020-02-01 22:12:00</td>\n",
       "      <td>0.044413</td>\n",
       "      <td>0.570869</td>\n",
       "      <td>True</td>\n",
       "      <td>-158.933850</td>\n",
       "      <td>0.579404</td>\n",
       "      <td>0.562333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>machine-1-1_y_29</td>\n",
       "      <td>2020-02-01 22:13:00</td>\n",
       "      <td>0.038682</td>\n",
       "      <td>0.560303</td>\n",
       "      <td>True</td>\n",
       "      <td>-157.474880</td>\n",
       "      <td>0.568839</td>\n",
       "      <td>0.551767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>machine-1-1_y_29</td>\n",
       "      <td>2020-02-01 22:14:00</td>\n",
       "      <td>0.024355</td>\n",
       "      <td>0.521797</td>\n",
       "      <td>True</td>\n",
       "      <td>-150.178240</td>\n",
       "      <td>0.530333</td>\n",
       "      <td>0.513261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>machine-1-1_y_29</td>\n",
       "      <td>2020-02-01 22:15:00</td>\n",
       "      <td>0.044413</td>\n",
       "      <td>0.467860</td>\n",
       "      <td>True</td>\n",
       "      <td>-127.848560</td>\n",
       "      <td>0.476396</td>\n",
       "      <td>0.459325</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           unique_id                  ts         y   TimeGPT  anomaly  \\\n",
       "95  machine-1-1_y_29 2020-02-01 22:11:00  0.606017  0.544625     True   \n",
       "96  machine-1-1_y_29 2020-02-01 22:12:00  0.044413  0.570869     True   \n",
       "97  machine-1-1_y_29 2020-02-01 22:13:00  0.038682  0.560303     True   \n",
       "98  machine-1-1_y_29 2020-02-01 22:14:00  0.024355  0.521797     True   \n",
       "99  machine-1-1_y_29 2020-02-01 22:15:00  0.044413  0.467860     True   \n",
       "\n",
       "    anomaly_score  TimeGPT-hi-99  TimeGPT-lo-99  \n",
       "95      18.463266       0.553161       0.536090  \n",
       "96    -158.933850       0.579404       0.562333  \n",
       "97    -157.474880       0.568839       0.551767  \n",
       "98    -150.178240       0.530333       0.513261  \n",
       "99    -127.848560       0.476396       0.459325  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anomaly_online = nixtla_client.detect_anomalies_online(\n",
    "    df,\n",
    "    time_col='ts',                  \n",
    "    target_col='y',                 \n",
    "    freq='min',                     # Specify the frequency of the data\n",
    "    h=10,                           # Specify the forecast horizon\n",
    "    level=99,                       # Set the confidence level for anomaly detection\n",
    "    detection_size=100              # How many steps you want for analyzing anomalies\n",
    ")\n",
    "anomaly_online.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 In this example, we use a detection size of 100 to illustrate the anomaly detection process. In practice, using a smaller detection size and running the detection more frequently improves granularity and enables more timely identification of anomalies as they occur."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot, we observe that both anomalous periods were detected right as they arose. For further methods on improving detection accuracy and customizing anomaly detection, read our other tutorials on online anomaly detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56728 characters]",
      "text/plain": [
       "<Figure size 1200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "plt.figure(figsize=(12, 2))\n",
    "plt.plot(anomaly_online['ts'], anomaly_online['y'], label='y', color='navy', alpha=0.8)\n",
    "plt.plot(anomaly_online['ts'], anomaly_online['TimeGPT'], label='TimeGPT', color='orchid', alpha=0.7)\n",
    "plt.scatter(anomaly_online.loc[anomaly_online['anomaly'], 'ts'], anomaly_online.loc[anomaly_online['anomaly'], 'y'], color='orchid', label='Anomalies Detected')\n",
    "for t in ['2020-02-01 21:00:00', '2020-02-01 21:47:00']:\n",
    "    plt.axvline(pd.to_datetime(t), color='red', linestyle='--', alpha=0.7, label='Anomaly Behavior Captured' if t == '2020-02-01 21:00:00' else None)\n",
    "\n",
    "plt.axvspan('2020-02-01 21:00:00', '2020-02-01 21:02:00', color='orchid', alpha=0.3, label='Anomalous Period')\n",
    "plt.axvspan('2020-02-01 21:47:00', '2020-02-01 22:11:00', color='orchid', alpha=0.3)\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an in-depth analysis of the `detect_anomalies_online` method, refer to the tutorial (coming soon)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
