{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas_market_calendars in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (4.6.0)\n",
      "Requirement already satisfied: pandas>=1.1 in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from pandas_market_calendars) (2.1.4)\n",
      "Requirement already satisfied: pytz in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from pandas_market_calendars) (2024.1)\n",
      "Requirement already satisfied: python-dateutil in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from pandas_market_calendars) (2.9.0)\n",
      "Requirement already satisfied: exchange-calendars>=3.3 in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from pandas_market_calendars) (4.8)\n",
      "Requirement already satisfied: numpy in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from exchange-calendars>=3.3->pandas_market_calendars) (1.26.4)\n",
      "Requirement already satisfied: pyluach in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from exchange-calendars>=3.3->pandas_market_calendars) (2.2.0)\n",
      "Requirement already satisfied: toolz in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from exchange-calendars>=3.3->pandas_market_calendars) (0.12.1)\n",
      "Requirement already satisfied: tzdata in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from exchange-calendars>=3.3->pandas_market_calendars) (2024.1)\n",
      "Requirement already satisfied: korean_lunar_calendar in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from exchange-calendars>=3.3->pandas_market_calendars) (0.3.1)\n",
      "Requirement already satisfied: six>=1.5 in /Users/marianamenchero/miniforge3/envs/nixtla/lib/python3.10/site-packages (from python-dateutil->pandas_market_calendars) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "#| hide\n",
    "!pip install -Uqq nixtla\n",
    "!pip install pandas_market_calendars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "from nixtla.utils import in_colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide \n",
    "IN_COLAB = in_colab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    from nixtla.utils import colab_badge\n",
    "    from dotenv import load_dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Irregular timestamps\n",
    "\n",
    "When working with time series data, it is important to specify its frequency correctly, as this can significantly impact forecasting results. TimeGPT is designed to automatically infer the frequency of your timestamps. For commonly used frequencies, such as hourly, daily, or monthly, TimeGPT reliably infers the frequency automatically, so no additional input is required. \n",
    "\n",
    "However, for irregular frequencies, where observations are not recorded at consistent or regular intervals, such as the days the U.S. stock market is open, it is necessary to specify the frequency directly. \n",
    "\n",
    "TimeGPT requires that your data does not contain missing values, as this is not currently supported. In other words, the irregularity of the data should stem from the nature of the recorded phenomenon, not from missing observations. If your data contains missing values, please refer to our [tutorial on missing dates](https://docs.nixtla.io/docs/tutorials-missing_values).\n",
    "\n",
    "In this tutorial, we will show you how to handle irregular and custom frequencies in TimeGPT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Nixtla/nixtla/blob/main/nbs/docs/capabilities/forecast/11_irregular_timestamps.ipynb)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "if not IN_COLAB:\n",
    "    load_dotenv()\n",
    "    colab_badge('docs/capabilities/forecast/11_irregular_timestamps')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import packages \n",
    "\n",
    "First, we import the required packages and initialize the Nixtla client. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pandas_market_calendars as mcal\n",
    "from nixtla import NixtlaClient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nixtla_client = NixtlaClient(\n",
    "    # defaults to os.environ.get(\"NIXTLA_API_KEY\")\n",
    "    api_key = 'my_api_key_provided_by_nixtla'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 👍 Use an Azure AI endpoint\n",
    "> \n",
    "> To use an Azure AI endpoint, remember to set also the `base_url` argument:\n",
    "> \n",
    "> `nixtla_client = NixtlaClient(base_url=\"you azure ai endpoint\", api_key=\"your api_key\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| hide\n",
    "if not IN_COLAB:\n",
    "    nixtla_client = NixtlaClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Handling regular frequencies  \n",
    "\n",
    "As discussed in the introduction, for time series data with regular frequencies, where observations are recorded at consistent intervals, TimeGPT can automatically infer the frequency of your timestamps if the input data is a **pandas DataFrame**. If you prefer not to rely on TimeGPT's automatic inference, you can set the `freq` parameter to a valid [pandas frequency string](https://pandas.pydata.org/docs/user_guide/timeseries.html#offset-aliases), such as `MS` for month-start frequency or `min` for minutely frequency.  \n",
    "\n",
    "When working with **Polars DataFrames**, you must specify the frequency explicitly by using a valid [polars offset](https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.offset_by.html), such as `1d` for daily frequency or `1h` for hourly frequency. \n",
    "\n",
    "Below is an example of how to specify the frequency for a Polars DataFrame. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Restricting input...\n",
      "INFO:nixtla.nixtla_client:Calling Forecast Endpoint...\n"
     ]
    }
   ],
   "source": [
    "import polars as pl \n",
    "\n",
    "url = 'https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/air_passengers.csv'\n",
    "\n",
    "polars_df = pl.read_csv(url, try_parse_dates=True)\n",
    "\n",
    "fcst_df = nixtla_client.forecast(\n",
    "    df=polars_df,\n",
    "    h=12, \n",
    "    freq='1mo', \n",
    "    time_col='timestamp', \n",
    "    target_col='value', \n",
    "    level=[80, 95]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 86988 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(polars_df, fcst_df, time_col='timestamp', target_col='value', level=[80,95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Handling irregular frequencies\n",
    "\n",
    "In this section, we will discuss cases where observations are not recorded at consistent intervals.\n",
    "\n",
    "### 3.1 Load data \n",
    "\n",
    "We will use the daily stock prices of Palantir Technologies (PLTR) from 2020 to 2023. The dataset includes data up to 2023-09-22, but for this tutorial, we will exclude any data before 2023-08-28. This allows us to show how a custom frequency can handle days when the stock market is closed, such as Labor Day in the U.S. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Dividends</th>\n",
       "      <th>Stock Splits</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-09-30</td>\n",
       "      <td>10.00</td>\n",
       "      <td>11.41</td>\n",
       "      <td>9.11</td>\n",
       "      <td>9.50</td>\n",
       "      <td>9.50</td>\n",
       "      <td>338584400</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-10-01</td>\n",
       "      <td>9.69</td>\n",
       "      <td>10.10</td>\n",
       "      <td>9.23</td>\n",
       "      <td>9.46</td>\n",
       "      <td>9.46</td>\n",
       "      <td>124297600</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-10-02</td>\n",
       "      <td>9.06</td>\n",
       "      <td>9.28</td>\n",
       "      <td>8.94</td>\n",
       "      <td>9.20</td>\n",
       "      <td>9.20</td>\n",
       "      <td>55018300</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-10-05</td>\n",
       "      <td>9.43</td>\n",
       "      <td>9.49</td>\n",
       "      <td>8.92</td>\n",
       "      <td>9.03</td>\n",
       "      <td>9.03</td>\n",
       "      <td>36316900</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-10-06</td>\n",
       "      <td>9.04</td>\n",
       "      <td>10.18</td>\n",
       "      <td>8.90</td>\n",
       "      <td>9.90</td>\n",
       "      <td>9.90</td>\n",
       "      <td>90864000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date   Open   High   Low  Close  Adj Close     Volume  Dividends  \\\n",
       "0 2020-09-30  10.00  11.41  9.11   9.50       9.50  338584400        0.0   \n",
       "1 2020-10-01   9.69  10.10  9.23   9.46       9.46  124297600        0.0   \n",
       "2 2020-10-02   9.06   9.28  8.94   9.20       9.20   55018300        0.0   \n",
       "3 2020-10-05   9.43   9.49  8.92   9.03       9.03   36316900        0.0   \n",
       "4 2020-10-06   9.04  10.18  8.90   9.90       9.90   90864000        0.0   \n",
       "\n",
       "   Stock Splits  \n",
       "0           0.0  \n",
       "1           0.0  \n",
       "2           0.0  \n",
       "3           0.0  \n",
       "4           0.0  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/openbb/pltr.csv'\n",
    "pltr_df = pd.read_csv(url, parse_dates=['date'])\n",
    "pltr_df = pltr_df.query('date < \"2023-08-28\"')\n",
    "pltr_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will forecast the **adjusted closing price**, which represents the stock's closing price adjusted for corporate actions such as stock splits, dividends, and rights offerings. Hence, we will exclude the other columns from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 82024 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltr_df = pltr_df[['date', 'Adj Close']]\n",
    "\n",
    "nixtla_client.plot(pltr_df, time_col = \"date\", target_col = \"Adj Close\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Define the frequency \n",
    "\n",
    "To define a custom frequency, we will first extract and sort the dates from the input data, ensuring they are in the correct datetime format. Next, we will use the [`pandas_market_calendars package`](https://pypi.org/project/pandas-market-calendars/), specifically the `get_calendar` method, to obtain the New York Stock Exchange (NYSE) calendar. Using this calendar, we can create a custom frequency that includes only the days the stock market is open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dates = pd.DatetimeIndex(sorted(pltr_df['date'].unique())) # sort all dates in the dataset\n",
    "\n",
    "nyse = mcal.get_calendar('NYSE') # New Yor Stock Exchange calendar "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the days the stock market is open need to include all the dates in the input data plus the forecast horizon. In this example, we will forecast 7 days ahead, so we need to make sure our trading days include the last date in the input data as well as the next 7 valid trading days.\n",
    "\n",
    "To avoid dealing with holidays or weekends during the forecast horizon, we will specify an end date well beyond the forecast horizon. For this example, we will use January 1, 2024, as a safe cutoff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trading_days = nyse.valid_days(start_date=dates.min(), end_date=\"2024-01-01\").tz_localize(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, with the list of trading days, we can identify the days the stock market is closed. These are all weekdays (Monday to Friday) within the range that are not trading days. Using this information, we can define a custom frequency that skips the stock market's closed days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_weekdays = pd.date_range(start=dates.min(), end=\"2024-01-01\", freq='B')\n",
    "\n",
    "closed_days = all_weekdays.difference(trading_days)\n",
    "\n",
    "custom_bday = pd.offsets.CustomBusinessDay(holidays=closed_days)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Forecast with TimeGPT\n",
    "\n",
    "With the custom frequency defined, we can now use the `forecast` method, specifying the `custom_bday` frequency in the `freq` argument. This will make the forecast respect the trading schedule of the stock market."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:nixtla.nixtla_client:Validating inputs...\n",
      "INFO:nixtla.nixtla_client:Preprocessing dataframes...\n",
      "INFO:nixtla.nixtla_client:Querying model metadata...\n",
      "INFO:nixtla.nixtla_client:Restricting input...\n",
      "INFO:nixtla.nixtla_client:Calling Forecast Endpoint...\n"
     ]
    }
   ],
   "source": [
    "fcst_pltr_df = nixtla_client.forecast(\n",
    "    df=pltr_df, \n",
    "    h=7, \n",
    "    freq=custom_bday,\n",
    "    time_col='date', \n",
    "    target_col='Adj Close',\n",
    "    level=[80, 95]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📘 Available models in Azure AI\n",
    ">\n",
    "> If you are using an Azure AI endpoint, please be sure to set `model=\"azureai\"`:\n",
    ">\n",
    "> `nixtla_client.forecast(..., model=\"azureai\")`\n",
    "> \n",
    "> For the public API, we support two models: `timegpt-1` and `timegpt-1-long-horizon`. \n",
    "> \n",
    "> By default, `timegpt-1` is used. Please see [this tutorial](https://docs.nixtla.io/docs/tutorials-long_horizon_forecasting) on how and when to use `timegpt-1-long-horizon`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 73424 characters]",
      "text/plain": [
       "<Figure size 1600x350 with 1 Axes>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nixtla_client.plot(pltr_df, fcst_pltr_df, time_col = \"date\", target_col = \"Adj Close\", level=[80, 95], max_insample_length = 180)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2023-08-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2023-08-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2023-08-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2023-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2023-09-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2023-09-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2023-09-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date\n",
       "0 2023-08-28\n",
       "1 2023-08-29\n",
       "2 2023-08-30\n",
       "3 2023-08-31\n",
       "4 2023-09-01\n",
       "5 2023-09-05\n",
       "6 2023-09-06"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcst_pltr_df[['date']].head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the forecast excludes 2023-09-04, which was a Monday when the stock market was closed for Labor Day in the United States.\n",
    "\n",
    "## 4. Summary \n",
    "Below are the key takeaways of this tutorial:\n",
    "\n",
    "- TimeGPT can reliably infer regular frequencies, but you can override this by setting the `freq` parameter to the corresponding pandas alias.\n",
    "\n",
    "- When working with polars data frames, you must always specify the frequency using the correct polars offset.\n",
    "\n",
    "- TimeGPT supports irregular frequencies and allows you to define a custom frequency, generating forecasts exclusively for the specified dates."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
