# default quantiles taken from https://github.com/awslabs/gluonts/blob/08ab434b1e0946c21010ddbb4248288f8f043599/src/gluonts/evaluation/_base.py#L90C5-L90C68
# Forecasting at Scale: One Billion (1e9) Time Series with TimeGPT ⚡📈
Imagine you're tasked with forecasting for **one billion unique time series**—ranging from retail sales across thousands of stores to sensor data from millions of IoT devices. It's a monumental challenge, requiring not just statistical modeling but also cutting-edge tools to handle the scale and complexity of the data.
This project is a blueprint for scaling such a task, utilizing **Nixtla's foundation models for time series forecasting** and orchestrating the process efficiently using Python and AWS S3. Here's how you can tackle this kind of project.
## The Challenge 🎯
The goal is simple: forecast the future for **one billion different time series**, but the constraints are anything but simple. How do you handle the storage of this data? 🗄️ How do you parallelize the computation efficiently? 💻 And finally, how do you produce results quickly enough to be useful in decision-making? ⏳
### Enter Foundation Models for Time Series 🚀
**Nixtla** offers **TimeGPT** through an API that leverages foundation models capable of handling large-scale forecasting problems. These models are designed for flexibility and speed 🏎️, making them ideal for scenarios where you're dealing with an enormous volume of data and need results at a high cadence. ⚡
## Results 📊
| 📈 **Number of Series** | Number of Processes | ⏳ **CPU Time (hours)** |
-**`bucket`**: The S3 bucket where the data is stored.
-**`prefix`**: The path inside the S3 bucket where the input and output data is stored.
-**`n_partitions`**: The number of partitions to break the task into.
-**`series_per_partition`**: The number of time series in each partition.
-**`n_jobs`**: The number of processes to run in parallel.
### What Happens Behind the Scenes 🔍
The code will:
1. Check if the forecast for each partition has already been generated. ✅
2. Generate new time series data for each partition. 🧬
3. Use Nixtla’s API to compute forecasts for each partition. 🔮
4. Save the results and the time taken to S3. 💾
## Scaling to Billions 🚀
This approach is designed to **scale**—whether you’re forecasting for **one million** or **one billion** series. By partitioning the data, processing it in parallel 🧠, and leveraging foundation models like those provided by Nixtla, you can handle even the most massive forecasting tasks efficiently. ⚙️
### Final Thoughts 💡
Forecasting at scale is no easy feat, but with the right tools, it’s entirely achievable. This project demonstrates how modern time series forecasting techniques can be applied to massive datasets in an efficient, scalable way. By leveraging AWS infrastructure, foundation models, and clever parallel processing, you can forecast the future for billions of unique data series—**unlocking insights** that can power decision-making at an unprecedented scale. 🌍✨
# TimeGPT vs Prophet: Time Series Forecasting Benchmark
## Overview
This repository offers a detailed benchmarking framework for comparing the performance of TimeGPT against Prophet and StatsForecast in time series forecasting. We provide datasets with over 300,000 series across various frequencies, including daily, weekly, 10-minute, and hourly intervals. Users can also incorporate their own datasets for a more personalized analysis. **TimeGPT was not trained on this datasets.**
## Results
The results show that, on average, TimeGPT zero-shot forecasts are **58% more accurate and 92% faster than Prophet's**. The improvements are consistent across 8 frequencies, ranging from minute to quarterly data.
## Notes
- Results were generated using a VM with 96 cores and 196 GB of RAM.
- Prophet and StatsForecast was executed in paralell.
- TimeGPT uses the AzureML endpoint.
- Since the AzureML endpoint does not support GPU and scalable requests, the results can improve.
## Repository Structure
-`/data`: Parquet files with time series data.
-`/src`: Source code for running benchmarks and experiments.
-`/data/results`: Outputs and analysis from benchmark runs.
## Data Structure
Datasets should adhere to this structure:
-**unique_id**: Identifier for each series.
-**ds**: Timestamp of observation.
-**y**: Target variable for forecasting.
-**frequency**: Description of data frequency (e.g., 'Daily').
-**pandas_frequency**: Pandas frequency string (e.g., 'D').
-**h**: Forecasting horizon. (The last `h` periods of each series will be used as test.)
-**seasonality**: Seasonality of the series (e.g., 7 for daily).
## Running Experiments
### Makefile
The repository includes a Makefile to streamline the process of running experiments. The key commands are:
1.**evaluate**: Runs the evaluation for a specified method (`timegpt`, `prophet`, or `statsforecast`).
2.**summarize_results**: Summarizes the results from the evaluation.
### Evaluation Flow
1.**Run Evaluation**: Use `make evaluate method=<method_name>` where `<method_name>` is either `timegpt`, `prophet`, or `statsforecast`. The script filters out files containing specific strings (like 'catalogue') and runs the experiment for each `.parquet` file in the `/data` directory. The results will be written in `/data/results`.
2.**Summarize Results**: After running evaluations for each method, execute `make summarize_results` to aggregate and summarize the results, which will be written in this `README.md` file.
## Getting Started
1.**Prepare Data**: Ensure your Parquet files are in `/data`. If you want access to the original datasets, please write to `support@nixtla.io` with your use case.
2.**Create conda environment**: Run `conda env create -f environment.yml` and activate the environment using `conda activate timegpt-benchmark`.
3.**Run Benchmarks**: Use the Makefile commands to run evaluations and summarize results.
# Salesforce's Moirai performs great in hourly data and is much faster than Chronos but is still up to 33% less accurate and less efficient than statistical models when considering monthly, weekly, and yearly data
We present a comprehensive, reproducible evaluation demonstrating that a Statistical Ensemble—comprising AutoARIMA, AutoETS, AutoCES, and DynamicOptimizedTheta—substantially surpasses [Salesforce Moirai](https://github.com/SalesforceAIResearch/uni2ts), a foundational model for time series forecasting with over 311 million parameters. The **Statistical Ensemble achieves 33%, 33%, and 15% superior performance in CRPS, MASE, and SMAPE metrics, respectively**, across benchmark datasets including M1, M3, M4, and Tourism. A **Simple Seasonal Naive achieves 17% and 0.5%, superior performance in MASE, and SMAPE metrics, respectively. However, Morai is 25% more accurate than a seasonal naive in terms of CRPS**. Benchmark datasets include M1, M3, M4, and Tourism.
These datasets cover more than **100,000 unique time series**, offering a robust comparison of the models. Efficiency-wise, **Moirai is 3.5x faster than the Statistical Ensemble but 160x slower than a seasonal naive forecast**, marking a trade-off between speed and accuracy in different forecasting frequencies.
# Introduction
Following our recent [benchmark demonstrating Amazon Chronos's lesser accuracy and slower speed compared to classical statistical models](https://github.com/Nixtla/nixtla/tree/main/experiments/amazon-chronos), the community sought a similar analysis for Moirai. We commend the Salesforce AI team for releasing the first fully open-source foundational time series model, complete with weights, data, and code. Morai's accuracy shines with hourly data, a noteworthy achievement we're eager to highlight. Our acknowledgment extends to Salesforce for recognizing our prior contributions to this research field.
Foundational models like Salesforce's Moirai signify a notable advance in time series forecasting, leveraging deep learning and extensive datasets for pre-training to enhance predictions. Despite Moirai's impressive parameter count (311 million) and scope, our findings suggest that traditional forecasting methods grouped into a Statistical Ensemble often outperform in accuracy. This benchmark continues our exploration of statistical versus deep learning models in forecasting.
In our assessment, Salesforece's Moirai shows a more promising path than Amazon Chronos in handling hourly data, hinting at the potential to surpass classical statistical methods eventually.
## Empirical Evaluation
Expanding upon our prior work, this study evaluates over 100,000 unique time series from the M1, M3, M4, and Tourism datasets across various frequencies. Our analysis also benchmarks against the Seasonal Naive model, a staple in traditional forecasting methods.
## Results
The **Statistical Ensemble achieves 33%, 33%, and 15% superior performance in CRPS, MASE, and SMAPE metrics, respectively**, across benchmark datasets including M1, M3, M4, and Tourism. A **Simple Seasonal Naive achieves 17% and 0.5%, superior performance in MASE, and SMAPE metrics, respectively. However, Morai is 25% more accurate than a seasonal naive in terms of CRPS**.
Efficiency-wise, **Moirai is 3.5x faster than the Statistical Ensemble but 160x slower than a seasonal naive forecast**, marking a trade-off between speed and accuracy in different forecasting frequencies.
It is critical to highlight that Morai may possess an unfair advantage over the statistical ensemble due to its training methodology. Specifically, Morai was trained using all the datasets that are currently being used for evaluation. In contrast, the statistical ensemble was not exposed to the test dataset during its training phase.
The complete code to replicate all results is available at [GitHub](https://github.com/Nixtla/nixtla/tree/main/experiments/salesforce-moirai). This study underscores statistical models' continued relevance and superiority in specific scenarios, challenging the assumption that foundational deep-learning models are always the best solution for time series forecasting.
This revision integrates the comparative performance of the Statistical Ensemble and Salesforce's Moirai, highlighting key findings from your data. Please ensure to replace the placeholder for the new table image with an actual image link or embed the table directly if the platform supports LaTeX rendering.
## Reproducibility
To ensure the reproducibility of our findings, the Statistical Ensemble experiments were conducted on an AWS c5a.24xlarge instance, equipped with 96 vCPUs and 192 GiB of RAM. In contrast, the experiments for Salesforce Moirai were carried out on an AWS g5.4xlarge GPU instance, which includes 16 vCPUs, 64 GiB of RAM, and an NVIDIA A10G Tensor Core GPU with 24 GiB. All necessary code and detailed instructions for reproducing the experiments are available in this directory.
