# Nixtla TimeGPT vs. Azure AutoML: A Comprehensive Performance Analysis This experiment evaluates the performance of **Nixtla TimeGPT's zero-shot inference** against **Microsoft's Azure AutoML** in the domain of time series forecasting. Our analysis shows that TimeGPT **surpasses Azure AutoML by 12%, 12%, and 10% in MAE, RMSE, and MASE metrics** and has **300x improvement in computational efficiency**. This evaluation spanned over 3,000 distinct time series across various data frequencies, with considerations for Azure AutoML's cost constraints. # Introduction [Azure AutoML](https://learn.microsoft.com/en-us/azure/machine-learning/concept-automl-forecasting-methods?view=azureml-api-2), a product of Microsoft, offers a robust automated machine-learning solution that caters to a wide array of predictive tasks, including time series forecasting. TimeGPT is a foundational model for time series forecasting that can be accessed [through an API](https://docs.nixtla.io/). While Azure AutoML is known for its adaptability and ease of use, our findings reveal that TimeGPT offers superior accuracy and efficiency, especially in the context of time series data. ## Empirical Evaluation Our study involved a detailed comparison of both models across various datasets, including Hourly, Daily, Weekly, and Monthly data frequencies. The datasets were chosen from the test set of the [TimeGPT-1 paper](https://arxiv.org/abs/2310.03589), ensuring a diverse set of time series for evaluation. The selection process was designed to manage computational complexity and adhere to Azure AutoML's dataset size requirements, with a cap of 3,000 observations to maintain cost-effectiveness. ## Results The following table shows the main findings of our analysis, presenting a comparison of performance metrics (MASE, MAE, RMSE) and computational time (in seconds) across different datasets. The best results are highlighted in **bold** for clarity. image ## Reproducibility All experiments were conducted in controlled environments to uphold the integrity and reproducibility of our results. TimeGPT evaluations were performed using a 2020 MacBook Air with an M1 chip, ensuring accessibility and practicality. In contrast, Azure AutoML experiments were carried out on a cluster of 11 STANDARD_DS5_V2 virtual machines equipped with substantial computational resources to showcase its scalability and power. ### Instructions 1. Configure Azure AutoML according to the official Microsoft documentation. 2. Set the environment variables in a `.env` file using `.env.example` as example. 3. Set up a conda environment using: ```bash mamba create -n azure-automl-fcst python=3.10 conda activate azure-automl-fcst pip install uv uv pip install -r requirements.txt ``` 4. Download the data using ```python python -m src.utils.download_data ``` If you're interested in replicating the results, write us at `support@nixtla.io` to give you access to the data. 5. Filter the datasets to prevent AzureML from crashing ``` make filter_data ``` 6. Run the forecasting tasks for TimeGPT, SeasonalNaive, and AzureAutoML using the following: ``` make run_methods ``` Notice that AzureAutoML will send the job to the predefined cluster. 7. Retrieve AzureAutoML forecasts once they are ready: ``` make download_automl_forecasts ``` 8. Run evaluation ``` make evaluate_experiments ``` ### References - [TimeGPT 1](https://arxiv.org/abs/2310.03589) - [StatsForecast](https://github.com/Nixtla/statsforecast/) - [Distributed AzureAutoML for forecasting](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline/automl-forecasting-demand-many-models-in-pipeline.ipynb)