diff --git a/experiments/foundation-time-series-arena/README.md b/experiments/foundation-time-series-arena/README.md index 4ce876a0..73c70c6b 100644 --- a/experiments/foundation-time-series-arena/README.md +++ b/experiments/foundation-time-series-arena/README.md @@ -2,6 +2,7 @@ > TL;DR: Foundation models for time series outperform alternatives and are ready to be tested in production. TimeGPT-1 is (so far) the most accurate and fastest model but TimesFM from Google comes very close. Some models are still outperformed by classical alternatives. +Note: The Amazon team responded to the original benchmark with this [PR](https://github.com/Nixtla/nixtla/pull/382) that shows, according to them, that by changing some parameters, Chronos is significantly faster and more accurate. We are currently reviewing the PR. # Introduction @@ -20,6 +21,8 @@ We at Nixtla have provided some [early success stories](https://techcommunity.mi However, the field [is still divided](https://news.ycombinator.com/item?id=39235983) on how all the different foundation models compare against each other. In the spirit of collaboration, we are starting a new project, `xiuhmolpilli`, in honor of how ancient civilizations celebrated the end of cycles, to build a benchmark to compare all the different foundation models for time series data in a large scale dataset and against classical, ML and Deep Learning Models. + + # Empirical Evaluation This study considers **over 30,000 unique time series** from the Monash Repository, M-Competitions, Wikipedia page views, among others, spanning various time series frequencies: Monthly, Weekly, Daily, and Hourly. Our evaluation compares five foundation models for time series data in terms of accuracy and inference times. We have also included comparisons to a large battery of statistical, machine learning, and deep-learning models, to provide a benchmark against traditional forecasting methods.