Schedule-free optimization of the transformers-based time series forecasting model

Abstract

The task of time series forecasting is important for many scientific, technical, and applied fields, such as finance, economics, meteorology, medicine, transportation, and telecommunications. Existing methods, such as autoregressive models and moving average models, have their limitations, especially when working with non-stationary and seasonal data. In this work, the basic architecture of transformers was modified to solve time series forecasting problems. Additionally, state-of-the-art optimizers were investigated and experimentally compared, including AdamW, stochastic gradient descent (SGD), and new methods such as schedule-free SGD and schedule-free AdamW, to improve forecasting accuracy and the efficiency of the training procedure for the transformer architecture. Modeling was conducted on meteorological data that included seasonal time series. The accuracy evaluation of the optimization methods studied in this work was performed using a range of different performance indicators. The results showed that the new optimization methods significantly improve forecasting accuracy compared to the use of traditional optimizers.