Forecast evaluation for data scientists: common pitfalls and best practices

  title={Forecast evaluation for data scientists: common pitfalls and best practices},
  author={Hansika Hewamalage and Klaus Ackermann and C. Bergmeir},
  journal={Data Mining and Knowledge Discovery},
  pages={788 - 832}
Recent trends in the Machine Learning (ML) and in particular Deep Learning (DL) domains have demonstrated that with the availability of massive amounts of time series, ML and DL techniques are competitive in time series forecasting. Nevertheless, the different forms of non-stationarities associated with time series challenge the capabilities of data-driven ML models. Furthermore, due to the domain of forecasting being fostered mainly by statisticians and econometricians over the years, the… 

FRANS: Automatic Feature Extraction for Time Series Forecasting

This research introduces a notion of static and dynamic features, which then enables it to develop its autonomous Feature Retrieving Autoregressive Network for Static features (FRANS) that does not require domain knowledge.

Causal Effect Estimation with Global Probabilistic Forecasting: A Case Study of the Impact of Covid-19 Lockdowns on Energy Demand

—The electricity industry is heavily implementing intelligent control systems to improve reliability, availability, security, and efficiency. This implementation needs technological advancements, the

Hierarchical learning, forecasting coherent spatio-temporal individual and aggregated building loads

A novel multi-dimensional hierarchical forecasting method built upon structurally-informed machine-learning regressors and established hierarchical reconciliation taxonomy is proposed, providing traditionally disjointed hierarchical forecasting methods a fertile route toward a novel generation of forecasting regressors.

Forecasting the Monash Microgrid for the IEEE-CIS Technical Challenge

Effective operation of a microgrid depends critically on accurate forecasting of its components. Recently, internet forecasting competitions have been used to determine the best methods for energy



Evaluating time series forecasting models: an empirical study on performance estimation methods

This paper presents an extensive empirical study which compares different performance estimation methods for time series forecasting tasks, including variants of cross-validation, out-of-sample (holdout), and prequential approaches.

DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks

N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting

It is proved that the hierarchical interpolation technique can approximate arbitrarily long horizons in the presence of smoothness, and N-HiTS, a model which addresses both challenges by incor-porating novel hierarchical interpolations and multi-rate data sampling techniques is introduced.

A Backtesting Protocol in the Era of Machine Learning

Machine learning offers a set of powerful tools that holds considerable promise for investment management. As with most quantitative applications in finance, the danger of misapplying these

A note on the validity of cross-validation for evaluating autoregressive time series prediction

On the use of cross-validation for time series predictor evaluation

AdaRNN: Adaptive Learning and Forecasting of Time Series

This paper proposes Adaptive RNNs (AdaRNN) to tackle the TCS problem by building an adaptive model that generalizes well on the unseen test data and proposes Temporal Distribution Characterization to better characterize the distribution information in the TS.

Experimenting with prequential variations for data stream learning evaluation

This article empirically evaluates the prequential methodology considering its three common strategies used to update the prediction model, namely, Basic Window, Sliding Window, and Fading Factors to identify which of these variations is the most accurate for the experimental evaluation of the past results in scenarios where concept drifts occur.