forecast performance and data mining *forecast is the ultimate challenge for any econometric model....

FORECAST PERFORMANCE AND DATA MINING *Forecast is the ultimate challenge for any econometric model.*Any model that fails to predict is useless. Exceptions?*History may not repeat itself, but it rhythms.

Remarks on econometric forecast: (1)Forecast and decision- cost of forecast (forecast is

irrelevant per se) Forecast without properly defined cost function means nothing.(2)Forecast standard deviation please…(3)Statistical measures for prediction accuracy can be problematic in the real world.

(4)Out-of-sample monitoring Is yesterday’s model capable of explaining

today’s data?(5)Correcting data mining bias in a “historically adequate” model. Predictions based on historically adequate

model resulting from extensive specification search are subject to data mining bias.

Data MiningA general modeling process that applies a sequence of statistical and non-statistical tools to a data set in an attempt to obtain a final model.

References:Ye, J (1998) “On Measuring and Correcting the Effects of Data Mining and Model Selection.” JASA 93 p120-131White, H (2000): “A Reality Check for Data Snooping.” Econometrica

Simulation Results Cost of Data MiningGeneralized degree of freedom (GDF) is defined as the sum of sensitivity of each fitted value to perturbations in the corresponding value.

Reality CheckWhenever a good forecasting model is obtained by an extensive specification search, there is always the danger that the observed good performance results not from actual forecasting ability, but instead of luck. Stock Investment News letter Simulation using Stationary bootstraps