Forecasting errors

Contents

Here are the main metrics for measuring forecasting errors:

Scale-dependent measurement:

MAE, MDAE, MSE, RMSE

Percentage :

MAPE, sMAPE, MASE, sMdAPE

In relative terms:

MRAE, MdRAE, RelMAE
U-statistic

Scale-dependent measurements

We use the following terminology: if y1, …, yn represents a time series, then ŷi represents the ith predicted value, where i ≤ n. For i ≤ n, the ith error ei (aka residual) is then

Our goal is to find a forecast that minimizes errors. A number of measurements are commonly used to determine the accuracy of a forecast. These include Mean Absolute Error (MAE), Mean Square Error (MSE), and Root Mean Square Error (RMSE).

MdAE calculates the median absolute difference between two digital vectors.

real <- c(1.1, 1.9, 3.0, 4.4, 5.0, 5.6)
predicted <- c(0.9, 1.8, 2.5, 4.5, 5.0, 6.2)
mdae (actual, planned)

Percentage measurements

Some other metrics are Mean Absolute Percentage Error (MAPE), Mean Absolute Scale Error (MASE), and Symmetric Mean Absolute Percentage Error (SMAPE).

Median absolute scale error (MdASE). Taking the median instead of the mean of the absolute testing and training errors makes this metric more robust to outliers since the median tends to be a more robust measure of central tendency in the presence of outliers.

Like MASE and other scaled performance measures, this scale-free measure can be used to compare forecasting methods within a single series or between series.

Just like MASE, this metric is well suited for series with intermittent demand because it will not give infinite or indefinite values unless the training data is a flat time series. In this case the function returns a large value instead of inf.

Theil U Statistics

Finally, there is Theil's U statistic, which is calculated by the formula

If U < 1, the forecasting technique is better than guessing. If U = 1, then the forecasting technique is equivalent to an estimate. If U > 1, then the forecasting technique is worse than guessing.

In fact, U is also called Theil's U2 statistic. There is also a less often used U1 stat

U1 takes values between 0 and 1, with values closer to 0 representing greater forecast accuracy.

Relative measurements

And here are the relative measurements:

Here the prediction is to be compared to the so-called reference prediction B.