On the Comparison of Interval Forecasts
We explore interval forecast comparison when the nominal conﬁdence level is speciﬁed, but the quantiles on which intervals are based are not speciﬁed. It turns out that the problem is diﬃcult, and perhaps unsolvable. We ﬁrst consider a situation where intervals meet the Christoﬀersen conditions (in particular, where they are correctly calibrated), in which case the common prescription, which we rationalize and explore, is to prefer the interval of shortest length. We then allow for mis-calibrated intervals, in which case there is a calibration-length tradeoﬀ. We propose two natural conditions that interval forecast loss functions should meet in such environments, and we show that a variety of popular approaches to interval forecast comparison fail them. Our negative results strengthen the case for abandoning interval forecasts in favor of density forecasts: Density forecasts not only provide richer information, but also can be readily compared using known proper scoring rules like the log predictive score, whereas interval forecasts cannot.