This paper stems from my general interest in Murphy's Law ("If something can go wrong, it will") , although of course I didn't mention that in the original paper - Nature is supposed to be pretty serious, after all ! But I'd claim that although the example may seem a bit frivolous by the standards of Nature, it still contains an important lesson: that neglect of the "base rate" of a phenomenon being forecast can have dramatic effects on the usefulness of even "accurate" forecasts. Suppose one is planning a country walk, and wants to know what the weather will be like. It seems obvious that one should seek out the most accurate forecast available, and respond accordingly. It turns out, however, that the base-rate effect can make putting one's faith in weather forecasts a sub-optimal strategy for deciding what to do.
The UK Meteorological Office 24-hr forecast of rain currently achieve around 83 per cent accuracy, while the probability of rain on the hourly timescale relevant to walks is around 0.08 (ref. 4). This leads to the following table for the various outcomes of forecast and weather over 1000 1-hour walks:
Rain No rain Sum
Forecast of rain 66 156 222
Forecast of no 14 764 778
rain
Sum 80 920 1000
The table reveals the impact of the base-rate error in the interpretation of forecasts of rain. With forecast accuracies of 83 per cent, one might expect that a forecast of rain during the walk would prove correct 83 per cent of the time. However, the hourly base-rate of rain in the UK is so low that forecasts of rain are over twice as likely to be wrong as right: from the table, the probability of rain, given a forecast of rain - i.e. Pr(Rain | Forecast of rain) - is 66/222 = 0.30, while Pr(No rain | forecast) = 156/222 = 0.70
This suggests that those who ignore forecasts and simply assume rain won't fall during their walk will fare better than those who abide by Met Office forecasts. A decision-theoretic analysis shows that this is indeed the case. Let R, F and W represent the events of rain falling during the walk, rain being forecast, and going on the walk, respectively. The forecasting accuracy is captured by Pr(F |R) = A and Pr(F |~R) = B,while the responses to the forecasts are represented by Pr(W | F) = m and Pr(W | ~F)= n; one can then show that
Pr(R & W) = Pr(R)[Am + (1 - A)n] etc. (1)
Let Ltot represent the loss function, comprising the losses resulting from the outcomes of the various decisions:
Ltot = L00 Pr(R & W) + L11 Pr(~R & ~W) + L10Pr(~R & W) + L01Pr(R & ~W) (2)
Optimal strategies minimise Ltot. Substituting from (1), and keeping terms in m and n,
Ltot ~ m{Pr(R)AK - Pr(~R)B} + n{Pr(R)(1 - A)K - Pr(~R)(1 - B)} (4)
where K = (L00 - L01)/ (L11 - L10) (4)
represents the relative losses resulting from the outcomes in (2). With A = 0.83, B = 0.17, Pr(R) = 0.08, we find that basing our decision on Met Office forecasts (m = 0, n = 1) gives Ltot ~ (K - 56)/74, while blithely ignoring forecasts of rain (m = n = 1) gives Ltot ~ (K - 12)/13. Thus unless one is rather worried about getting wet (K > 2), the base-rate effect makes cheerful disregard of forecasts of rain the optimal strategy.
Similar reasoning also reveals that, contrary to popular belief, always carrying an umbrella is a sub-optimal strategy unless one is morbidly afraid of getting wet (K > 56). Indeed, unless K > 12 the base-rate effect makes even insouciant optimism a better strategy.
References
3. Yeatman, A. Meteorological Office Personal Communication 1996
4. Chandler T.J., Gregory, S. The Climate of the British Isles (Longman, London, 1976). Table 6.1 p 136.