About


The Atlanta Fed's macroblog provides commentary and analysis on economic topics including monetary policy, macroeconomic developments, inflation, labor economics, and financial issues.

Authors for macroblog are Dave Altig, John Robertson, and other Atlanta Fed economists and researchers.


« Does Lower Pay Mean Smaller Raises? | Main | The Impact of Extraordinary Policy on Interest and Foreign Exchange Rates »

December 05, 2016


Using Judgment in Forecasting: Does It Matter?

​​

Many professional forecasters use statistical models when making their near-term projections for real gross domestic product (GDP) growth. A 2013 special survey on the forecasting methods of the Survey of Professional Forecasters  found that 18 out of 21 respondents featured a statistical model prominently in their current-quarter economic projections. Nevertheless, there is fairly compelling evidence that many professional forecasters incorporate judgment in their forecasts of the first estimate of real GDP growth for a quarter—even when much of the source data used to construct the GDP estimate are available.

In the October 2016 Wall Street Journal Economic Forecasting Survey (WSJ), the most common panelist projection for annualized third-quarter real GDP growth was 2.5 percent, and the second most common one was 3.0 percent. The first digit after the decimal point, or tenths digit, of these two numbers are "5" and "0." Of the 58 individual forecasts of third-quarter growth in the survey, 21 had a tenths digit of "0" or "5," a total that is almost twice as large as we would expect if all tenths digits were equally likely to be submitted.

This pattern isn't unique to the most recent quarter's GDP forecast. The following chart shows the historical frequency of the tenths digit in past WSJ surveys for first estimates of real GDP growth over the period from the first quarter of 2003 to the third quarter of 2016, made about three weeks before the release.

Almost 40 percent of these 2,390 forecasts have a tenths digit of "0" or "5." In contrast, the historical distribution of published first estimates of real GDP growth from the fourth quarter of 1991 to the third quarter of 2016 and real gross national product (the most common measure of U.S. production in an earlier era) growth from the third quarter of 1965 to the third quarter of 1991 has a tenths digit of either "0" or "5" only 18 percent of the time. The historical Atlanta Fed's GDPNow forecasts have a "0" or a "5" tenths digit only 15 percent of the time.

More formally, one easily can reject the hypothesis at the 1 percent significance level that the tenths digit of the WSJ panelist forecasts are either uniformly distributed or follow the Benford distribution for tenths digits after rounding to the nearest tenth (see this paper by economists Stefan Gunnel and Karl-Heinz Todter, who found similar relative frequencies of "0s" and "5s" in professional forecasts of German GDP growth and consumer price index inflation).

If we assume that near-term GDP growth forecasts with a tenths digit of "0" or "5" typically involve more judgment than forecasts with another tenths digit, a natural question is whether these more judgmental forecasts are less accurate than others. Of the 2,390 WSJ growth forecasts mentioned above, the ones with a tenths digit of "0" or "5" (after rounding to the nearest tenth) had an average error of 0.786 percentage points without regard to sign, and the others had an average error of 0.743 percentage points. These accuracy metrics are not statistically different at even the 10 percent significance level. Moreover, because of the panel nature of WSJ forecasts, we can measure how often a forecaster has a tenths digit of "0" or "5" (after rounding). Of the 44 panelists who submitted at least 30 three-week-ahead GDP forecasts during the period of the first quarter of 2003 through the third quarter of 2016, the correlation of the panelists "0" or "5" tenth digit frequency and their average error without regard to sign is only 0.13 and not significantly different from 0.

Although at least some professional forecasters appear to make judgmental adjustments to their near-term GDP projections, the evidence presented here does not suggest it comes, on average, at the cost of accuracy.

December 5, 2016 in Forecasts , GDP | Permalink

Comments

In the 70s when macro models first became commercially available and widely used, an economist at the Boston Fed wrote an annual article comparing the forecasting track records of models run without any judgemental adjustments (e.g. Ray Fair), those with adjustments (e.g. Chase, Wharton, DRI), and purely judgemental. His work consistently showed the combination of statistical modeling with judgemental adjustments produced the most accurate forecasts. Looks like things have not changed much.

Posted by: Douglas Lee | December 08, 2016 at 07:39 AM

Post a comment

Comments are moderated and will not appear until the moderator has approved them.

If you have a TypeKey or TypePad account, please Sign in

Google Search



Recent Posts


March 2017


Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  

Archives


Categories


Powered by TypePad