Not So Great Expectations: Part I

We recently wrote about Andrei Shleifer’s new research incorporating CFO sentiment survey data to identify expectation errors common among both CFOs and Wall Street analysts. Shleifer’s paper used quarterly polling data from CFOs in the United States, sent out to ~200 firms each quarter dating back to 1998 from the Duke/CFO Magazine Business Outlook Survey. But perhaps the best data set available to understand actual sentiment expectations of CFOs is a rarely studied but vast treasure trove of survey data from the Bank of Japan. Since 1974, the Bank of Japan has conducted its extensive quarterly TANKAN survey of 10,000 Japanese companies with a 99% response rate. This is perhaps the most comprehensive and long-running survey of company expectations in the world.

And we have spent the last two months studying this massive data set to understand the structure of Japanese corporate expectations, how they err and how to use gauges of their sentiment as equity investors. In this piece, we will describe what we learned about expectations and their errors and next week we will look at how to invest based on the data.

It may be tempting to dismiss these findings as a peculiarity of the Japanese environment and Japanese investor psychology. While that logic might make sense for the company-level forecast errors, it is problematic when applied to market movements. Since the 1990s, foreign investors have been the single largest plurality of owners of Japanese stocks at about one third of the market today compared to about 15% for individual Japanese investors. When efficient markets theorist and Nobel Laureate Eugene Fama shocked a journalist from the New Yorker by arguing that the government was largely responsible for the subprime mortgage crisis, the journalist responded, “But Fannie and Freddie’s purchases of subprime mortgages were pretty small compared to the market as a whole, perhaps twenty or thirty per cent.” Fama, laughed and said “Well, what does it take?”

Below are the average business confidence forecasts of all surveyed Japanese firms for each quarter since 1974. We show simply whether the directionality of their forecast in business confidence was right or wrong when compared to the actual business confidence survey results three months later. Please note the disproportionate grouping of outcomes in the “WRONG” quadrants.

Figure 1: Average Business Confidence Forecast vs. Average Actual Confidence Next Quarter (1)

61% of the time (both “WRONG” quadrants above), surveyed businesses were wrong in their forecasts of whether business conditions would improve or deteriorate next quarter relative to this quarter. Only 39% of the time (“RIGHT” quadrants above) were they correct in the directionality of their forecasts. Overall, a coin flip would have been more accurate at forecasting the change in sentiment of the market just three months out.

In fact, we tested whether a simple prediction that business confidence would be about what it was yesterday (“no change” assumption) would have been a more accurate forecast methodology than the directional changes companies were betting on.

Figure 2: Total Forecast Error Using Company Forecasts vs. “No Change” Assumption (1)

Note: Company Forecast Error is the difference between the percentage of respondents who viewed business conditions as favorable minus those who viewed conditions as unfavorable at time “t” versus what the companies had forecast that number would be next quarter at time “t-1”. Total Error is the sum of all the directional errors for all 178 quarters. Total Absolute Error is the sum of the absolute value of the errors.

The company estimates had 42x aggregate forecast error over the last 178 quarters than just assuming business conditions were unchanged. To test if expertise gave any additional advantage, we broke out the results by industry and found that 69% of the 44 industries were worse than a coin flip at correctly guessing whether business conditions would improve or deteriorate just one quarter out. Though clearly the most knowledgeable experts about their own industries, their forecasts were systematically off.

Okay, well, maybe if companies couldn’t predict changes in business confidence for the next quarter, perhaps their short-term directional bets give us some insight into longer-term trends (i.e., the Nostradamus “called it too early” syndrome). Below is a comparison of how the short-term forecasts scored relative to actual business confidence three months, one year, and three years into the future.

Figure 3: Directional Forecast Accuracy vs. Actual Business Confidence 3, 12, and 36 Months Out (1)

The forecast error declines slightly with the randomness and uncertainty that longer horizons bring, but they still were not able to beat a coin flip, with 51% directionally wrong forecasts three years out. There was no advantage to following the business sentiment forecast surveys on any horizon.

Interestingly, in Japan the most common error was to presume that business conditions would deteriorate relative to today, when in fact they went on to improve (top-left quadrant of Figure 1). On average, respondents underestimated by 2% per quarter. Indeed, if we break up the data from 178 quarters of forecasts with 44 industries, we can visualize the pessimistic forecast bias in the 6,175 observations over 50 years:

Figure 4: Histogram of Human Forecasts of Business Confidence and Actual Outcomes (1974–2018; n=6175) (1)

Note: Tail risk was perfectly estimated (not visible on left bar) in frequency with 78 forecasts of 78 drops below 25% in business confidence. The skew in systematically pessimistic forecasting does not appear to be compensation for extreme bad outcomes.

The bright red portion to the left of the bell curve of reality (blue) represents the excess quantum of negative forecasts accumulated over time. Reality (the blue curve) had an average and median change of about 0% while the human forecasts had an average and median of -2%. This explains why, over time, a “no change” forecast methodology would have been systematically more accurate than the human forecasts.

This systematic pessimism has been a long-running feature of the Japanese market. Below, we show the long-term chart of business confidence forecast error.

Figure 5: Forecast Error over Time Using Company Forecasts (1)

There have been three bouts of particularly vehement pessimism about the future: the infamously depressing last few years of the Jimmy Carter era; the mid to late 80s, when Reagan was threatening and imposing tariffs on Japan’s autos, semiconductors, electronics and appliance industries; and finally, the last decade following the great recession.

In his other work, Inefficient Markets, Shleifer argued, “people do not deviate from rationality randomly, but rather most deviate in the same way … investor sentiment reflects the common judgement errors made by a substantial number of investors, rather than uncorrelated random mistakes.” The data suggest that Japanese CFOs are particularly pessimistic deviants.

The next time you read a market news headline that goes something like “Business Confidence Sours, Etc.,” we hope you will take it with a grain of salt in light of these base rates. Markets frequently trade down on this sort of news in the short term, but the historical odds are that the expectations are overblown.

As investors, what can we take away from this? We will dive deeper into that question next week in Part II of this series.
 
(1) Source: Bank of Japan TANKAN surveys, Verdad Analysis