Home » Behind the Headlines » Measuring Forecasting Methodology & Accuracy

Measuring Forecasting Methodology & Accuracy

1-30-21 How can we tell if the adjusted starts forecasting method produces reliable results?

This plot of predicted spending from the starts cash flow model compared to actual spending is a check on this analytical modeling method. It shows a comparison of the cash flows predicted from all construction starts vs actual spending. If the forecast plot is accurate, then actual spending should move in the same direction, at the same slope. While we sometimes see lag in the plots movement, over time, the cash flow model of new starts does a good job of predicting where spending is headed.

In this first plot, prepared back in Nov, based on October data, note the divergence of residential in Jun-Jul-Aug 2020. Actual residential spending finished much higher than predicted. In 3 months, the actual spending pushed 15% higher than starts predicted. Tracking actual spending vs predicted, several of these months varied by more than 4 standard deviations, an unusual occurrence that has occurred less than 2% of the time in past 240 months. All of the other instances occurred during the Great Recession. This variance was suspected to indicate initially adjusting starts data to forecast too great an amount delayed or canceled or too slow a return of delayed projects to full workload.

The nonresidential buildings plots (and the residential plot prior to 2020) are remarkably close, providing an indication the method of analysis employed, cash flow of all construction starts to get spending forecast, is reasonably accurate.

Settings in the pandemic forecast model resulted in the residential divergence. First, projects delayed were predicted to take six to eight months to come fully back up to production. But residential project spending was fully back to prior levels by August, within 3 months from the May bottom. About 60% of the return to prior spending was supported by growth in residential renovations. The rapid growth in spending is represented by the steep recovery in the spending curve between May and August. Second, a small portion of jobs delayed were predicted to be canceled permanently. Based on the spending data, this likely did not occur at all, or the impact was very small. Finally, Dodge at that time was forecasting that residential new starts in 2020 would finish the year down slightly. With December starts data now in, residential starts for 2020 finished up 4%. In fact, over the final 5 months of 2020, new residential construction starts posted 4 of the 5 highest monthly totals since 2004-2006. Residential new starts finished 2020 at a 15-year high, with almost 50% of new activity for the year posting in the final 5 months, which will put a lot of that spending into 2021.

The updated chart below incorporates these changes to residential (only residential has been modified): no delayed projects canceled; all delayed spending restarted by August; new construction starts beginning in August, for the final 5 months of 2020, fastest growth in 15 years. This shows the latest starts data as adding to the recovery forecast between May and December and moving the future forecast residential spending line up on the index.

The table below shows the 2020 forecasts published at midyear by numerous analysts, the first opportunity to incorporate impacts from the pandemic recession. This table compares Construction Analytics Midyear July 2020 forecast to eight firms that reported nonresidential forecasts in the AIA Midyear July Outlook. Two of those firms, FMI and ConstructConnect, also published full forecast reports at midyear. The Actual totals for 2020 based on data through December, are shown in the first column. There will be very little change to Actual 2020 data when revisions are issued 7-1-21. Forecasts, all compared to Actual 2020, are marked best, 2nd best, worst. Where there’s limited comparison (Total, Residential and Non-building), only the closest is marked.

Construction Analytics midyear 2020 forecast garnered more bests than any other firm when comparing Midyear estimates to actual totals for the year. No one got residential correct, some reasons cited above, but Construction Analytics was the closest . I think it’s fair to say, Construction Analytics Midyear 2020 forecast was closest to 2020 Total Actual overall. Though, I do remember some other times with red in my column.

This next table shows the current forecast for full year 2021 forecast published as of January 2021. Of the eight nonresidential markets in the AIA Outlook report, the spread between hi and low forecast is 14%-17% for 4 markets, but 24%-25% for 3 markets and a spread of 45% for lodging. Spreads that wide are indicating some forecasts are all over the place. This will get compared next January when we know the Actual amounts for 2021. Watch closely nonresidential buildings.

Also, in July, along comes the Midyear Outlook, when usually forecasts improve a bit. That also will get compared next January.

1 Comment

  1. This is an excellent article Ed, and very useful. High marks for accuracy of Zarenski analytics. Thank you so much.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Follow Construction Analytics on WordPress.com

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 392 other followers


%d bloggers like this: