Did 2016 Projections Accurately Predict the Cubs?

We’re inundated with various projection models during the offseason, each with a unique spin on forecasting players. It figures that some are going to be better than others. ZiPS and Steamer, for example, are featured on FanGraphs and updated as the season rolls along. But how good are these projections models? Did they predict last season with a good degree of accuracy?

I went back and charted qualified Cubs players’ 2016 ZiPS and Steamer weighted on base average (wOBA) projections against their actual 2016 stats.

Overall, Steamer and ZiPS did an okay job of forecasting players’ seasons. The correlation between 2016 projections and reality was a mild .49 (.1-.3 = weak, .4-.6= mild, .7-.99= strong), meaning that the systems were able to capture approximately 24% of what actually happened.

Projections wildly underestimated Willson Contreras, Dexter Fowler, Kris Bryant, and David Ross, while overestimating Jason Heyward. Forecasted relatively accurately, however, were Javy Baez, Addison Russell, Jorge Soler, Miguel Montero, Tommy La Stella, Matt Szczur, and Anthony Rizzo.

The fine writers at Beyond the Box Score graded the major prediction models’ accuracy in 2016 (see below) and determined Steamer was the best at forecasting wOBA. ZiPS actually performed terribly in 2016, finishing near the bottom in every major offensive category.

How do we interpret these projections, then? The takeaway here is that ZiPS and Steamer miss about 75% of what goes into a baseball season, largely because the computers simply can’t account for major mechanical changes. For example, Willson Contreras and Kris Bryant’s dramatic swing changes resulted in overshooting their prognoses. Unless baseball statisticians develop a deep learning model, perhaps named Theo, we will never know whether a projection will hold true. Yet, despite the shortcomings of ZiPS and Steamer, both do a modest job at predicting numbers in the face of baseball’s natural randomness.