The Sabermetric Revolution: Assessing the Growth of Analytics in Baseball (17 page)

Since the linear weights and plus/minus systems have complementary strengths and weaknesses, it is not surprising that one of the new approaches is to combine them. For instance, the APM score for a player can be used as a dependent variable and regressed on the player’s box score stats (points, shot percentage, assists, rebounds, etc.), yielding a result called Statistical Plus/Minus. While this approach may fill some holes by identifying the specific
nature of a player’s contribution, it does not fill them all.
²⁰
Hence, analysts continue to experiment with other methods that endeavor to identify metrics regarding the interdependencies of team play.

Similar to basketball, in football all the players on the field are likely to have an impact on every play. For instance, if an offensive guard misses a block, a pass play is likely to abort or a run is likely to fall short. This guard has an important role in virtually every offensive play.
²¹
This is in distinct contrast to baseball, where a batter is up maybe four times a game, which might amount to only 5 percent of the plays in the game. A baseball fielder (outside of the battery) might be involved in anywhere from zero to ten plays a game. Because football and basketball players can impact every play (allowing for the defense/offense platoon in football), each player can be more decisive in the determination of the game’s outcome. But unlike in basketball, where there are only five players, football has twenty-two players plus two kickers (not counting special teams), so the importance of an individual player is bound to be smaller than in basketball.

In football, because all eleven players are probably contributing in some way to the success of the play and each player’s execution cannot be precisely measured, there is a large issue with attribution. A running back depends on the blocks and blocking scheme of the linemen, the handoff (and fakes) from the quarterback, the defensive alignment and execution, the play call given the game situation, and so on. Every play has these interdependencies.

And there are other conundrums. For instance, after extensive study of video and interviews, the ESPN football analytics team attempted to allocate responsibility for a successful long pass. They began with the hunch that a long pass requires not only distance but precision from the quarterback and so a quarterback should receive more of the credit for a successful long pass than the end who caught it. Similarly, a short pass is easier to throw and harder to catch, so the receiver should get more of the credit. What they discovered is exactly the opposite. For most long passes, the receiver has to adapt to the throw, slowing down, speeding up, taking a different angle, repositioning
himself vis-à-vis the defender. Most short passes demand speed, precision, and timing from the quarterback, whereas the receiver runs to a designated spot. One might question this finding, but the fact that all these variables are in play is hard to deny.

It is also hard to deny that some of the traditional metrics that have been used in football, such as the well-traveled NFL Passer Rating, are based on arbitrary elements and can be improved upon with careful thought and analysis.
²²
Such modifications, elementary though they may be, can still provide a competitive advantage to teams that are willing to be open-minded about new metrics.

So, where does that leave football analytics? There are always new twists for player ratings systems and for evaluating game strategies.
²³
One of the more prominent analyses of game strategy was published by University of California, Berkeley economist David Romer in 2006. Romer asked the questions: when should teams facing fourth down go for the first down, when should they attempt a field goal, and when should they punt? In order to answer these questions, Romer needed to generate a probability function of point outcomes at different yard lines for both the offensive and defensive teams. The analysis must assess the chances of making the necessary yardage for a first down or touchdown times the point value of success (e.g., seven points for a touchdown with extra point) and the expected value for the opponent if the team fails to get a first down or touchdown, given that the opponent will start a series at a particular yard line. In order to undertake this analysis, Romer generated a yardage chart (
Figure 10
).

Romer compared his model with actual team choices and concluded that coaches generally are too conservative on fourth down.
²⁴
The risk-averse coaches seem to prefer making the call that is anticipated by football convention rather than the call that maximizes expected point value. The potential praise for being aggressive and right apparently is more than offset psychically by the scorn that would rain down on the coach if the aggressive play did not pan out. Romer’s analysis has since been corroborated and extended by several studies.
²⁵
Notable among the extensions is the work of Brian Burke at advancednflstats.com. Burke has elaborated the expected points based on field position model to include the down and yards to go, as well as adjustments for the game score.

Figure 10. The Values of Situations

Source: David Romer, “Do Firms Maximize: Evidence from Professional Football,”
Journal of Political Economy
114 (2006): 346.

As suggested earlier, Bill Belichick appears to be the exception that proves the rule. In an important game against the Indianapolis Colts on November 15, 2009, Belichick’s swagger got the best of him. Nursing a six-point lead with 2:08 to go, Belichick decided to go for it on fourth-and-2 from his own 28 instead of punting the ball and making Peyton Manning and the Indianapolis Colts go 80 yards for the score. The fourth-down play failed, the Colts took over possession and scored with ease, winning their ninth straight game (35–34) and getting in the driver’s seat for home-field advantage in the AFC playoffs. Belichick was subsequently raked over the coals by commentators.

One indication of the difficulty in finding suitable football performance metrics is the lack of consistency in the major indicators. As discussed in
Chapter 3
, a reliable performance metric should have at least two qualities: a strong relationship with team wins and predictive ability. Predictive ability comes from year-to-year consistency. If a metric succeeds in explaining a high percentage of the variation in team wins, but it fails in that individual players do not have consistently high or low performance from one year to the next as measured by the indicator, then the metric will do little good to
GMs who are trying to put together a winning team. Using data from 1994 through 2007, David Berri and Martin Schmidt find the key football metrics have very low consistency.
²⁶
For instance, the percent of variation in a metric in year two explained by variation in the metric in year one for major performance variables is as follows.
²⁷

For Quarterbacks:

For Running Backs:

Compare these statistics with some of those in basketball:

Or, with baseball:

Any new composite performance metric will be made up of these standard indicators. If the indicators vary widely from year to year for the same player (presumably, in part, because they depend heavily on one’s teammates and opposing players), then they will not provide a reasonable basis for personnel decisions. In this regard, the low year-to-year correlations in football performance metrics stand out, and present a particularly difficult challenge
to metricians seeking to identify more significant statistics.
²⁸
Nonetheless, simple advances in some conventional statistics are attainable and may help teams get a leg up on the competition, as appears to have been true for the New England Patriots.

It is important to observe, however, that we have not directly traced the Patriots’ on-field success to their use of new metrics or analytical modes. As with baseball, it appears that it is the most intelligent front offices that embrace new methods, especially new methods requiring specialized knowledge that the top executives may not possess, and the success of these teams may be a function of elements of front office intelligence that are not connected to statistical analysis.

Does hiring a metrician pay off for basketball, football, and soccer teams? Have teams that have innovated in the use of statistical analysis to assist in personnel or strategy decisions benefitted? As with baseball, these are not easy questions to answer.

Suppose you are a metrician on an NBA team. You are assured that your statistical analysis will matter in team decision making because you are also the GM. Next, assume that you have examined the rosters of the last twenty-five teams that have won an NBA championship and you noticed that in twenty-four of those cases, the championship team had an all-time Top 50 player on the roster. You conclude that for your team to be a top competitor, it needs to hire at least one of the top ten players in the league. That’s the easy part.

How do you get one of these players? If your team lacks the better prospects in the league, then it is not likely you’ll be able to trade for a superstar.
²⁹
Because of the salary cap and the Bird Exception, signing a superstar via free agency is an improbable option. The best option might be to shoot for the bottom of the standings and hope to get an early draft pick. Because of the NBA draft lottery, even this strategy (which some teams appear to follow) is uncertain.
³⁰
There are, of course, permutations of these strategies, but the point should be clear: given the rigidity of the NBA’s labor market, it is never
a simple matter to put together a winning team—no matter how perceptive your statistical analysis might be.

That said, there is little evidence that those teams leading the way with analytics have fared better than the teams with more traditional approaches. To be sure, Grousbeck’s Celtics benefitted from an elementary analytical insight, but not one that required complex statistical skills.

Similar to basketball and football, in soccer the nondiscrete and interdependent nature of play makes it substantially more difficult to identify compelling and practical performance metrics. One prominent soccermetrics blogger, Graham MacAree, commented:

There seems to be a perception that football [meaning soccer in U.S. vernacular] statistics are entering some sort of golden age. With the proliferation of statistics sites such as WhoScored and EPL Index, not to mention the massive popularity of OPta’s Twitter feeds and the Guardian’s sadly-discontinued Chalkboard service, it’s not hard to see why. Information is available where previously there was none.

But anyone claiming that the
Moneyball
revolution is under way in football is sadly mistaken. The current statistics fail (and fail utterly) at passing Bill James’ language test. If a player makes two fewer tackles than average but one more interception with more completed passes, for example, we have no way of figuring out how to put those statistics into context. What we currently have are numbers, not meaning.
³¹

MacAree then continues on a more sanguine note: “Football is a complicated game . . . but . . . There’s obviously some structure in the sport, and that alone is proof that we’re not looking at an impossible problem.”

Relatedly, a debate about the significance of statistical analysis erupted shortly after Lionel Messi broke an international record by scoring his eighty-sixth goal during the 2012 calendar year in the top division of the Spanish soccer league (La Liga). The former manager of Messi’s Barcelona team, Pep Guardiola, declared that neither words nor statistics can adequately describe Messi’s impact on a game. The only way to grasp Messi’s impact, Guardiola declared, is to watch him play in the virtuosity of the moment.
³²
Of course,
the very fact that Messi broke a scoring record suggests that his talent can be described, at least in part, by numbers.