The NFL’s quarterback rating is bizarre and misleading, so it is no surprise that many have tried to invent the better mousetrap. What is a bit surprising is that ESPN has invented its own metric, which it will undoubtedly promote relentlessly. And, in the twist that is maybe what is most surprising of all, it’s actually pretty good, or at least a well grounded attempt to move the ball forward (at least when it comes to this kind of thing). As Chase explains:
The [formula behind the existing] quarterback rating is complicated, but it can be reduced to a simple formula. That’s what Bob Carroll, Pete Palmer and John Thorn discovered in their groundbreaking book, The Hidden Game of Football. Essentially, QB rating is equivalent to yards per attempt, but with a 20-yard-bonus for each completion, an 80-yard-bonus for each touchdown, and a 100-yard-penalty for an interception. Such adjustments should seem ridiculous to every reader, which is why everyone finds quarterback rating ridiculous. By way of comparison, PFR’s ANY/A formula — in addition to including relevant data on sacks — gives no bonus for completions, a 20-yard bonus for touchdowns and a 45-yard penalty for interceptions.
But on Thursday, ESPN released the methodology behind its new QB Rating. And last night, ESPN aired an hour-long segment at 8 PM to discuss the new formula. So how does ESPN’s formula look? There’s some good and some bad, which means it has exceeded my expectation. . . .
ESPN has essentially limitless resources, which means it can: (1) hire some very good thinkers; (2) hire some very good computer programmers; (3) gain access to thousands of hours of high quality film; (4) hire lots of people to log key data for each play; and (5) discuss ways to tweak the formulas with current and former NFL QBs. On the surface, ESPN would be in perfect position to really knock this one out of the park. . . .
Unfortunately, ESPN is keeping its formula a secret from the public. There are lots of legitimate reasons for doing this, but that decision makes it impossible to fully criticize the ESPN QBR. It’s possible that there are serious bugs in the formula, but we have no way of knowing or discovering them. . . . For all the flaws in traditional passer rating or any of the formulas I’ve come up with over the years, you at least know what the flaws are. You can recreate the rankings because you have access to the formula. You can catch errors. You understand why rankings appear the way they do. And you can catch a simple programming mistake that throws off the computer ratings because of human error.
Still, I think the formula will have some utility. ESPN is building on the great work Brian Burke has done on expected points, and that makes sense. The QBR seems lot like what Football Outsiders has done with their method of ranking quarterbacks. I don’t think ESPN is just being ESPN here; I think the powers that be have actually spent a bit of time kicking the tires and deciding what works. But too much is left to having faith in ESPN.
See also Aaron Schatz’s critique here. But Chase sums up my initial take well here:
My biggest complaint is that this looks to be very retrodictive or explantory statistic but will be used in a predictive manner.
That Matt Ryan or Sam Bradford had X or Y in this particular statistic will tell you very little about how they will play next week or even next season. Moreover, maybe I’m being crotchety, but I still find it extremely difficult to use metrics like this to actually make football decisions, which to me is the gravamen of any useful statistic. If you’ve got a three way quarterback competition in the spring and fall camp, and you chart every pass skeleton and team throw your quarterbacks make (which you should), would you really try to filter those results through this formula (assuming you even could)? No, you wouldn’t. And maybe I’m being retrograde, but when it comes to actually evaluating your talent or your opponent I’m still more interested in a very simple Yards Per Attempt metric with a subtraction of 20-40 yards per interception. (I do think there is some wisdom in subtracting out yards after the catch as ESPN’s metric does, but I’m also not sure that it’s right to just completely sever it.)
This disconnect from the practical is exemplified by ESPN’s focus on “clutch” performance. I’m a big believer that if your quarterback is an 80 he’s not going to suddenly become a 90 in a “clutch” situation, and in any event if you want your metric to reflect a player’s clutch ability you get enough with just equal weighting the discrete events. If one player plays well in clutch situations and another doesn’t, that will already be reflected in the metric; there’s no need to arbitrarily ratchet up the “clutch” effect based on a very limited sample size.
In any event, I’ll let the better stats guys continue to break down the ESPN formula, but will simply reiterate my concern that this is a solution without a problem. Indeed, in a world where offensive lineman still continue to throw off almost no statistics, do we really need ESPN to use its considerable resources of time, money, and brains for hire to create yet another quarterback metric? It’s a valiant first attempt at getting the new ESPN stats group going, but I hope they push the game in new areas rather than create more statistics like these.