ESPN’s new Total Quarterback Rating

The NFL’s quarterback rating is bizarre and misleading, so it is no surprise that many have tried to invent the better mousetrap. What is a bit surprising is that ESPN has invented its own metric, which it will undoubtedly promote relentlessly. And, in the twist that is maybe what is most surprising of all, it’s actually pretty good, or at least a well grounded attempt to move the ball forward (at least when it comes to this kind of thing). As Chase explains:

The [formula behind the existing] quarterback rating is complicated, but it can be reduced to a simple formula. That’s what Bob Carroll, Pete Palmer and John Thorn discovered in their groundbreaking book, The Hidden Game of Football. Essentially, QB rating is equivalent to yards per attempt, but with a 20-yard-bonus for each completion, an 80-yard-bonus for each touchdown, and a 100-yard-penalty for an interception. Such adjustments should seem ridiculous to every reader, which is why everyone finds quarterback rating ridiculous. By way of comparison, PFR’s ANY/A formula — in addition to including relevant data on sacks — gives no bonus for completions, a 20-yard bonus for touchdowns and a 45-yard penalty for interceptions.

But on Thursday, ESPN released the methodology behind its new QB Rating. And last night, ESPN aired an hour-long segment at 8 PM to discuss the new formula. So how does ESPN’s formula look? There’s some good and some bad, which means it has exceeded my expectation. . . .

ESPN has essentially limitless resources, which means it can: (1) hire some very good thinkers; (2) hire some very good computer programmers; (3) gain access to thousands of hours of high quality film; (4) hire lots of people to log key data for each play; and (5) discuss ways to tweak the formulas with current and former NFL QBs. On the surface, ESPN would be in perfect position to really knock this one out of the park. . . .

Unfortunately, ESPN is keeping its formula a secret from the public. There are lots of legitimate reasons for doing this, but that decision makes it impossible to fully criticize the ESPN QBR. It’s possible that there are serious bugs in the formula, but we have no way of knowing or discovering them. . . . For all the flaws in traditional passer rating or any of the formulas I’ve come up with over the years, you at least know what the flaws are. You can recreate the rankings because you have access to the formula. You can catch errors. You understand why rankings appear the way they do. And you can catch a simple programming mistake that throws off the computer ratings because of human error.

Still, I think the formula will have some utility. ESPN is building on the great work Brian Burke has done on expected points, and that makes sense. The QBR seems lot like what Football Outsiders has done with their method of ranking quarterbacks. I don’t think ESPN is just being ESPN here; I think the powers that be have actually spent a bit of time kicking the tires and deciding what works. But too much is left to having faith in ESPN.

See also Aaron Schatz’s critique here. But Chase sums up my initial take well here:

My biggest complaint is that this looks to be very retrodictive or explantory statistic but will be used in a predictive manner.

That Matt Ryan or Sam Bradford had X or Y in this particular statistic will tell you very little about how they will play next week or even next season. Moreover, maybe I’m being crotchety, but I still find it extremely difficult to use metrics like this to actually make football decisions, which to me is the gravamen of any useful statistic. If you’ve got a three way quarterback competition in the spring and fall camp, and you chart every pass skeleton and team throw your quarterbacks make (which you should), would you really try to filter those results through this formula (assuming you even could)? No, you wouldn’t. And maybe I’m being retrograde, but when it comes to actually evaluating your talent or your opponent I’m still more interested in a very simple Yards Per Attempt metric with a subtraction of 20-40 yards per interception. (I do think there is some wisdom in subtracting out yards after the catch as ESPN’s metric does, but I’m also not sure that it’s right to just completely sever it.)

This disconnect from the practical is exemplified by ESPN’s focus on “clutch” performance. I’m a big believer that if your quarterback is an 80 he’s not going to suddenly become a 90 in a “clutch” situation, and in any event if you want your metric to reflect a player’s clutch ability you get enough with just equal weighting the discrete events. If one player plays well in clutch situations and another doesn’t, that will already be reflected in the metric; there’s no need to arbitrarily ratchet up the “clutch” effect based on a very limited sample size.

In any event, I’ll let the better stats guys continue to break down the ESPN formula, but will simply reiterate my concern that this is a solution without a problem. Indeed, in a world where offensive lineman still continue to throw off almost no statistics, do we really need ESPN to use its considerable resources of time, money, and brains for hire to create yet another quarterback metric? It’s a valiant first attempt at getting the new ESPN stats group going, but I hope they push the game in new areas rather than create more statistics like these.

  • Steve

    It seems that so many of the new metrics are trying to discount YAC from the QB (ESPN, Pro Football Focus).  How much credit do they deserve?  Isn’t that the attribute about Montana that everyone loved:  putting the ball in a spot where the WR can catch and run.

    Where would you assign value here:  Brett “Favres” a pass into double coverage for a 30 yd gain, or Brady hits a wide open many 10 yds down the field who runs for another 20.  Should we really give more credit to forcing a ball rather than reading coverage and allowing your WR to catch and run?

    Now, I think on screen passes, the QB should not get full credit, but there are certainly some instances where the QB deserves credit for making reads and putting the ball in an optimum spot.

  • Anonymous

    I agree. I think the problem — and the insight of subtracting YAC — is that when you do that it often does reveal the real QBs versus the wannabees because young/mediocre quarterbacks tend to have a disproportionate number of their yards come from screens and runningback check-downs. But there is a measure of patience there, and I’m not convinced quarterbacks are totally useless when it comes to yards after the catch. 

  • Anonymous

    I agree. I think the problem — and the insight of subtracting YAC — is that when you do that it often does reveal the real QBs versus the wannabees because young/mediocre quarterbacks tend to have a disproportionate number of their yards come from screens and runningback check-downs. But there is a measure of patience there, and I’m not convinced quarterbacks are totally useless when it comes to yards after the catch. 

  • Steve

    Yeah, that makes sense.

    This video shows two of my favorite examples of the QB deserving a TON of credit for YAC.  Of course, it’s because Brady and Manning got the offense into a good play.  Obviously this is impossible to account for with any metric, but it’s really fun to watch. 

    Watch Brady keep Gronkowski, Hernandez, AND Welker in to block creating a 2 man route combo:

  • caleb

    Seems odd to totally remove YAC. Running a good screen play is a legitimate skill for an NFL QB – some QB’s run screens much better than others, and it seems that skill is totally ignored in this metric.

    Also, why is letting “clutch” situation have an influence a negative thing? Sure, *in theory* your QB is an 80 no matter what, but the eye test doesn’t lie sometime – some guys really struggle in 4th quarter down 5 points and some guys are lights-out in those circumstances. Whether it *should* happen or not is surely debatable, but I think it’s difficult to argue that it doesn’t happen with some guys. 

  • Jesse

    Wouldn’t QB X be at least partially responsible for the success of a screen if that screen only worked because the defense was cheating to defend the pass? QB X is burning the defense through the air -> DEs begin to fly upfield to rush the passer -> Screen play works -> QB should deserve some credit?

  • Stan Brown

    Of course, there are all kinds of ways QBs affect the game.  Your comment re: screens is no different than his impact on a draw play and he gets zilch credit for those.   How do you measure the ability to check out of bad plays and into good?  How do you measure the ballhandling that fools a defender and opens a bigger running lane?  How do you measure the impact of a QB who scares defenses into dropping 2 deep safeties so that the offense can run the ball all night?

  • Stan Brown

    Btw, the whole concept of clutch is stupid.  If a player actually raises his level of play for the 4th quarter of close games and in the playoffs, he should be fined for gross unprofessional behavior.  A pro brings it every week for 4 quarters.  Unless a team is undefeated, this so-called ‘clutch’ player didn’t bring his best in a game his team lost.  He should be fined.

    Clutch is just another way of saying that a player lacked the professionalism to bring his A game every game, all game long.

  • srp

    Worse yet, points scored in the first quarter count just as much as those at the end. Walsh’s 49ers won a ton of games with Montana’s stats skewed to the first half, because they got more conservative with a big lead. Should Montana be penalized for his first-half efficiency?

  • David Myers

    To be plain spoken, the NFL formula is grievously misunderstood. I’ve been writing about it in one form or another for a month. Most critiques are a proof of the ignorance of the critic, rather than a real fault of the formula. I’m a little angry to be honest, Chris, because most of what I’ve done with the NFL rating has been retweeted by Brian Burke six or seven times, and most of what I’ve talked about could have been discovered by a decent high school algebra class.

    In the modern NFL, ANY/A and the old style NFL rating are functionally equivalent. It’s just not evident at first glance.  And I think I could teach a sophomore high school class how to handle  the derivation in my link.