Thoughts About Drafting Quarterbacks

So let’s say you’re [Chicago general manager] Pace, and you’ve determined that you really want Trubisky. You call the 49ers and trying to work out fair compensation if the Browns do not pick him at one. You think Trubisky’s going to be the long-term Bears quarterback, starting in 2018 or later…. The market for quarterbacks is always weird…. For quarterbacks, NFL history says you pay Four Seasons prices. That’s why I can’t fault Pace for what he did. He wasn’t willing to risk losing the guy he loved.” — Peter King, The MMQB

I like my odds

The biggest story from the 2017 NFL draft was the surprise move by the Chicago Bears to pay a hefty price to the San Francisco 49ers — the 3rd overall pick plus the 67th and 111th picks, plus a 2018 3rd round pick — in order to move up a single spot in the draft, where they selected North Carolina quarterback Mitchell Trubisky. It was a stunning trade, in no small part because Chicago, having just spent significant money to acquire free agent QB Mike Glennon, was not an obvious candidate to draft a quarterback let alone pay a steep price to do so. But the Trubisky trade wasn’t the only trade involving a first round QB, as both the Chiefs and Texans also paid premiums to get their (hopeful) quarterbacks of the future, namely Patrick Mahomes and Deshaun Watson, respectively. Per both the traditional Jimmy Johnson trade chart as well as Chase Stuart’s updated version, both trades were expensive for the teams trading up: the Chiefs paid 120 cents on the dollar under the Jimmy Johnson chart and 170 cents on the dollar under Chase’s version, while the Texans paid approximately 125 cents on the dollar under the Jimmy Johnson chart and 154 cents on the dollar under Chase’s chart.

Was this rational? Well, a common response — and it’s the response baked into Peter King’s excellent MMQB piece from today — is that if the QB is a success and becomes a star and leads the team to Super Bowls, then the price tags for these teams will be cheap. The counter is essentially an extrapolation from the trade value charts: while that may be true ex post, on an ex ante basis the price tags are hefty — and later success won’t change that — particularly given that we all know many highly drafted QBs nonetheless fail.

I am not particularly interested in proving one perspective right or wrong (and I have not watched enough of Trubisky to really evaluate him), but I am quite interested in how to think about this question, as it’s one of the most vexing questions in all of sports and it’s a discussion that seems to frequently get off track. Further, it’s clear NFL teams approach QB draft picks entirely different than they do every other person; as Bill Barnwell wrote about and then separately said to me, “Teams are working with entirely different trade parameters for QBs than they are for players at most other positions.” To begin the process of answering the question, I wanted to lay out a handful of statements that I think (believe?) to each be true, although some point in different directions.

  • The most important thing is to understand the concept of “base rates,” or Bayesian probability. The basic idea — which can quickly get complicated — is that if you think there is a 95% chance of something happening (say, of Mitch Trubisky being the next Peyton Manning), the probability of that is not 95%, it is instead a blended probability that includes your estimated probability and the base rate, or historical success rate, for similar selections. Around 50% of 1st round QBs fail, and that number goes up for every round. (UDFAs actually produce more starters than 6th and 7th rounders, but that’s in part because there are so many more QBs that enter the league as UDFA than late rounders) As Dan Kahneman explained in Thinking, Fast and Slow (which every draftnik should read):

    [Bayesian reasoning] is named after an English minister of the eighteenth century, the Reverand Thomas Bayes, who is credited with the firsy major contribution to a large problem: the logic of how people should change their mind in light of evidence…. There are two ideas to keep in mind about Bayesian reasoning and how we tend to mess it up. The first is that base rates matter, even in the presence of evidence about the case at hand. This is often not intuitively obvious. The second is that intuitive impressions of the diagnosticity of evidence [i.e., the accuracy of your evaluation of how likely a QB prospect is to succeed] are often exaggerated…. The essential keys to disciplined Bayesian reasoning can be simply summarized:

    (1) Anchor your judgment of the probability of an outcome on a plausible base rate.

    (2) Question the diagnosticity of your evidence.

  • When I looked at the evidence most recently, the best data I could come up with was that QBs selected in the first round “succeeded” (fairly loosely defined; not limited to All-Pros and Super Bowl champs) around 50% of the time. The odds were slightly higher for top 10 picks and then slid down as the first round continued.
  • If you haven’t guessed, the failure to account for base rates is the flaw in King’s argument about Chicago (at least as presented; whether it was in fact a bad deal is something we’ll get to): Even if Pace is “sure” Trubisky was his guy, that doesn’t imply that no price is too steep as the base rates will anchor that probability down to something closer to 60-65%, even assuming a very high degree of confidence by the Bears. Ignoring base rates is one of the largest draft fallacies, and while it afflicts decisionmakers on both sides of the ledger it is most commonly an issue for those who are over optimistic about specific prospects. So trading up for a QB can be very risky.
  • However, having a good starting QB is extremely valuable. It is essentially impossible to win in the NFL with a bad QB. It is difficult, though not impossible, to win without a serviceable or replacement-level starter QB. It is much, much easier to win with a top tier QB. The position is essentially the most valuable one in all of sports.
  • Furthermore, a good QB on his rookie deal is maybe the most valuable asset in all of sports. Hitting on a rookie QB presents enormous organizational-level rewards: not only do you find your franchise savior, the CBA and rookie wage scale — plus the fifth year option for first rounders — results in massive savings for the franchise, money that can be spent on other assets. Barnwell: “No player in the NFL is creating more surplus value for his team than Dak Prescott, who appears to be an above-average quarterback in line to make less than $2 million combined over the next three years. Not far behind him is Carson Wentz, who should still be a massive bargain despite being guaranteed $21.8 million over the next three seasons (with a fifth-year option to come). When you consider that the free market guaranteed Brock Osweiler two years at an average of $14 million per year and Mike Glennon $18.5 million for one season, even the $7.3 million Wentz is going to earn is a relative pittance.”
  • Finding a serviceable QB is a real challenge, one exacerbated by the concept of a draft itself. When combined with a scarcity of good, starter-level QBs in the NFL more generally, the draft hands over to chance whether or not a team will ever hit on a rookie QB: A team needs a top pick in a year when there are such QBs available, and those sorts of QBs only come around every few years. In other years the team either must weigh drafting a QB too high or taking other needs (and teams drafting near the top typically have many). Skill and organizational stability also play roles, but there is a lot that simply comes down to chance — not to mention a negative cycle where downtrodden teams reach for first round QBs and thus propagate the cycle of failure->fire coaching staff/GM->new regime reaches to pick new “savior” QB of the future->failure->repeat, while a few teams go from one top-flight QB to the next, like the Packers from Favre to Rodgers or the Colts from Manning to Luck. Exacerbating this cycle is that elite QBs almost never come onto the free agent market.
  • Nonetheless, it is at least theoretically possible to compare the odds and cost of trading up and paying a premium for a first round QB — and thus taking a 50-70% chance on him becoming an effective starter — versus recreating that 50-70% probability by accumulating QBs in later rounds, i.e., the 2nd through 4th round. (Tom Brady aside, rounds 4-7 rarely produce quality NFL starters, and while there are a fair number of productive UDFAs, that is a much larger pool.) The analogy here is that most NFL draft picks are 2- or 3-star high school recruits, but that’s a quirk of the numbers because there are many more 2- and 3-star recruits by volume than there are 4- and 5-star recruits; a given 4- or 5-star recruit is significantly more likely to be drafted than a 3-star recruit.

Based on the above, below are some tentative conclusions:

  1. Good QBs, and rookie QBs in particular, are extremely valuable, and while the market price to acquire them is high it’s at least arguable that the potential payoffs justify even very steep price tags, even if the binary probability of success versus failure remains near 50% (or worse for later rounds). In other words, it still might be a coin flip whether Trubisky, Mahomes or Watson succeed, but if they do they would produce an incredible amount of surplus value to the entire organization in terms of on-field production as well as relative cost savings versus free agents and stability (i.e., job security).
  2. Nonetheless it is still possible to “overpay” (typically in terms of trade value) for rookie QBs, an issue that is most likely to arise if a team ignores the impact of base rates, i.e., the cost of a trade looks much more attractive if you think the likelihood of success of a potential QB is 85-95%, whereas even if that is your organization’s internal judgment — and I’d question you if it was — the true probability is somewhere closer to 60-70% given the impact of base rates. The same analysis applies in later rounds as well, applied to lower base rates of success (often dramatically lower), albeit compared to lower costs.
  3. It may be possible to more cost-effectively replicate the odds of hitting on a first rounder by taking multiple shots at QBs in the 2nd through 4th round range, until the organization hits on a QB. Seattle effectively used a version of this in hitting on Russell Wilson.
  4. Nonetheless, the most high percentage pool for identifying future star NFL QBs remains the first round, and that’s even after factoring in “first round” QBs who had no business being picked in the first round (EJ Manuel, Jake Locker, etc.).
  5. Finally, none of the above matters if your organization and coaching staff lacks the structure, knowledge and expertise to develop and coach a young quarterback. Without that you are doomed.

  • mfsmit

    There’s another finite resource that needs to be taken into consideration: starts. How many games does it take to know whether a QB has what it takes to be a viable starter? It’s certainly more than 2-3. It’s probably more than 16. How long before it was clear that Derek Carr was good? We’re still not sure if Teddy Bridgewater is good. People thought Colin Kaepernick and RG3 were good after 16 games, then it turned out they weren’t that good after all.

    How many coaches/GMs will be allowed to spend 16 games on each of half a dozen mid-range draft picks at QB before ownership and the fan base lose patience? For this reason a player with a 60% chance of success is more than 3x as valuable as a player with a 20% chance.

  • Brad Warbiany

    You also have to look at risk in another way, though. What is the risk you lose out on the guy you want, and what is the next best replacement at the position.

    The Bears were picking #3. The Browns had already picked, and didn’t take a QB. Some might think that SF needed a QB after losing (i.e. running out of town) Colin Kaepernick, but look at it ex post, they didn’t take a QB until the 3rd round. That QB was CJ Beathard, who is one of those later round “see how he develops” QB’s.

    So the Bears gave away the farm to a team that didn’t even take the position the Bears wanted. Granted, perhaps if Trubisky was THAT amazing, the 49ers would have taken him. But instead it sounds more like they just wanted to fleece some poor sap out of WAY more than a player they didn’t even want was worse.

    But let’s say the Bears didn’t bite on it, and then the 49ers take Trubisky. Would the Bears be better off with Mahomes or Watson, even taking them “too high” at #3, and retaining all those other picks they gave up? They paid a high price but is the replacement value of a 10 pick (Watson) or a 12 pick (Mahomes) that much worse than a 2 pick?

    If they were trading 15 spots up to #2, ok… I get it. Pay out the nose if you REALLY need a QB. But you’re trading up *1* spot from a team that doesn’t desperately need a QB with the knowledge that even if they take the QB you want, you have some pretty decent guys that you’ll be first in line for.

  • smartfootball

    I think the Bears were less concerned with the 49ers (though it was not unfathomable that they would take Trubisky) than they were with SF trading with some other team who would take him. Peter King’s article indicates that the Bears were the only really serious contender, but of course Chicago didn’t know that.

  • Brad Warbiany

    True… SF could have traded down to someone else. People thought SF might be interested in trading down overall based on their earlier “open for business” quote, and this was proven true by the fact that they accepted the Bears trade.

    But still… At what point do you say “Ok, I’ll take Watson or Mahomes and keep all those draft picks”?

    I think it’s hard to look at this and say that the Bears didn’t get fleeced. Of course we’re looking at it in hindsight, but even if your analysis suggests that QBs are rightfully worth valuing extremely highly, it doesn’t really justify that Trubisky at 2 is necessarily SO much better than Watson or Mahomes to give up what they gave up.

  • Nuno Bojar

    I have no idea why Cutler was an NFL starter for so many years.

  • Steve / Sports + Numbers

    Regarding the conclusions:
    1. An additional complexity is that the base includes years where there was a “consensus” #1 pick (e.g., Manning, Vick, Luck etc.) and years where there was not (e.g., this year). Admittedly this would include some hindsight bias too, but I am guessing you could go to pre-draft rankings and identify years with lower variance at the top to see a somewhat higher success rate in those with corresponding lower success rate (at least at QB) in high variance years.

    2. This is where ownership has to act as a brake on management. Management, particularly if performing poorly to date, has incentive to reset the clock in terms of when a decision should be taken on their fate while also swinging for the fences. Spending $2 on $1 of expected value makes sense when the upside is a Dak Prescott-style 5-10x return that would provide job security for a decade.

    3. I suspect there is a capacity issue where teams do not have the coaching resources or volume of first team reps available to assess what they really have. Adding 2 more developmental QBs doesn’t help if you already have a better backup (e.g., Brady/Bledsoe, Warner/Green) and don’t realize it.

    4. It’s so hard to tease out the cause/effect around this – particularly with QBs. Teams see that 1st round QBs have more success and so they draft QBs in the 1st round. Some stats:
    Years Avg QBs Avg 1st Rd QBs
    93-99: 10 2.3
    00-09: 13 2.6
    10-17: 12 2.8

    5. Agreed. Path dependence of players is a huge issue. Was Player X a bust because he was not talented or because the team lacked the quality/quantity of resources to develop him?

    Anyway, loved the post. Hope you don’t mind a full post in the comments.

  • Pingback: Smart Football: Thoughts about Drafting Quarterbacks - Touch the Banner()

  • Jonas

    I don’t see a lot of potential in trading-up generally, but the SF-Bears trade was one of the least logical I’ve seen.

    Even if Trubisky is “your guy”, the drop in probability of success between the draft’s QB1 and QB2 is tiny. You’re not giving away loads of draft picks on a 50% chance that he becomes a star, you’re giving away loads of picks to increase your chances by 4%. That’s not smart.

    I think, if I had to draft a QB, I’d wait till the end of the 1st round to take one (so I have a fifth year option), and then grab another QB in the 3rd or 4th round every year after that.

  • fps129

    I agree only when when you dont have a top 5 pick in a draft where the QB position is loaded. Otherwise, if you have a surefire prospect in your lap by the time it’s your chance to pick, you gotta take him.

    If you really want to hedge your bets, go the Redskins route and draft a developmental but promising QB in the mid rounds just in case so you can have a Kirk Cousins ready when your RG3 doesn’t work out.

  • Joe Bobbles

    the bears got fleeced, for a QB who is not exceptional, and on a terrible team that will not support his development.

    Pace is done.

  • IrishBarrister

    The main issue I see with these types of trades is that, at least viewed from the outside, they seem to ignore the second prong of Danny Kahneman’s summary of Bayes’ theorem, i.e., what makes us think guy X is so much better than the base rate group? Simply coming to the conclusion that you think Trubisky is your guy and averaging it with the base rate is not enough (in my view). It requires the further step of analyzing what sets Trubisky apart from other quarterbacks drafted in the first round to justify your high opinion of him. To be clear though, I’m not saying that analysis has be based entirely on statistics; intangibles may be considered as well.

    And in that, I hear anything from the Bears that justified the alleged differentiation of Trubisky to many other first round quarterbacks in the same way I did for a guy like Andrew Luck, or maybe even a Marcus Mariota. I wouldn’t mind the move up if it was better supported in this regard.

  • Donny Landis

    The BIGGEST upshot of using base rates in handicapping success probability in this context is that, not only does it anchor the probability of your own estimation, but it dramatically lowers the “switching costs.”

    Let’s say you have QB1 and QB2. In your assessment QB1 is 85% likely to succeed and QB2 is only 75% likely to succeed. On it’s face that 10% looks like a hefty price to pay. If you were to anchor both probabilities with a base rate as above (to dramatically oversimplify, let’s just give equal weight to 50% base and your estimate), the result is QB1: 67.5% vs. QB2: 62.5%. In that light, there’s no way to substantiate such expensive trades.

  • gomer_rs

    Manning wasn’t a consensus #1, there was a real Leaf/Manning debate.

  • gomer_rs

    That’s basically what Seattle did with Flynn/Wilson and SF did twice, once with Montana as the developmental player, and once with Steve Young as the developmental player.

  • Sports Village

    This is the first time visit to your blog.
    Very interesting article.
    Keep posting.
    Football coaching camps in hyderabad
    Tennis Clinics in hyderabad</a