The most tepid defense of scouting you’ll ever read

The above is the title of my new piece, available over at the NY Times Fifth Down. Thanks to everyone over there for putting it up. Feel free to post comments here, where it is easier for me to respond.

  • Mike

    FYI- your link to the NYT is broken, but your article is still easy enough to find, as its on the front of the Fifth Down page.

    Chris- Great article. I can’t disagree with anything you’ve written, but i pose a question to you:

    Where do you stand on the inherent value to a draft pick of the team that drafts him? Ive contended over the years (though not empirically)that the success of the high quality athletes (in the first round, especially) depends highly upon the team and system that takes them. Brief examples include, though their stories have yet to conclude, qb’s like Roethlisberger and Flacco. If Flacco goes say, to my beloved (ugh) Bills and starts right away, is he nearly as successful as if he goes to the Ravens, with their established defense?

    Using the Bills as another example, there was much discussion all year in our little circle that they made a huge mistake passing on Orakpo to pick Maybin. Orakpo was largely seen as having a successful rookie year, while Maybin is already fighting the bust label. I could make the argument that Orakpo excelled because of the 3-4 which allowed him to roam the outside, while Maybin was plugged into a 4-3 strictly as a DE. In cases like these, I tend to think that it was more the team and the coach that dictated the success or failure of the draft pick, as opposed to the players ability, as they both were good enough athletes to be considered high first round picks.

    I guess the question is, since scouts seem to have trouble finding anything to empirically measure regarding the players, do you think their focuses might be better attributed to studying players more intensively in the context of the team that may draft them? With every passing year, I seem to think more and more that the situation into which the pick is drafted is usually much more important than the overall scout-driven draft status.

  • The link is fixed now; for some reason it changed. Mike: good questions, and I will respond a bit later today. Keep ’em coming.

  • Fantastic article, Chris.
    As my buddy Maurile once explained, scout speak is filled with Buchsbaum-speak. Buchbaum was a terrific talent evaluator and an equally skilled writer. He used many of the phrases you hear scouts and talent evaluators used today. Younger scouts couldn’t copy the quality of his work, so they copied his style instead. This caught on, so now everyone uses the same style since it was so respected, even if they have no idea what they’re talking about.

    (In some ways, it’s a lot like a law firm. Junior associates will talk, speak and act like the senior associates or partners they admire, even if the things coming out of their mouth make no sense and they look like an idiot, doing it. It must be ingrained in our brains to act like the people we admire, even if we can’t match the underlying quality of their work.)

    I think the point about the wisdom of crowds is a great one. I fully agree with you there.

  • Chris S.

    Overall, I agree that it was a good article, though I think it could have been shorter and crisper. Though I recognize the intent was to spell out the challenges of player evaluation, to me you seemed to write in circles a little bit in the second half of the piece.

    I agree with your point that vague evaluations should be taken with a grain of salt (“Being a good short yardage RB is less about size/power and more about vision and feel.”) But I don’t understand the knock on the supposedly meaningless verbiage (“waist-bender, slow feet, no anchor” etc.). If it’s accurate shorthand for flaws in their playing technique and can be corroborated by looking at the film, that seems valuable to me.

  • Phil

    I’m going to say that you’re not giving scouting enough of its due,

    there is a difference between not having quantifiable data and not having data

    think about the data most employer’s have when hiring a college graduate to a typical job, you have a GPA, maybe you have the list of classes that went into obtaining that GPA, you have how they can conduct themselves in an interview, maybe they call up a professor and get a recommendation, almost always one the candidate provides the name of.

    Now compare that to what NFL scouts do, they can watch almost every single play a player plays in, in any number of situations, can show up at practice,

    I would say that outside perhaps baseball, they have better more relevant information prior to making hiring decisions than probably in hiring process that I can think of.

    No it can’t be done by mathmatical formula, but really how much in life really can be, that’s what the vast majority of human beings are paid to do, create analysis that a computer can’t

  • Ike

    Chris, one thing you neglected to tackle in your piece is the call for more imaginative data.

    With the video libraries we now have available, there may indeed be hidden indicators at work that no one has though to tap.

    Yes, there are MANY variables at play in football, that don’t get canceled away through the “batter-pitcher funnel” we have in baseball. But the sabremetrics guys still found some gems that correlated to success.

    The thing football does have going for it is better processing power. The baseball guys didn’t have the powerful computers we have now, and we can do some really basic regression analyses on the football data. We just need someone to kick someone else in the pants to start, and be open to the possibility that a solution exists.

    And is cost effective.

  • “the scout offers streams of platitudes and meaningless verbiage like…”

    “N.F.L. teams rely on advice from guys who don’t coach football and who base their conclusions on totally irrelevant factors”

    Brilliant write-up, Mr.Brown. While I don’t believe you are intentionally bad-mouthing scouts, just the ‘mysticism’ that it has essentially spawned.

  • George

    I thought this was a really effective and enjoyable piece. Glad to see you writing again. It’s hard to be nuanced to deal with all the streams of garbage and reconcile it with the good faith effort people put forward.

    I also agree with some commenters (though I don’t think you disagreed) that there is a way forward to get better stats, but right now the draft is a strange world.

  • Mike: You ask a good question. I think to answer it you’d need to divide it into a couple of inquiries. One: Do some teams’ draft picks have better “careers” (however you define such success as a metric) than others? I.e. if you’re drafted by New England, even if you’re a lower draft pick than someone drafted by, say, Seattle, are you expected to have a better career?

    And second: If the answer is yes (and it might vary depending on the position), can we separate out whether that is due to the better environment (i.e. Flacco going to a better “situation”), or simply because those teams are better at evaluating talent? One of the findings in the Massey-Thaler study is that although, say, a late 1st rounder tends to be better than a 2nd rounder, teams that “trade up” tend to overpay for those guys, both in terms of what they trade and what they pay. This kind of thing could help determine whether they have better careers for nature (talent selection) or nurture (environment/quality of teammates) reasons.

    If your theory is true it requires a yes answer to the first question and the answer to the second must be that teams are all generally as good at the actual scouting/selection process but are not as good of places to play, for whatever reason. That might be true, all things considered, though it strikes me as odd that teams that are consistently worse off the field would still be just as good (or at least close enough) at evaluating talent as the best teams. Though this could be possible considering all the help and extra data available about the draft. The theory clearly has appeal, and I think you’re right that if it applies to anyone it applies to quarterbacks, who are so teammate dependent and whose “development” is seen as integral to their careers.

    Chase: I think you’re right that the scouting jargon has been adopted not to convey information but instead to make the practitioners appear like experts. It’s a subtle way of consolidating power and for the younger scouts to try to fit in, and we all can get suckered by it.

    Chris S: I could be wrong (I’m not sure of a good way to measure it) but much of what scouts say really is meaningless. My point here is not that they are stupid, but that the process of evaluating talent is largely done by sort of “pre-verbal” parts of the brain — it’s not really even something that can be broken down into super-granular descriptions. Indeed, how do you compare “waist-bender” with “tight-hips” or whatever else? They don’t actually contribute to making decisions nor do they help you rank players. It’s not like you can add up three “waste bender” type criticisms against four “explosiveness is explosives” and then say, hey, four compared to three is good. Again, doesn’t mean that the end-evaluation is wrong, just that the descriptions are unhelpful.

    Ike: I think one of the problems with the draft is that there is so much information and little of it is helpful. GPA and test scores are highly correlated with graduation rates and college GPAs, and are decently correlated with career success. None of the numbers scouts use have such strong correlations, and, further, there really is almost too much data — a sort of paralysis of metrics that could be looked at, none of which are all that helpful, which sends you back into subjective “impression” mode. (Which again, is not necessarily a bad place to be, just not the optimal one.)

  • Reinhard

    Hi Chris awsome website glad to see some new articles.

    Considering the absence of “hard” data what is it that you would like scouts to write on their reports?

    Good or bad? Decent or meh? 3 out of five stars? That would be even less meaningful than their observations.

    My impression is that they watch a lot of film on a player. They like to find a couple of particularly illuminating plays: the LB was able to run deep with a wideout. Two guys with similar forty times ran different speeds on the same play. Etc…

    After building an overall idea of the player they go back and look at some of the specific causes of the successes/failures.

    Let’s say player blows two blocks in a game, and when they go and look at it they saw that he was bending at the waist, causing him to get off balance and lose power. Essentially he was lunging/falling towards the target instead of exploding through it by using a good football position (weight near toes, knees bent, waist bent, back straight, pad level low, shoulders over the toes… right?)

    Furthermore, after years of writing these reports along with the specific reasons for success of failure, you build some accountability. If there were three prospects you downgraded for waist bender, but they were all able to rectify the problem and build success, in the future you can know to de-emphasize the predictive value this trait.

  • takeitdown

    Ike: I think one of the problems with the draft is that there is so much information and little of it is helpful. GPA and test scores are highly correlated with graduation rates and college GPAs, and are decently correlated with career success. None of the numbers scouts use have such strong correlations, and, further, there really is almost too much data — a sort of paralysis of metrics that could be looked at, none of which are all that helpful, which sends you back into subjective “impression” mode. (Which again, is not necessarily a bad place to be, just not the optimal one.)

    I think the above quoted aspect is critical. On the NY times article, I wrote a grammatical error/typo riddled response, the crux of which is basically this:
    Stats and tests become useless (show no correlation) if not broken up into subgroups. When people set out to show tests or stats are useless, they use a stat against a very broad group. That hardly ever works due to noise. They show no correlation, and feel they’ve proven their point. In reality, the criteria for a 255lb back are different than for a 190 lb back, so you wouldn’t expect 3 cone drill alone, for example, to produce statistical significance, when you don’t even account for “type of back.” The 190 pound guy needs a way better 3 cone than the 255 lb back, to just be even.

    Stats don’t work that well for football, but they work remarkably better when utilized with a bit of a common sense approach. Like in population or pharmacological testing, find the relevant subgroup, and then look. If you give me two productive corners in college, with good hands, ability to track the ball, and what appears to be good 2 step burst…and then tell me one runs a 4.4 while another runs a 4.65, I can bet you very solid money the former one is going to be better. Sure, it’s clouded by some yahoo drafted in the 7th round who runs a 4.3 and was never a decent football player. But, in the designated subgroups (this guy’s already a quality player, in category x) the numbers start to have meaning and utility.

    The combine still has utility. There’s still some use in numbers before true sabermetrics come to town. It just requires a little more subtlety and parsing in football than baseball.

  • Brian

    The real problem is that the scouts on TV weren’t good enough to be real scouts on NFL teams or with the national scouting agencies like BLESTO; if they were you’d never know who they are because they’d be in a war room right now. I’ve met a couple of scouts and read their reports (if you want a good one, Russ Lande at GM Jr. sells a copy of his scouting reports), most of them are detailed and very thorough. From there, it’s a matter of team needs and preferences. The scouts on TV get a bad rap because they are in fact bad at evaluating talent, the good scouts are out actually finding talent.

    With that being said, bad draft picks are usually made because the GM overrides the scouts’ recommendations or the scouts fail to convince the GMs hard enough. Sometimes the decisions are made to sell tickets or to keep a division rival from having a player. Not saying the scout is always right (see: Tom Brady), but they get a hell of a lot more right than Kiper.

  • Old South

    I think there’s a simple analogy that helps to explain some things. A focus of the article was in reconciling the fact that scouts are horrible at predicting individuals yet pretty good at predicting the behavior of a group (i.e., first round picks ARE better than second round picks, on average).

    Players behave like stocks. Few people can invest and beat the market over the long run, just as scouts can barely beat reasonable chance in the situation of the draft. But we all know that index funds will almost always do well because they nullify the risks of individual variability by spreading it across a large, diverse group that covers the market. When you group your players/stocks into a draft round/index fund, you can see at a general level why one is consistently correct and another is high-risk high-reward.

  • Chad Phillips

    Great article. As a coach, I like to come here for a football fix during the off-season. I love the NFL draft, and I find it interesting how inexact of a science it is. One thing that I found really interesting was that the combine data only seemed to correlate with success for running backs. Another thing that interested me was that the higher the pick, the better the pick on average. I wonder how much of this is impacted by the fact that a 1st or 2nd rounder will get lots of opportunities to prove himself (and therefore rack up more games played and more stats) because of the amount of money they have invested. A late round guy might not make it past the first preseason game or two, essentially not given much of an opportunity at all. Just more food for thought.

  • Tyler

    Obviously, a coach is going to take issue with scouting done by non-coaches, as that has been taking place ever since the draft began. Your concerns are legitimately valid, but I think you are devaluing what scouts do and their importance to the process.

    Coaches like to think they have a better eye for talent, and they might, but they don’t have the time or resources to put things together like scouts do. Scouts attend multiple games, practices, and workouts every week while coaches are busy preparing their team. These are things the coaches simply cannot do, as the laws of time and space unfortunately restrict them. Sure, they can see the game tape, but they don’t see guys in person until the Combine, and they don’t see the little things that differentiate great players- attitude after good and bad plays, body movement relative to other athletes, competitiveness, coaching receptiveness, etc.

    In my experiences, good scouts spend more time looking at the little things that differentiate players more than the obvious (40 time, height, raw arm strength, etc.)

    Ultimately, it’s subjective, but so is scouting in baseball and basketball. While basketball scouting is relatively easier than the other sports, even the most sabermetrically inclined teams invest in scouting heavily. In fact, there’s been a shifting of the pendulum back towards scouting as an under-appreciated value because everyone is using the same statistics now.

  • Dave

    It’s a wonderful world to see such a wild combination of ideas in one place.

  • Mr.Murder

    “…that tells you nothing, because everyone has success under them.”
    Leach’s evil system is being balled out of the game, to save all of us from the dangers of stats as a basis of production.

    That argument would nullify the Value of a Jerry Rice. Everyone put up stats in the system he played.

    Wes Welker was playing brilliantly until injury. A Texas Tech “system guy” who did well for two pro teams.

    They need film and a variety of measurables to justify their investment. The workouts should confirm your observations, and distinguish players in similar groupings of ability and traits.

  • Coach H

    With baseball and football, talent evaluation will always be a subjective trade. If we could quantify each characteristic in which we deemed important then we would not need to hire scouts to run all over the country observing prospects. In baseball, the most overly-statistical sport in the world, you still need to send scouts to watch players because you won’t be able to quantify intangibles such as attitude, disposition, body language, team camaraderie, intensity, and performance in clutch situations.

    For football, I believe scouts do not put enough emphasis on evaluating players’ brains. I do not mean to only assess their intellectual capacity, but rather a combination of intellect, disposition, attitude, and emotional stability. The easiest example to name is Ryan Leaf. You could say the Chargers could have done a little more evaluation on Leaf, and they might have discovered that he is kind of a head-case.

  • for those interested, here is guide used by (taken from, actually) NFL scouts


  • 4.0 Point Stance

    Old South, I was kicking around a version of your “index fund” idea in my head this weekend while I busy not watching the draft. Could a team gain a competitive advantage by firing all its college scouts or reassigning them to scout opposing pro teams for actual, scoutable things like tendencies and schemes?

    Meanwhile you could create an “index fund” of player ratings based on a conglomeration of other peoples’ ratings, and pick based on that rating tempered with knowledge of your teams’ needs. You’d save a heck of a lot of money, time, and stress, and you’d probably end up with a comparable draft haul.

    Of course folks will go apoplectic because you’re missing out on “hidden gems” that only the genius scouts could find. The example I always hear in New Orleans is Marques Colston. But let’s be honest – the Saints didn’t actually think that highly of Colston, or they wouldn’t have waited until the 7th round to take him. The Saints lucked into his talent; they didn’t have some secret pre-draft knowledge.

  • srp

    I think the wisdom of crowds theory must survive competition from the halo effect theory, which says that higher picks get more playing time because of psychological biases and incentives facing the coaches and personnel folks. It’s a lot harder to admit a mistake by sitting a high pick and giving the reps to an undrafted free agent.

    Statistical analysis may improve analysis somewhat. I wonder whether the use of time-and-motion analysis–breaking player assignments down into their elementary movements and then seeing how quickly and accurately players can execute them–combined with videotape databases might add insight.

  • Just what I was looking for 😉

  • AnotherDave

    Great read .. thanks for tackling a difficult issue. I also agree with another poster above who said that baseball team’s are shifting back to blowing out their scouting network because everybody in baseball is using the same performance data now.

    I also think a lot can be taken from the defensive-statistic UZR (user zone rating) in baseball. The stat applies a zone to each part of the field and measures balls fielders should be expected to get to based on where the ball lands in each zone. I think this can be used in the NFL to measure defensive “range” or at least offer some fundamentals concepts that can be applied to different areas all in the interest of quantifying player performance.