The futility of measurement, NFL combine edition

From Jonah Lehrer in the WSJ:

. . . We live in a society obsessed with maximum performance. Think of exams like the SAT and the GRE. Though these tests take only a few hours, they’re supposed to give schools and companies a snapshot of an individual’s abiding talents.

Or consider the NFL Scouting Combine, in which players entering the draft perform short physical and mental tasks, such as the 40-yard dash. The Combine is meant to measure physical ability; that’s why teams take the results so seriously.

It’s easy to understand the allure of such maximal measures. They don’t take very long, so we can quantify many people. Also, they make assessment seem relatively straightforward, reducing the uncertainty of selecting a college applicant or football player.

But as Mr. Sackett demonstrated with those supermarket cashiers, such high-stakes tests are often spectacularly bad at predicting performance in the real world. . . .

Even the NFL Combine is a big waste of time. According to a recent study by economists at the University of Louisville, there’s no “consistent statistical relationship” between the results of players at the Combine and subsequent NFL performance.

The reason maximal measures are such bad predictors is rooted in what these tests don’t measure. It turns out that many of the most important factors for life success are character traits, such as grit and self-control, and these can’t be measured quickly.

Consider grit, which reflects a person’s commitment to a long-term goal. As Angela Duckworth, a psychologist at the University of Pennsylvania, has demonstrated, levels of grit consistently predict levels of achievement, such as graduation from West Point and success in the National Spelling Bee.

The problem, of course, is that students don’t reveal their levels of grit while taking a brief test. Grit can only be assessed by tracking typical performance for an extended period. Do people persevere, even in the face of difficulty? How do they act when no one else is watching? Such traits often matter more than raw talent. We hear about them in letters of recommendation, but hard numbers take priority.

The larger lesson is that we’ve built our society around tests of performance that fail to predict what really matters: what happens once the test is over.

  • Silkyice

    I think the orginal premise of the combine was sound. Scouts couldn’t see all the athletes play. Film and TV was not as available. You might need to compare players from different divisions (D2 and D1) or different conferences. But now, you can watch every single play a player has made over the college career right on your laptop.

    I do believe there are certain minimum requirements to be succesful in the NFL. But once those are met, their actual previous performance on the field matters tremendously more. My guess is that almost everyone who is in the combine probably meets those minimums anyway.

    Here is a baseball analogy. We will have a combine for pitchers. We will measure who throws the fastest. Someone might throw 98. Does that mean he is a better pitcher than someone who throws 92? Now if someone can only throw 72, then sure, the guy that throws 98 will be more succesful. But if someone throws 72, he’s not getting in the combine anyway.

    For instance, NFL lineman probably need to be able to bench press 225 25 times or more. If someone can only do it 10, then they might struggle in the NFL. If an All-American can do it 30, and someone else can do it 45, I don’t for one minute think that the All-American is inferior. Unless the game tape shows that he is.

    I think Auburn’s 3rd string running back had a better 40 time than Mark Ingram in the combine. One would be an idiot to take him over Mark Ingram in the draft. Now if Ingram’s forty time was 5.1, then maybe I would consider it.

    I do agree that the combine does not measure toughness and grit. I do believe that those are extremely important. But that isn’t the reason I think the combine is a waste of time. I believe that after certain requirements are met (which they all meet), combine numbers pale in comparison to game tape.

    I see two values in the combine. One, to make sure a stud from d2/d3 can really measure up. Two, to measure commitment in some form in that the player is serious enough about the NFL and working hard, that he shows up in shape for the combine.

  • Rob Pitzer

    I doubt even its most ardent defenders would suggest that the combine should be viewed in isolation. But as part of the overall picture it’s invaluable.

    The much-maligned 40 time for example… the entire range for legit RB prospects is typically something like 4.30 to 4.70. So each tick represents 1/40th the range and 4/100ths represents a full 10%. That’s relevant (at least to the extent that the measurements are accurate [or at least accurate relative to one another]). And knowing whether players with similar collegiate performance are in the 80th or the 20th percentile on speed can be very important. Particularly as you approach the middle of the pack in terms of NFL-worthy prospects.

    As for the “grit” argument… most (not all) players who make it far enough to snag a combine invite have already demonstrated an outstanding level of competitiveness, work ethic and etc. Making it as a major college football player is incredibly difficult for most in and of itself.

    IOW… it’s become so fashionable to dis the combine and the quantitative measures there that it’s cliched. But it’s also dead wrong. The combine tests are absolutely predictive (at least for RBs, WRs and TEs) once you incorporate all of the other information you have (such as work ethic, character, injury history, toughness, collegiate peformance and etc.)

  • ken kennedy

    I have always said the combine measures athletes but not neccesarily football players.

  • Jim

    I am with Rob on this the combine gets a lot of crap because that is the thing to do. Of course game film is more important but than I don’t you will find any of the top teams and for that matter most teams that will disagree. The combine is to see if what they are seeing on film matches up and to see if players actually will work hard. If you have a QB for example show up and bomb the wunderlick that will bring up questions on why he did not prep and if this will carry over. It also gives the team a chance to look at medical problems and how they are progressing as well as a chance to talk to the player in an interview. That some teams misuse this tool is not the fault of the tool itself.

  • Brian

    HR departments have known for years that interviews are the worst predictor of job success. Work-related tests are far better. Future NFL players have a built-in test: it’s called college football. I can’t imagine a player becoming a better player in the pros than college because he excelled at the combine. The tape doesn’t lie.

  • Anonymous

    The combine is useful inormation. The perception of the combine as vital is the flaw. The overemphasis on one or two days of drills versus years of actual college practice and play makes no sense. This is compounded by the salary structure that is based on selection order instead of potential skill. A guy with a four year record of excellent play can have two bad days or even be injured at the wrong time and it costs him a lot of money. A guy with a sketchy performance record suddenly becomes a superstar and cashes in. Same thing in fantasy football – how many players are selected based on last week as opposed to a longer period of performance?

  • Chase

    Thanks for the link, Chris. A couple of points:

    The combine measures grit and work ethic, and it also measures perhaps even more important traits: desire, ambition, responsibility and maturity.

    1) You can’t have a terrible work ethic and kill it at the combine. If a guy has a tremendous workout, you know exactly how he performs when no one watches: he puts in the work. Now putting in the work at the gym doesn’t mean you will be a good NFL player, but I think it’s silly to act like these guys wake up one day and do the combine. It’s the culmination of years of gym-training. And I think in large groups, the players with the best combines have better work ethics and are more dedicated to their training than the players with the worst combines. It’s not all — or even mostly — natural ability on display.

    2) If you can’t bust your ass to do well at the combine, what does that say about you? A weak combine could be an indicator that the player just didn’t take the combine that seriously. And given the importance scouts and GMs place on the combine, what does it say about an athlete that doesn’t take it seriously? In the real world, people do incredibly stupid tasks all the time for the good of their company. They don’t like it, but they do it because it’s their job. Well, for these players, it’s their job to do as well as possible at the combine. It’s got to be eye-raising if you see an athlete that didn’t take the combine seriously.

    For me, a terrible combine speaks more bad than a strong combine speaks good. But it’s certainly a valuable tool. Query: If the NFL eliminated the combine, do you think the NFL draft would be a more or less efficient market? Based on everything I wrote above, I believe it would be a less efficient one. If you feel the same way, then you have to think the combine does have some value.

  • Guest

    The only things that are important in the combine are interviews and medical evaluations. The workouts are taken within context of game films performance. Game films performances often overrides combine workouts.

  • Art

    I’d say you are right 90% of the time, but Sam Shields for the packers is a prime example of a kid who made it in the NFL based solely on combine numbers. His game day tape was average at best, but elite speed + plus solid coaching at the NFL level made him into a extremely good rookie with tons of upside.

  • The Fridge

    Interesting observation about what kind of society we live in. Thank you for taking the time to show your perspective.