Recently on a plane, the guy beside me was reading the same book I was – Malcolm Gladwell's Outliers. My fellow passenger didn't think this was remarkable as the airport bookstores had huge displays. Gladwell has become somewhat of a household name for his skill at popularizing social science through collecting compelling anecdotes. Blink and The Tipping Point were entertaining enough to read, and that's why the guy beside me had made an impulse purchase.
I had been more proactive in getting my hands on the book. We here at Criteria had actually been eagerly waiting for Gladwell's book ever since May when we stumbled across a truly odd speech he gave at a New Yorker conference. Gladwell's speech, which was explicitly delivered as a sneak preview of the book, covered what he called the "mismatch" problem. His thesis was that the way employers evaluate prospective employees — including the practice of pre-employment testing — is at a complete "mismatch" with what is required. As evidence he offered three loosely linked examples – sports combines where amateur athletes are evaluated before a draft; certification requirements for teachers; and the University of Michigan's affirmative action program for law school admissions.
Gladwell opens with anecdotes from the National Hockey League's pre-draft combine and then goes on to discuss the NBA and NFL drafts. He relates the finding that the aptitude test given to the NFL quarterbacks has no correlation with their performance. As usual, his evidence for this contention is entirely anecdotal. In a blog post in the spring, we described evidence showing that the test may be predictive of QB success, and it doesn't help Gladwell's case that he mocks the fact that Eli Manning and Tony Romo scored well on the test, while Vince Young and David Garrard did not. Even back in May 2008, he should have understood that his alleged exceptions weren't exactly disproving the rule. By the end of the opening example, Gladwell has shown that he probably doesn't know all that much about sports and has launched a puzzling, and difficult to support, argument. We were eager to see him make his point in print, but this line of thought didn't make it into the book.
In his speech, Gladwell next moved on to discussing teachers. We can't disagree with him that good teachers are important, or that it might be a good idea to broaden the pool from which new teachers are selected. But we were a bit confused by his argument about hiring standards. Teacher quality, he tells us, is much more predictive of student achievement than classroom size, and so it is worth investing in. But according to Gladwell, there are no criteria to predict whether someone will be a successful teacher. This is quite a jump, and again one is left with only colorful anecdotes for what is a very sweeping point with broad social significance.
Finally, Gladwell points out that even though the University of Michigan Law School had, in the past, given extra consideration to minority applicants and made concessions on testing standards, there was no evidence of differences in "success" years after graduation. Gladwell does not address that the admissions tests were not designed to predict success after graduation, but rather performance in the first year of law school. Nor does he address that there are statistical problems with evaluating a selection tool on a sample that was selected in the first place using that tool. But most important, Gladwell seems to be arguing that based on the experiences of the University of Michigan, law schools should abandon altogether their use of test scores to select applicants.
Well, somewhere between his May speech and the November publication of Outliers Gladwell must have realized he had his own mismatch problem. His evidence didn't match his thesis. He must have changed the direction of his book significantly because Outliers is barely relevant to employee selection. Instead, it's a motley collection of examples arguing that exceptionally successful people are not entirely self-made and that their ascent is due also to extraordinary good fortune with regard to the opportunities they were presented with. (This is hardly a shocker of a thesis, and is right up there with Blink's bold contention that first impressions are often correct, except when they are not.) But where's the employee selection angle? Where his speech proclaimed that it was "time to shut down the combines", his book only discusses the mildly interesting fact that the birthdays of professional athletes tend to cluster near-critical cut-off dates for selection into elite programs. Where he promised to show how the process of hiring and training pilots is completely at odds with what is required, he only ends up discussing how cultural factors can have a tragic influence on the dynamics between pilots in a cockpit. Rather than discuss the process of hiring teachers, he describes one successful charter school, and the demands it makes of its students. He also speculates on cultural and linguistic factors that might correlate with math perseverance.
We're left to wonder why the change in focus from the speech to the book... could it be Gladwell realized that after all there is substantial evidence for the effectiveness of aptitude testing as a predictor of job success? Or did he just decide that "what determines exceptional success?" is a more interesting question (and easier to write about) than "how can we hire better?"
