Don't Ask, Don't Tell: The New Rules of the SAT and College Admissions

Today's blog post is by Dr. Howard Wainer, who is the Distinguished Research Scientist at the National Board of Medical Examiners, as well as Professor of Statistics at the Wharton School of the University of Pennsylvania. Dr. Wainer received his Ph.D. from Princeton University, has won numerous scholarly awards, and spent 21 years as Principal Research Scientist in the Research Statistics Group at the Educational Testing Service. He is also, as far as we know, the only member of Criteria's Scientific Advisory Board to have swum the English Channel.

On September 22, 2008, the New York Times carried the first of three articles about a report, commissioned by the National Association for College Admission Counseling, that was critical of the current college admission exams, the SAT, and the ACT. The commission was chaired by William R. Fitzsimmons, the dean of admissions and financial aid at Harvard.

The report was reasonably wide-ranging and drew many conclusions while offering alternatives. Although well-meaning, many of the suggestions only make sense if you say them fast.

Among their conclusions were:

1. Schools should consider making their admissions "SAT optional," which is allowing their applicants to submit their SAT/ACT scores if they wish, but they should not be mandatory. The commission cites the success that pioneering schools with this policy have had in the past as proof of concept.
2. Schools should consider eliminating the SAT/ACT altogether and substituting instead of achievement tests. They cite the unfair effect of coaching as the motivation for this — they weren't naive enough to suggest that because there was no coaching for achievement tests now that, if they became more high stakes coaching for them would not be offered. Rather, they argued that such coaching would be related to schooling and hence more beneficial to education than is coaching that focuses on test-taking skills.
3. That the use of the PSAT with a rigid qualification cut-score for such scholarship programs as the Merit Scholarships be immediately halted.

I will not attempt to discuss all three of these here, just the first one — if there is sufficient interest shown in this topic this entry will be followed by others.

Has the admissions process been hampered in schools that have instituted an SAT optional policy?

The first reasonably competitive school to institute such a policy was Bowdoin College, in 1969. Bowdoin is a small, highly competitive liberal arts college in Brunswick, Maine. A shade under 400 students a year elect to matriculate at Bowdoin, and roughly a quarter of them choose not to submit their SAT scores. In Table 1 is a summary of the classes at Bowdoin and five other institutions whose entering freshman class had approximately the same average SAT score. At the other five institutions the students who didn't submit SAT scores used ACT scores instead.

To know how Bowdoin's SAT policy is working we will need to know two things. First, how did the students who didn't submit SAT scores do at Bowdoin in comparison to those students that did submit them? And second, would the non-submitters performance at Bowdoin have been predicted by their SAT scores?

The first question is easily answered by looking at their first year grades at Bowdoin. These are shown in Figure 1 below.

Bowdoin students who did not send their SAT scores performed worse in their first year courses than those who did submit them.

Image

We see that non-SAT submitters did about a standard deviation worse than students who did submit SAT scores. And so, we can conclude that if the admissions office were using other variables to make up for the missing SAT scores, those variables did not contain enough information to prevent them from admitting a class that was academically inferior to the rest.

But would their SAT scores have provided information missing from other submitted information? Ordinarily, this would be a question that is impossible to answer, for these students did not submit their SAT scores. However, all of these students actually took the SAT, and through a special data-gathering effort at the Educational Testing Service, we find that the students who didn't submit their scores behaved sensibly. Realizing that their lower than average scores would not help their scores at Bowdoin, they chose not to submit them. Below (Figure 2) is the distribution of SAT scores for those who submitted them as well as those who did not.

Those students who don't submit SAT scores to Bowdoin score about 120 points lower than those who do submit their scores.

As it turns out the SAT scores for the students who did not submit them would have accurately predicted their lower performance at Bowdoin. In fact, the correlation between grades and SAT scores was 12% higher for those who didn't submit them than for those who did.

So not having this information does not improve the academic performance of Bowdoin's entering class — on the contrary, it diminishes it. Why would a school opt for such a policy? Why is less information preferred to more? There are surely many answers to this, but one is seen in an augmented version of Table 1 (below).

We see that if all of the students in Bowdoin's entering class had their SAT scores included the average SAT at Bowdoin would sink from 1323 to 1288, and instead of being second among these six schools they would have been tied for next to last. Since mean SAT scores are a key component in school rankings, a school can game those rankings by allowing their lowest scoring students to not be included in the average. I believe that Bowdoin's adoption of this policy pre-dates US News and World Report's rankings, so that was unlikely to have been their motivation, but I cannot say the same for schools that have chosen such a policy more recently.