Gas station without pumps

2011 December 10

Is SAT prep worthwhile?

Filed under: Uncategorized — gasstationwithoutpumps @ 14:35
Tags: , , , ,

A recent NY Times Room for Debate discussion, Why Should SAT’s Matter, had a rather diffuse focus.  Half the participants seemed to be discussing whether the SATs were of value in predicting academic success, the other half seemed to be discussing whether expensive SAT prep courses were worth anything.

The psych professors said that the SAT is basically an IQ test and does a pretty good job (by the standards of psychometric measurements) of predicting who will do well in college—not just freshman year but beyond.

Paul Siemens (from a testing company) claimed that the SAT was really an achievement test, testing knowledge of high school subjects. He gave no evidence for this claim, and it seems to be pretty well refuted by the high scores of 7th graders taking SATs through talent searches.  An examination of the content of the SAT questions also shows that they don’t rely much on any high school content.

The one college admissions officer, Alan T. Paynter, recited the standard dogma of tests being just a part of the admissions process, and students with poor test scores emphasizing other aspects of their application.

Overall, the discussion did not have much of the flavor of a debate (unlike some Room for Debate discussions).  People weren’t addressing the same question for most of what they wrote.  The one part in common, whether expensive SAT prep courses were worthwhile, got general agreement—they are a waste of money.

There is considerable debate and discussion in the comments, but most of it anecdotal and unsupported by any research.  A lot of the comments are very definitely self-serving:  People selling test prep claiming that they produce gains of hundreds of points (claims not validated in any controlled studies done by people other than those selling expensive prep courses).

It would be good to have a debate between informed people with real data about whether SAT prep does anything, and whether expensive courses are any better than cheap prep books.  Personal belief: test prep mainly helps at the low end, where students don’t have any idea how to take a test—I’m not going to be investing in any test prep for my son, who can do this sort of test cold without much difficulty. He took the SATs in 6th grade as an entrance requirement for a gifted program and did better than the graduates of many test-prep classes. In 10th grade, he missed one question on the PSAT, lowering his “writing skills” section from 80 to 75.  The PSAT has such a low ceiling that one question can move you 5 scale points (that would be 50 points on the SAT).  It is ludicrous that they use it for the National Merit Scholarship Qualifying Test—in my day the NMSQT was a separate test, much better at distinguishing among the top students than the PSAT, but that is a rant for a different day.

It might also be good to have a debate between those who claim that the SAT is basically an IQ test and those who claim that it is basically an achievement test, assuming that anyone with reasonable training in psychometrics can be found to back the “achievement” side.  Based on what I’ve seen of it, the SAT is somewhere in between, with most of the math questions requiring intelligence but only a little core knowledge, and many of the reading questions requiring mainly vocabulary and only moderate intelligence.  I assume that people have looked at the correlation between “g” (the global intelligence construct that IQ tests often have as their main factor) and the various SAT scores, and that it is reasonably high.  But achievement tests are also highly correlated with “g”, so this does not immediately tell us whether the SAT is more like an IQ test (testing general reasoning ability) or an achievement test (testing mastery of specific material).

2010 November 29

Not prepping for SAT

Filed under: Uncategorized — gasstationwithoutpumps @ 00:02
Tags: , , ,

A common question that comes up for parents and teachers of high-school students is “How much test preparation is appropriate before taking the SAT or ACT tests?”  The question is not an easy one to answer, as there are several related questions that are mixed together in often confusing ways.

  • Do the tests really matter?
  • Is it ethical to prepare for the SAT or ACT? Or should the exams be taken “cold”, like IQ tests?
  • Does test preparation improve scores on the tests?
  • What sort of test preparation is most effective?
  • Is time and money spent on test preparation better invested in some other pursuit?

I’ll try to give my opinion on all these, backing it up with research results where I’m aware of any.

Do the tests really matter?

Newsweek recently published an article “Going SAT-Free” that reported on several top colleges no longer requiring SAT or ACT tests: “about 830 of the country’s 2,430 accredited four-year colleges do not use the SAT or ACT to admit the majority of applicants. (Some schools require a test if you have a low GPA or class rank.)”

That leaves about 2/3 of colleges (including most public colleges) still requiring either the SAT or the ACT.  So students need to take the exam.  How important a high score on the exam is depends on the specific college, but in general, higher scores will translate to a higher probability of getting in.  (58% of colleges in one survey reported that the test scores were of considerable importance in admissions decisions.)

Is test prep ethical?

Some people have likened SAT tests to IQ tests, which present test takers with unfamiliar questions in order to determine how well they think.  Preparing for an IQ test invalidates the test, rendering the resulting scores meaningless.  Can the same be said for the SAT and ACT?

First, neither the SAT nor the ACT is attempting to measure IQ or other supposedly static properties of the test taker.  They are intended to measure the student’s preparation for college work and predict how well they will do in college.

For achievement tests, the whole point of the test is to measure how much a student has learned—to measure the total learning that the student has acquired over several years.  Preparation for achievement tests is essential, as what they are measuring is how well prepared the student is.

SAT and ACT tests are neither ability tests nor achievement tests, but a mix of the two concepts.  They measure a combination of what the student knows (vocabulary, math concepts, and so forth) and how well they can solve simple puzzles using that knowledge.  Given that the tests measure knowledge, it is not only ethical for students to prepare for the tests, but essential that they do so in some form.

The ethics question is then reduced to determining whether it is fair for wealthy students to spend more on preparing for the tests than poor students can.  Since the best preparation for the exams is a good education for the preceding 10 years, it would be very difficult to eliminate the effects of wealth.  Indeed, access to a superior education is one possible definition of wealth, independent of more conventional financial measures.

I can only conclude that preparing for the SAT and ACT tests is ethical.

Does test prep help?

The biggest debate seems to be about how coachable the SAT and ACT test scores are.  There is little doubt that students who have had 10 years of excellent education do much better than students who have had 10 years of execrable education.  The debatable question is whether short courses on content or coaching on test-taking techniques make any difference.  There is a multibillion dollar test-preparation industry, so there is a lot of incentive for marketers to sell snake oil.

The best report I’ve found analyzing the actual effectiveness of test preparation is Derek C. Briggs’s paper Preparation for
College Admission Exams, published by the National Association of College Admission Counseling.  It looks at all the published research on the topic and concludes that “Contrary to the claims made by many test preparation providers of large increases of 100 points or more on the SAT, research suggests that average gains are more in the neighborhood of 30 points.”

Of course, even just retaking a test, with no intervening coaching, can improve tests scores (on average about 15 points per section as reported by the College Board).  This report on change in averages can be misleading, since students at the ends of the distribution are likely to move towards the middle on retaking (a phenomenon known as regression to the mean), so that top students should not expect any boost from retaking the test. Briggs estimates the “coaching effect”, how much bigger the gain is from coaching than from retaking the exam without coaching:

  • Coaching has a positive effect on SAT performance, but the magnitude of the effect is small.
  • The effect of coaching is larger on the math section of the exam (10–20 points) than it is for the critical reading section (5–10 points).
  • There is mixed evidence with respect to the effect of coaching on ACT performance. Only two studies have been conducted. The most recent evidence indicates that only private tutoring has a small effect of 0.4 points on the math section of the exam.

Briggs later says “From a psychometric standpoint, when the average effects of coaching are attributed to individual students who have been coached, these effects cannot be distinguished from measurement error. … On the other hand, if marginal college admission decisions are made on the basis of very small differences in test scores, a small coaching effect might be practically significant after all.”

This raises the question of whether 20–30 points is going to make a difference in admissions decisions.  Briggs looked at that question also.  At the low end of the scale, a 20-point difference in SAT score would not affect admissions, but  at the high end (600–750), 40% of surveyed college admissions officers thought a 20-point difference for math or a 10-point difference for critical reading would affect chances, but only 20% thought  a 20-point difference for the writing section would change the probability of admission.

It appears that the effect of short-term test preparation is small, but that admissions officers are looking at differences in scores that are well below the noise level of the tests, so retaking tests in the hopes of getting a higher score randomly could be worthwhile, and test preparation could increase the chance of increasing one’s score enough to affect admissions decisions.

So it looks like doing some test prep may improve scores.

What sort of test prep is most helpful?

Of course, just because some test prep is worthwhile does not mean that any specific course is worthwhile. Briggs distinguishes between student-driven prep (using the example questions provided by the test publishers or studying content and doing sample tests from books) and coaching with a live teacher.

Briggs reports that “No forms of test preparation had statistically significant positive effects on SAT-V scores” and that books, courses, and tutors all had small positive effects on SAT-M.  (Interestingly, use of a computer prep program had a small negative effect.)

Is test prep cost-effective?

Briggs says “Beyond that which occurs naturally during students’ years of schooling, the only free test preparation is no
test preparation at all. This is because all test preparation involves two costs: monetary cost and opportunity
cost.”

The financial costs are easy to analyze. Given the small gains from commercial coaching courses and the roughly similar gains from using a test prep book, there doesn’t seem to be much financial sense to paying for the much more expensive commercial courses.  The books are cheap (and readily available from libraries) so there seems no financial barriers to using them.

The opportunity cost is the time spent on test prep that might more usefully have been spent studying for classes, sleeping, or doing extracurricular activities (like sports, theater, or community service).  Here the analysis is more difficult, but I think favors spending fairly little time on test prep.  Time spent pursuing a passion or serving the community is more likely to improve one’s chances of admission to college than small gains in test scores will.  Of course, time wasted hanging out at the mall or playing video games is unlikely to have any positive effect.

What will we do?

Since my son did very well on the SAT math and critical reading sections when he took them in 6th grade (over 700), I see no reason for him to waste time on test prep for those sections.  He did less well on the writing portion, getting the lowest possible score on the essay, but at the time he had never had instruction in timed essay writing, and had never even heard of the 5-paragraph essay so beloved of SAT graders.  I expect that he will need little prep for the essay writing also, as he will undoubtedly get more practice on the 5-paragraph essay than any sane person could stand in his high school classes.

2010 August 28

Test prep for kindergarten?

Filed under: Uncategorized — gasstationwithoutpumps @ 17:51
Tags: , ,

Every year at the beginning of the school year, there are a round of stories about the cut-throat preparation of kids for entry tests for kindergarten (like this one, a few weeks ago).

Test prep for kindergarten seems to be limited mostly to New York City, thanks to a dysfunctional way of assigning kids to schools and the sincere but misguided belief by many New York parents that where a child goes to kindergarten determines their entire scholastic career through graduate school.  Those of us in the rest of the country are rather bemused by the media frenzy over what is really a very local phenomenon. The teacher matters much more than the school does, especially in the early grades, so the scramble to get into a “good school” is a bit silly even if understandable.

I’ve always been rather bothered by the “test preparation” industry.  Mostly they are selling snake oil, so they don’t do too much harm other than taking people’s money.  Occasionally they are selling ways to cheat (like access to the questions that are asked on particular IQ or school entrance exams, which are not supposed to be available to test-takers ahead of time), and then they really are harmful.

Test preparation for kindergarteners is particularly ridiculous, as testing kindergarteners is ridiculous.  There are very few psychometric tests that are reliable for kids under the age of 6.  The things a child needs to be ready for kindergarten are fairly simple and do not need testing to determine.  Furthermore, there is no evidence that a heavily academic kindergarten really offers much advantage to a bright student.

When my son was entering kindergarten (very bright and already reading), I looked for the best teachers I could find.  Although we looked a private school for gifted kids, we ended up with a bilingual program at a public school, because they had some superb teachers, lots of books in the classroom, plenty of library time, and the extra instruction in Spanish would provide something novel for him to learn.  That year I took a sabbatical in the spring in a different city, so we had to do another search for kindergartens.  We looked at several, including both public and private schools, and ended up with a public school that served a very poor neighborhood (though it was temporarily located in another building halfway across the city, due to renovation work at the school).  The reason again was an awesome teacher, a very rich environment in the classroom, and lots of books for him to read.  The range of books to read was important, as he was reading at a high 2nd-grade level by the middle of kindergarten.

%d bloggers like this: