Gas station without pumps

2010 November 29

Not prepping for SAT

Filed under: Uncategorized — gasstationwithoutpumps @ 00:02
Tags: , , ,

A common question that comes up for parents and teachers of high-school students is “How much test preparation is appropriate before taking the SAT or ACT tests?”  The question is not an easy one to answer, as there are several related questions that are mixed together in often confusing ways.

  • Do the tests really matter?
  • Is it ethical to prepare for the SAT or ACT? Or should the exams be taken “cold”, like IQ tests?
  • Does test preparation improve scores on the tests?
  • What sort of test preparation is most effective?
  • Is time and money spent on test preparation better invested in some other pursuit?

I’ll try to give my opinion on all these, backing it up with research results where I’m aware of any.

Do the tests really matter?

Newsweek recently published an article “Going SAT-Free” that reported on several top colleges no longer requiring SAT or ACT tests: “about 830 of the country’s 2,430 accredited four-year colleges do not use the SAT or ACT to admit the majority of applicants. (Some schools require a test if you have a low GPA or class rank.)”

That leaves about 2/3 of colleges (including most public colleges) still requiring either the SAT or the ACT.  So students need to take the exam.  How important a high score on the exam is depends on the specific college, but in general, higher scores will translate to a higher probability of getting in.  (58% of colleges in one survey reported that the test scores were of considerable importance in admissions decisions.)

Is test prep ethical?

Some people have likened SAT tests to IQ tests, which present test takers with unfamiliar questions in order to determine how well they think.  Preparing for an IQ test invalidates the test, rendering the resulting scores meaningless.  Can the same be said for the SAT and ACT?

First, neither the SAT nor the ACT is attempting to measure IQ or other supposedly static properties of the test taker.  They are intended to measure the student’s preparation for college work and predict how well they will do in college.

For achievement tests, the whole point of the test is to measure how much a student has learned—to measure the total learning that the student has acquired over several years.  Preparation for achievement tests is essential, as what they are measuring is how well prepared the student is.

SAT and ACT tests are neither ability tests nor achievement tests, but a mix of the two concepts.  They measure a combination of what the student knows (vocabulary, math concepts, and so forth) and how well they can solve simple puzzles using that knowledge.  Given that the tests measure knowledge, it is not only ethical for students to prepare for the tests, but essential that they do so in some form.

The ethics question is then reduced to determining whether it is fair for wealthy students to spend more on preparing for the tests than poor students can.  Since the best preparation for the exams is a good education for the preceding 10 years, it would be very difficult to eliminate the effects of wealth.  Indeed, access to a superior education is one possible definition of wealth, independent of more conventional financial measures.

I can only conclude that preparing for the SAT and ACT tests is ethical.

Does test prep help?

The biggest debate seems to be about how coachable the SAT and ACT test scores are.  There is little doubt that students who have had 10 years of excellent education do much better than students who have had 10 years of execrable education.  The debatable question is whether short courses on content or coaching on test-taking techniques make any difference.  There is a multibillion dollar test-preparation industry, so there is a lot of incentive for marketers to sell snake oil.

The best report I’ve found analyzing the actual effectiveness of test preparation is Derek C. Briggs’s paper Preparation for
College Admission Exams, published by the National Association of College Admission Counseling.  It looks at all the published research on the topic and concludes that “Contrary to the claims made by many test preparation providers of large increases of 100 points or more on the SAT, research suggests that average gains are more in the neighborhood of 30 points.”

Of course, even just retaking a test, with no intervening coaching, can improve tests scores (on average about 15 points per section as reported by the College Board).  This report on change in averages can be misleading, since students at the ends of the distribution are likely to move towards the middle on retaking (a phenomenon known as regression to the mean), so that top students should not expect any boost from retaking the test. Briggs estimates the “coaching effect”, how much bigger the gain is from coaching than from retaking the exam without coaching:

  • Coaching has a positive effect on SAT performance, but the magnitude of the effect is small.
  • The effect of coaching is larger on the math section of the exam (10–20 points) than it is for the critical reading section (5–10 points).
  • There is mixed evidence with respect to the effect of coaching on ACT performance. Only two studies have been conducted. The most recent evidence indicates that only private tutoring has a small effect of 0.4 points on the math section of the exam.

Briggs later says “From a psychometric standpoint, when the average effects of coaching are attributed to individual students who have been coached, these effects cannot be distinguished from measurement error. … On the other hand, if marginal college admission decisions are made on the basis of very small differences in test scores, a small coaching effect might be practically significant after all.”

This raises the question of whether 20–30 points is going to make a difference in admissions decisions.  Briggs looked at that question also.  At the low end of the scale, a 20-point difference in SAT score would not affect admissions, but  at the high end (600–750), 40% of surveyed college admissions officers thought a 20-point difference for math or a 10-point difference for critical reading would affect chances, but only 20% thought  a 20-point difference for the writing section would change the probability of admission.

It appears that the effect of short-term test preparation is small, but that admissions officers are looking at differences in scores that are well below the noise level of the tests, so retaking tests in the hopes of getting a higher score randomly could be worthwhile, and test preparation could increase the chance of increasing one’s score enough to affect admissions decisions.

So it looks like doing some test prep may improve scores.

What sort of test prep is most helpful?

Of course, just because some test prep is worthwhile does not mean that any specific course is worthwhile. Briggs distinguishes between student-driven prep (using the example questions provided by the test publishers or studying content and doing sample tests from books) and coaching with a live teacher.

Briggs reports that “No forms of test preparation had statistically significant positive effects on SAT-V scores” and that books, courses, and tutors all had small positive effects on SAT-M.  (Interestingly, use of a computer prep program had a small negative effect.)

Is test prep cost-effective?

Briggs says “Beyond that which occurs naturally during students’ years of schooling, the only free test preparation is no
test preparation at all. This is because all test preparation involves two costs: monetary cost and opportunity
cost.”

The financial costs are easy to analyze. Given the small gains from commercial coaching courses and the roughly similar gains from using a test prep book, there doesn’t seem to be much financial sense to paying for the much more expensive commercial courses.  The books are cheap (and readily available from libraries) so there seems no financial barriers to using them.

The opportunity cost is the time spent on test prep that might more usefully have been spent studying for classes, sleeping, or doing extracurricular activities (like sports, theater, or community service).  Here the analysis is more difficult, but I think favors spending fairly little time on test prep.  Time spent pursuing a passion or serving the community is more likely to improve one’s chances of admission to college than small gains in test scores will.  Of course, time wasted hanging out at the mall or playing video games is unlikely to have any positive effect.

What will we do?

Since my son did very well on the SAT math and critical reading sections when he took them in 6th grade (over 700), I see no reason for him to waste time on test prep for those sections.  He did less well on the writing portion, getting the lowest possible score on the essay, but at the time he had never had instruction in timed essay writing, and had never even heard of the 5-paragraph essay so beloved of SAT graders.  I expect that he will need little prep for the essay writing also, as he will undoubtedly get more practice on the 5-paragraph essay than any sane person could stand in his high school classes.

About these ads

12 Comments »

  1. My child did well too when taking the SATs in 6th grade. PSAT scores in 10th grade were along similar lines. A modest amount of prep was done before the PSATs in 11th grade since being sure to make National Merit Semifinalist seemed like a reasonable return for about ten hours of work. Similiarly, about ten hours of work was done before taking junior year spring SATs. This was mostly in the form of 5 or 10 minutes of vocabulary study most days, one practice test, and some reading and study in the exam book. My child’s final SAT score was great and we were glad to be ‘one and done’. Would the scores have been different (aka lower) without the work? Hard to say. My child thinks that it helped – and, for the elite schools, the difference between 720 and 780 may matter.
    Based on the very random data I have – about ten kids, mostly but not all pretty smart, prep helps some but not all. It seems to be not so much that each kid gains a little but that some gain none and some gain a lot. It seems to help kids who are nervous, perhaps merely by making them feel more confident. It helps kids who are very unfamiliar with the test.
    My theory is that just working with the books is as good as having a course for many kids, but some who just won’t study on their own can benefit from a course. I suspect that courses for lower class kids could help a great deal since they will know the least about the test.
    I tend to think that prep is not dishonest – admissions folks are surely very aware of who is likely to prep extensively. (ANd, in fact, I’ve wondered if my child’s scores will be at least slightly discounted because of the school attended and our zipcode!) And, the genie is well out of the bottle – there is no way to have kids take the tests cold now – making test prep books that are pretty cheap (particularly if you just buy last year’s book which is always dirt cheap at the used book store) is actually a reasonably equalizer.

    Comment by sharon stanfill — 2010 November 29 @ 07:27 | Reply

  2. I created a low cost SAT prep class in 1979 with predominantly MIT undergrads as the teachers. While it is still low cost compared to most programs, 31 years later the cost has crept up towards $100, which makes me cringe, but unlike the other programs run by MIT ESP, SAT Prep cannot run without paying its staff. Still, the fin aid is trivial – I know of few cases in which anybody has been turned down.

    One of the motivations for starting it was a Boston office FTC report (not formally released, I believe) that concluded – to their surprise – that SAT coaching helped. Once I knew that it helped wealthier kids, then poorer kids had to be given the option, even if it would not, as you point out, undo a decade of habits.

    The background weakness was a far bigger issue in the verbal areas than in math, simply because raw vocabulary is a lot harder to “teach” in a short period than any other single skill on the tests, including reasoning.

    My experiences, both in direct tutoring and in running that program, in how much rise one can expect are at sharp deviance from the 15 points per section or 30 points per score area, though there are some caveats.

    The biggest thing that drags most kids down in their SAT coaching is a complete and utter lack of motivation – being forced to do this works no better than being forced to do almost anything else. The second biggest factor, assuming they are actually present and motivated, is how many different things we have to focus on in our time together. A moderately high scorer may have one or two areas of weakness per test or may be broadly weak. If, for example, on the math the majority of questions missed were in geometry, then raising the math score is easier. Ditto the reading or sentence completion or other such. If, instead, the student missed two questions in each of several subcategories, then it is far harder to make a meaningful jump. It takes more poking to determine what is a weakness vs. what is a carelessness or brief glitch.

    I prefer to get to students before they have formally taken the SAT the first time (in high school), because that increase will have a far greater impact on the admissions process than said increase would on a retest, as the colleges get to see the earlier test scores as well as the later, and it has far less impact on an admissions scenarios…

    But honestly, I work very hard to ensure that the students I work with have a realistic understanding of the level of work they are comfortable with, what is a stretch for them and how likely they are to go for it given the chance vs. pulling back in frustration. That varies widely from student to student.

    Being able to get into a college you are underprepared for is just not always a benefit.

    Comment by Josh Shaine — 2010 November 29 @ 08:10 | Reply

  3. I think it makes sense to prep for the specific elements of the test that would otherwise be unfamiliar — how to pace oneself, what kinds of problems are most common, how to eliminate wrong answers most easily, that kind of thing. I don’t think that takes very long for a student who already has the appropriate subject knowledge. For a student with extreme test anxiety, a formal prep class *may* be a good way to work on desensitization (though I can imagine it backfiring, too). I think in some cases the direct instruction involved in test-prep classes does do some good for students who have had very “fuzzy” instruction in their regular classes, but otherwise it’s a bit late for actual subject tutoring to do much.

    Comment by Helen — 2010 November 29 @ 09:44 | Reply

  4. It also helps to get kids used to simply sitting and working on a test for as long as the SAT takes. It’s a really long test these days and that is something many kids are not at all familiar with.
    Vocabulary is tough. We did a very small number of words each day by picking random ones from a flash card box until we hit unfamiliar ones. We redid the new ones for a couple of days to fix them well. I don’t know if it mattered for the SAT but it was generally good to do. The vocabulary stuff is tricky too – I’ve seen writing from kids who clearly crammed a lot of vocab without ever getting the nuances of the words – their writing stinks because of that lack of nuance.
    We also looked at the explanations of answers – that way , my child got how the SAT writers were thinking which was helpful.

    Comment by sharon stanfill — 2010 November 29 @ 10:55 | Reply

    • My son did not find the vocabulary difficult. Of course, he reads a lot, including a lot of fantasy that tends to have bigger vocabulary than mainstream literature.

      http://freerice.com/ might be a good way to prep for SAT vocabulary, though the questions tend to be so badly structured so that test-prep methods may actually work on them.

      Comment by gasstationwithoutpumps — 2010 November 29 @ 11:01 | Reply

      • The definitions on freerice are lousy, and at the higher reaches there are a whole lot of terms from law and heraldry and so forth that are just never going to be on the SAT. Number2.com is quite good.

        Comment by Helen — 2010 November 29 @ 12:15 | Reply

        • I agree that the upper levels of freerice.com are rather poorly done (probably by people not familiar with the words themselves, but relying on a poor dictionary to pick out the meanings). Students who consistently get to those levels of freerice.com don’t need to study vocab for the SAT, though.

          Comment by gasstationwithoutpumps — 2010 November 29 @ 18:13 | Reply

  5. This essay and comments were really helpful; my daughter hates the prospect of the SAT and would rather skip it, but I’ve told her it is still used by enough colleges for admissions that it would not be in her interest to skip. She is aiming for either arts conservatory or a college with a strong performing arts program. She’s a 13 year old (grade-skipped) sophomore doing junior curriculum, so she took the PSAT this year. The kids were encouraged to prepare using off the shelf test prep books, but as far as I could tell, very little prep was done, and it mostly seemed to just offer familiarization with formats. We don’t know results yet. I am hoping it will be a confidence booster for next year’s SAT.

    Comment by Tamara — 2010 November 30 @ 06:08 | Reply

  6. My daughter has taken a PSAT prep course and will take an SAT follow-up next month. The course was not focused on accumulating the knowledge that is tested on the SAT, but on tactics and strategy for maximizing your score.

    Some of the tactics were familiar: skip hard questions and go back if you have time, guess if you can eliminate a couple of answers, etc. But there were tactics I hadn’t heard of. For example, instead of trying to solve an algebra problem, just plug in each of the multiple-choice answers until you hit on the right one.

    What was completely new to me was the strategic piece. The instructors pre-tested the kids, and then set a goal for that kid. Then based on the kid’s goal, and areas of weakness, they developed individual strategies. For example, a kid who wants a 600 in math and is lousy at geometry might be able to just skip the geometry questions and still hit 600.

    Obviously the strategies have to stay pretty simple so the kid doesn’t get overwhelmed trying to remember the strategy while taking (maybe) the most important test of his or her life.

    Here’s a book by one of the instructors, and Amazon lets you look at a few pages, so you can see some of the ideas I talked about:

    My daughter thought it helped her, but I don’t have any scores to back that up.

    I think this kind of preparation changes the ethical question. My daughter spent hours (8 hours, I think) preparing for this test, not backfilling shortcomings in her education. Because we had the money and she had the time, she (probably) has an advantage for this test over kids who have not had such good preparation. Much of what she’s learned will have no practical application once she’s taken the test, but for this one test, she (probably) has an edge.

    Comment by Dan — 2010 November 30 @ 08:40 | Reply

  7. [...] Not prepping for SAT [...]

    Pingback by NaBloPoMo is over « Gas station without pumps — 2010 December 1 @ 00:18 | Reply

  8. The SAT Reasoning Test offered by the College Board is used as a standardized college admissions test in the United States. Students typically take the SAT at the finale of their High School careers, as most major universities use the SAT as placement criteria.

    Comment by SAT Study Guide — 2010 December 12 @ 20:51 | Reply

    • This comment was flagged as spam, and it really is, but it seemed relevant enough to allow through anyway.

      Comment by gasstationwithoutpumps — 2010 December 13 @ 18:46 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 274 other followers

%d bloggers like this: