Gas station without pumps

2014 May 3

Spread on SAT2 raw scores

Filed under: Uncategorized — gasstationwithoutpumps @ 12:15
Tags: , , , ,
On one of the mailing lists I read, someone asked
Can anyone explain to me how a raw score of 75 and a raw score of 59 are both 800’s on the scale for the physics test?  That seems a huge spread. I see similar stuff on other tests, but nothing spread quite this far.
One can find the distribution of scale scores as percentiles on the College Board web site at http://media.collegeboard.com/digitalServices/pdf/research/SAT-Subject-Tests-Percentile-Ranks-2013.pdf, but finding information about raw scores is harder.  The College Board says
The raw score is converted to the College Board 200- to 800-point scaled score by a statistical process called equating. Equating adjusts for slight differences in difficulty between test editions and ensures that:

  • A student’s score does not depend on the specific test edition she took.
  • A student’s score does not depend on how well others did on the same edition of the test.

[https://professionals.collegeboard.com/testing/sat-subject/scores/scoring]

I’ve found raw-score-to-scaled-score conversions for one version of the SAT test at http://media.collegeboard.com/digitalServices/pdf/research/SAT-RAW-Score-to-Scaled-Score-Ranges-2013.pdf, but I’ve not found them for SAT 2 tests, and I don’t know whether the person had access to more data than I’ve been able to find on the College Board site, or just had a couple of data points for students who both got a standard score of 800.
The scale score on the SAT subject test, like other scale scores, is intended to have the same meaning from year to year, despite differences in the underlying test questions.  Initially, tests are written so that questions span a range of difficulty, with some easy questions and some hard ones.Depending on the purpose of the test, the questions may cluster around a particular level of difficulty—if the test is intended as a pass/no-pass test, the questions hover around the pass threshold. Think of a driving exam, where the questions are intended to separate those who can drive safely from those who can’t.  There is no point to asking esoteric questions that even good drivers can’t answer, nor trivial ones that even bad drivers do well at.

When the point is to spread students out without a single boundary (as for college admissions), there needs to be a wider diversity of difficulty of questions.  You need some that are so difficult that few get them, and some that are so easy that few miss them, and everything in between. There need to be several difficult questions, to compensate for students randomly guessing correctly on one difficult question.
Because the scale scores are supposed to mean the same from year to year, and the scales are arbitrarily capped at 200 and 800, drift in student education or who takes the exams can make students pile up at one end of the scale or the other, even if initially the end scores were very rare.  That has happened in math (level one has few students with 800, level 2 has 9% of students getting 800), physics (8% of students get an 800), and most of the language exams (when mostly native speakers take the exam—for Chinese, lots of students get an 800).  Student selection seems to play more of a role than education (people who expect to do poorly don’t pay to take the test), so the bottom end of the scale is rarely used, while students pile up at the top end.
A more sensible system would not cap the scale scores (Lexiles, for example, are not capped) so that drifts in student population do not pile up people at one point on the scale.  Many of the underlying SAT subject tests have the ability to measure more at the top end, but the political or marketing (not scientific) decision to cap scores at 800 limit the utility of the tests for making distinctions.
Note: the ACT has similar artificial score limitations and suffers the same sort of unnecessary ceiling, though it is not clear whether the underlying questionshave the ability to make distinctions at the top end—when you get down to differences of one or two questions right, you are looking at noise, not signal.

 

2012 August 29

Tests for 11th grade

Filed under: home school — gasstationwithoutpumps @ 11:48
Tags: , , , , , , , , , , ,

In Home schooling restarts, I listed the courses I thought my son would take this year.  In this post I’ll list the exams I think he’ll need.

test date register by comments
pSAT Oct 17 ? He needs to take pSAT this year for National Merit Scholarships, as he has a good chance of getting one.
SAT Oct 6, Nov 3, Dec 1, Jan 26, May 4, Jun 1 Sep 7, Oct 4, Nov 1, Dec 28, Apr 5, May2 I think that he should take an SAT test early this year, to find out how much more work he needs on SAT essays.  If necessary, he would take the SAT again in the Spring. SAT exams can’t be on the same day as SAT Subject tests, though multiple Subject tests can be the same day.
ACT Oct 27, Dec 8, Feb 9, Apr 13, Jun 8 Sep 21, Nov 2, Jan 11, Mar 8, May 3 ACT is not popular on the coasts, but I’ve heard that some students do better on the ACT than on the SAT (and vice versa).  If his SAT scores aren’t what we hope for, it may be worth doing an ACT test as well.
SAT Spanish with Listening Nov 3 Oct 4 It would probably be better for him to take this text next year, after one more Spanish class.  It is only offered in November.  He could take the SAT Spanish without listening on any other SAT day.
SAT World History Dec 1, Jun 1 Nov 1, May 2 This test may be a way to validate that his idiosyncratic “world history through history of science” covered an adequate amount of world history.  The SAT subject test should be less demanding than an AP test.
SAT Physics May 4, Jun 1 Apr 5, May 2 This test should be pretty easy, compared to the AP Physics C tests, but he’ll need to finish E&M first.  It will help with schools that require two STEM SAT Subject tests.
AP Physics C: Electricity & Magnetism May 14 March? We’ll have to find a school willing to administer this exam, as no high school in the county does Physics C.
AP Computer Science A May 8 March? He may be able to take this at Pacific Collegiate, as they have an AP computer science course.
AP Spanish Language May 8 March? This is scheduled for the same time slot as AP CS, so one of them has to be moved to the makeup slot.
AMC 12 Feb 5, Feb 20 Dec 13 He could take either or both AMC 12 tests this year.  (Taking both increases his chance of being invited to take the AIME test.)
AIME Mar 13?? If he does well on the AMC 12, he could be invited to take the AIME in mid March or early April—it doesn’t seem to have been scheduled yet.

Combining these tests with previous years’ he’ll have 5 AP test scores and 3–4 SAT Subject scores. We’ll have to check the list of colleges he might apply to, to see if any need 5 SAT Subject tests.

Other dates

He has several other scheduled activities this year that need to be planned around.

Activity date comments
Performance Jan 19, 20 Imaginary Invalid at Broadway Playhouse (WEST Ensemble Players)
Performance May 4, 5 The Outsiders at Broadway Playhouse, Saturday morning rehearsal may conflict with SAT. (WEST Ensemble Players)
Performances ? Dinosaur Prom Improv, probably every 6–8 weeks.
Oregon Shakespeare Festival May? Dates for trip not yet on AFE calendar.
County Science Fair Mar 9 If he decides to do science fair this year, he’ll have to register by Feb 15.
California State Science Fair Apr 29–30?? Schedule not posted yet. Only relevant if he decides to do science fair and gets chosen to represent the county at state.
International Science and Engineering Fair May 12–17 Only relevant if he decides to do science fair and gets chosen to represent the county at ISEF. Conflict with AP exams can be resolved by taking exam at ISEF.

2012 June 3

Planning ahead for tests

Filed under: home school — gasstationwithoutpumps @ 22:22
Tags: , , , , ,

I was entering some of next year’s events on my calendar for planning purposes (like reserving the classroom I’ll be teaching in for the fall—we use a seminar room that is not under the control of the Registrar, so faculty are responsible for using Google Calendar to reserve the room).

I was particularly interested in figuring out when my son would be taking various scheduled exams: pSAT/NMSQT, SAT, ACT, SAT Subject tests, AP tests, … I’ll probably have to make sure that the tests get ordered sufficiently far in advance, and those deadlines tend to sneak up on me.

I had no trouble finding when the ACT tests are for the next two years (ACT Registration : Test Dates in the U.S., U.S. Territories, and Canada), but the SAT schedule was a little harder to find.  As usual, the College Board search box was useless, but Google found the information on Test Dates & Deadlines, though those are still “anticipated dates”. It seems that the College Board is not willing to commit too strongly to a schedule. I think I want my son to take both an ACT test and an SAT test during the year. I’ve heard that the writing components are quite different, and I don’t know which one he’ll do better on. It might also be best to do these tests early in the year, before his schedule gets too packed. The end of April and beginning of May was crazy this year, so (other than the AP exams) we’d like to stay away from extras around then.

The 2013 AP Exam Dates seem more definite, which is too bad, because they’ve scheduled two of the tests my son was planning to take (Computer Science A and Spanish Language) for the same time slot. That means he’ll have to request a late testing slot to resolve the conflict (taking the CS exam on May 23).

The pSAT is also scheduled for the next 2 years, and this taking of the test that will be the one that counts for National Merit Scholarships. Unless he is very ill on the test dates, I expect he’ll do fairly well—he only missed one question on the pSAT this year.

2010 November 29

Not prepping for SAT

Filed under: Uncategorized — gasstationwithoutpumps @ 00:02
Tags: , , ,

A common question that comes up for parents and teachers of high-school students is “How much test preparation is appropriate before taking the SAT or ACT tests?”  The question is not an easy one to answer, as there are several related questions that are mixed together in often confusing ways.

  • Do the tests really matter?
  • Is it ethical to prepare for the SAT or ACT? Or should the exams be taken “cold”, like IQ tests?
  • Does test preparation improve scores on the tests?
  • What sort of test preparation is most effective?
  • Is time and money spent on test preparation better invested in some other pursuit?

I’ll try to give my opinion on all these, backing it up with research results where I’m aware of any.

Do the tests really matter?

Newsweek recently published an article “Going SAT-Free” that reported on several top colleges no longer requiring SAT or ACT tests: “about 830 of the country’s 2,430 accredited four-year colleges do not use the SAT or ACT to admit the majority of applicants. (Some schools require a test if you have a low GPA or class rank.)”

That leaves about 2/3 of colleges (including most public colleges) still requiring either the SAT or the ACT.  So students need to take the exam.  How important a high score on the exam is depends on the specific college, but in general, higher scores will translate to a higher probability of getting in.  (58% of colleges in one survey reported that the test scores were of considerable importance in admissions decisions.)

Is test prep ethical?

Some people have likened SAT tests to IQ tests, which present test takers with unfamiliar questions in order to determine how well they think.  Preparing for an IQ test invalidates the test, rendering the resulting scores meaningless.  Can the same be said for the SAT and ACT?

First, neither the SAT nor the ACT is attempting to measure IQ or other supposedly static properties of the test taker.  They are intended to measure the student’s preparation for college work and predict how well they will do in college.

For achievement tests, the whole point of the test is to measure how much a student has learned—to measure the total learning that the student has acquired over several years.  Preparation for achievement tests is essential, as what they are measuring is how well prepared the student is.

SAT and ACT tests are neither ability tests nor achievement tests, but a mix of the two concepts.  They measure a combination of what the student knows (vocabulary, math concepts, and so forth) and how well they can solve simple puzzles using that knowledge.  Given that the tests measure knowledge, it is not only ethical for students to prepare for the tests, but essential that they do so in some form.

The ethics question is then reduced to determining whether it is fair for wealthy students to spend more on preparing for the tests than poor students can.  Since the best preparation for the exams is a good education for the preceding 10 years, it would be very difficult to eliminate the effects of wealth.  Indeed, access to a superior education is one possible definition of wealth, independent of more conventional financial measures.

I can only conclude that preparing for the SAT and ACT tests is ethical.

Does test prep help?

The biggest debate seems to be about how coachable the SAT and ACT test scores are.  There is little doubt that students who have had 10 years of excellent education do much better than students who have had 10 years of execrable education.  The debatable question is whether short courses on content or coaching on test-taking techniques make any difference.  There is a multibillion dollar test-preparation industry, so there is a lot of incentive for marketers to sell snake oil.

The best report I’ve found analyzing the actual effectiveness of test preparation is Derek C. Briggs’s paper Preparation for
College Admission Exams, published by the National Association of College Admission Counseling.  It looks at all the published research on the topic and concludes that “Contrary to the claims made by many test preparation providers of large increases of 100 points or more on the SAT, research suggests that average gains are more in the neighborhood of 30 points.”

Of course, even just retaking a test, with no intervening coaching, can improve tests scores (on average about 15 points per section as reported by the College Board).  This report on change in averages can be misleading, since students at the ends of the distribution are likely to move towards the middle on retaking (a phenomenon known as regression to the mean), so that top students should not expect any boost from retaking the test. Briggs estimates the “coaching effect”, how much bigger the gain is from coaching than from retaking the exam without coaching:

  • Coaching has a positive effect on SAT performance, but the magnitude of the effect is small.
  • The effect of coaching is larger on the math section of the exam (10–20 points) than it is for the critical reading section (5–10 points).
  • There is mixed evidence with respect to the effect of coaching on ACT performance. Only two studies have been conducted. The most recent evidence indicates that only private tutoring has a small effect of 0.4 points on the math section of the exam.

Briggs later says “From a psychometric standpoint, when the average effects of coaching are attributed to individual students who have been coached, these effects cannot be distinguished from measurement error. … On the other hand, if marginal college admission decisions are made on the basis of very small differences in test scores, a small coaching effect might be practically significant after all.”

This raises the question of whether 20–30 points is going to make a difference in admissions decisions.  Briggs looked at that question also.  At the low end of the scale, a 20-point difference in SAT score would not affect admissions, but  at the high end (600–750), 40% of surveyed college admissions officers thought a 20-point difference for math or a 10-point difference for critical reading would affect chances, but only 20% thought  a 20-point difference for the writing section would change the probability of admission.

It appears that the effect of short-term test preparation is small, but that admissions officers are looking at differences in scores that are well below the noise level of the tests, so retaking tests in the hopes of getting a higher score randomly could be worthwhile, and test preparation could increase the chance of increasing one’s score enough to affect admissions decisions.

So it looks like doing some test prep may improve scores.

What sort of test prep is most helpful?

Of course, just because some test prep is worthwhile does not mean that any specific course is worthwhile. Briggs distinguishes between student-driven prep (using the example questions provided by the test publishers or studying content and doing sample tests from books) and coaching with a live teacher.

Briggs reports that “No forms of test preparation had statistically significant positive effects on SAT-V scores” and that books, courses, and tutors all had small positive effects on SAT-M.  (Interestingly, use of a computer prep program had a small negative effect.)

Is test prep cost-effective?

Briggs says “Beyond that which occurs naturally during students’ years of schooling, the only free test preparation is no
test preparation at all. This is because all test preparation involves two costs: monetary cost and opportunity
cost.”

The financial costs are easy to analyze. Given the small gains from commercial coaching courses and the roughly similar gains from using a test prep book, there doesn’t seem to be much financial sense to paying for the much more expensive commercial courses.  The books are cheap (and readily available from libraries) so there seems no financial barriers to using them.

The opportunity cost is the time spent on test prep that might more usefully have been spent studying for classes, sleeping, or doing extracurricular activities (like sports, theater, or community service).  Here the analysis is more difficult, but I think favors spending fairly little time on test prep.  Time spent pursuing a passion or serving the community is more likely to improve one’s chances of admission to college than small gains in test scores will.  Of course, time wasted hanging out at the mall or playing video games is unlikely to have any positive effect.

What will we do?

Since my son did very well on the SAT math and critical reading sections when he took them in 6th grade (over 700), I see no reason for him to waste time on test prep for those sections.  He did less well on the writing portion, getting the lowest possible score on the essay, but at the time he had never had instruction in timed essay writing, and had never even heard of the 5-paragraph essay so beloved of SAT graders.  I expect that he will need little prep for the essay writing also, as he will undoubtedly get more practice on the 5-paragraph essay than any sane person could stand in his high school classes.

%d bloggers like this: