Gas station without pumps

2011 May 22

Jay Mathews gets it wrong again

Filed under: Uncategorized — gasstationwithoutpumps @ 11:37
Tags: , ,

Once again, Jay Mathews of the Washington Post looks at the wrong statistic to get ludicrous results: High school list shows hidden differences .

I’ll give him the benefit of the doubt on intentions: that his interest is really in raising the education levels of American high-school students, particularly in poorer neighborhoods, rather than in simply selling AP tests.

His measurement, though, is clearly wrong.  What matters is whether students are learning, and if you want to measure at the AP level, what matters is how many students are passing the tests.  Neither the number taking them nor the percentage passing are really relevant—both are easily manipulated by administrators and teachers controlling who takes the test.  He praises himself for inventing a measure that depends not on student outcomes, but just on how many students are made to take a test (tests taken divided by graduating seniors). By his measures, a school that forced all students to take AP tests while teaching them absolutely nothing is a great school, while one that taught students well but did not encourage test taking was an abject failure.  A school with an enormous dropout rate would do even better (fewer graduates to divide by).

There is an equally simple measure he could use that would be enormously more meaningful: the number of tests passed divided by the number of entering freshmen.  (Even better would be to divide by the number of freshman who entered 4 years earlier,  so that fluctuations in population would not add noise.)  This measure would still have the desirable effect of pushing schools to challenge more students (you can’t get high numbers of tests passed if no one is preparing for them), but avoids much of the bad effect of pushing unprepared students into courses way over their heads (the number passing doesn’t go up if you add a lot of students at the bottom).

Mr. Mathews has a magical belief that simply exposing students to “rigorous courses” transforms them into well-educated students, whether or not they are capable of the work or learn anything in the courses.  He has led an AP-for-everyone movement that has resulted in substantial watering down of AP courses, to the point where students who could have benefited from a college-level course in high school can no longer trust the AP label on a course to mean that (see High school course title inflation).

Since Mr. Mathews has been informed repeatedly over the years by many people that what he is measuring is really not the right thing, I can only assume that he is either completely resistant to learning from others or of admitting mistakes. (Either that, or his intent really is to sell AP exams, not to improve education.)  What surprises me is that he continues to get huge corporate support for his endeavor.  The corporations are not usually stupid or resistant to learning, so I can only assume that they do have some less savory reason for continuing to push Mr. Matthews’s nonsense.

I also noticed that even within his own distorted system, he is not above cooking the books.  For example, he does not include Pacific Collegiate School, a lottery-entry charter with 337 tests taken and a graduating class of  62.  In previous years he excluded them for spurious reasons (he thought the students were too smart, though he did not exclude test-entry schools for the gifted)—this year he does not even discuss their absence from his list (at least, not where I can find it).


  1. I notice that Illinois Math and Science Academy and University High School are not included either. I guess they are “too selective”, as opposed to the other listed schools for gifted that must be only “not too selective”. Of course, when my kids attended Uni, they didn’t really offer AP classes per sé. Many students still took the exams, anyway – and passed.

    Comment by Laura Walsh — 2011 May 22 @ 12:30 | Reply

  2. In an email message, it was pointed out to me that

    “The paper is part of The Washington Post Company, a diversified education
    and media company that also owns educational services provider Kaplan, Inc. …”

    Kaplan is a major prep test provider, and my understanding is that it is
    also the most profitable of the WP’s companies … his mission is to sell
    Kaplan test prep materials. Duncan Black, lucid economist and blogger, has
    been calling the WP “Kaplan Test Prep Daily” for a few years now based on
    their compromised journalistic position in several areas.

    Comment by gasstationwithoutpumps — 2011 May 22 @ 13:22 | Reply

  3. Through e-mail, another reader suggested a different measure:

    instead of giving the high school equal credit for each score of 3 (typically considered passing) or higher, I think the formula should use (AP_score – 1), thus giving 1 point for a 2 and 4 points for a 5.

    I would be fine with such a measure also, though I think it makes too much of the distinction between a 5 and a 4: the scoring system of AP is not really a linear scale.

    Comment by gasstationwithoutpumps — 2011 May 22 @ 14:52 | Reply

  4. Ack, someone else pointed out the Kaplan thing. I do know that somewhere either in the WaPo special section today (I get the print edition on Sundays) or in his chat, he purposely left out charter schools and magnet schools because their selectiveness would “skew the data.”

    I just thought he spent too much time talking about himself.

    Comment by Tom Panarese — 2011 May 22 @ 16:09 | Reply

    • But he included several charter and magnet schools, including several schools for the gifted, so his selection criteria for schools seemed to be purely whether or not he liked them. (Or maybe whether they advertised Kaplan courses???)

      Comment by gasstationwithoutpumps — 2011 May 22 @ 16:52 | Reply

      • You’re right … on both counts. Maybe they do advertise Kaplan courses?

        Don’t be shocked if he writes a book soon.

        Comment by Tom Panarese — 2011 May 24 @ 17:05 | Reply

  5. The U.S. News & World Report rankings also exclude the most selective schools, so the secret to getting to the top of their rankings is to be selective, but not too selective. And encourage your students to take only a few AP tests and do very well on them, since their rankings give a great deal of weight to the percentage of tests passed, not the number of tests taken.

    Comment by Dan — 2011 May 23 @ 06:15 | Reply

  6. Mr. Mathews has a magical belief that simply exposing students to “rigorous courses” transforms them into well-educated students

    Mathews based the “Challenge” rankings on research showing that students who had taken AP courses and failed the AP exams did better in college than students who hadn’t taken AP courses at all. I think there have been at least a couple of studies showing that. (I’m not going to take the time to dig up the references.)

    I’ve never seen him say that exposure to rigorous work produces well-educated students. His argument is entirely that AP students who fail the AP test fare better in college than kids who haven’t taken AP courses at all.

    He added the “Equity and Excellence” score, which is the score I care about, in the last Challenge Index, I think. Explaining why he created the Equity and Excellence score, he cited D.C. schools that had large numbers of students taking AP exams with practically no one passing.

    Last but not least, the rankings have nothing to do with Kaplan test prep, a company owned by the Post corporation. I think Mathews became interested in the question of low-income students not being allowed to take serious courses when he wrote a book about Jaime Escalante in South LA.

    Comment by Catherine Johnson — 2011 May 24 @ 08:05 | Reply

  7. […] Jay Mathews gets it wrong again […]

    Pingback by Blogoversary « Gas station without pumps — 2011 June 5 @ 10:52 | Reply

  8. […] “Challenge Index” that has given rise to the sharpest criticism over time (see here, here, and here, for example) because of its methodology, which is reductionist in the extreme. It […]

    Pingback by High-School Rankings Are Meaningless — And Harmful | Sense and Nonsense — 2013 August 25 @ 15:22 | Reply

  9. […] “Challenge Index” that has given rise to the sharpest criticism over time (see here, here, and here, for example) because of its methodology, which is reductionist in the extreme. It […]

    Pingback by High-School Rankings Are Meaningless — And Harmful | MASSunderstanding — 2014 June 25 @ 08:10 | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: