Accents, please

http://accent.gmu.edu/browse_atlas.php is a cool site that has recordings of many people reading the same passage of English text, to provide samples for determining accents.  Information is recorded about the person’s native language and how they learned English.  You can find the samples by browsing on a world map, searching by native language, or doing more sophisticated searches (by age, gender, other languages, how English was learned, and so forth).

It is a great resource for theater students who need to hear a regional accent, as well as for linguistics researchers. Some of the samples have phonetic transcriptions and some of those have been annotated, to indicate the ways in which the accent differs from “standard” English.

Not prepping for SAT

A common question that comes up for parents and teachers of high-school students is “How much test preparation is appropriate before taking the SAT or ACT tests?”  The question is not an easy one to answer, as there are several related questions that are mixed together in often confusing ways.

• Do the tests really matter?
• Is it ethical to prepare for the SAT or ACT? Or should the exams be taken “cold”, like IQ tests?
• Does test preparation improve scores on the tests?
• What sort of test preparation is most effective?
• Is time and money spent on test preparation better invested in some other pursuit?

I’ll try to give my opinion on all these, backing it up with research results where I’m aware of any.

Do the tests really matter?

Newsweek recently published an article “Going SAT-Free” that reported on several top colleges no longer requiring SAT or ACT tests: “about 830 of the country’s 2,430 accredited four-year colleges do not use the SAT or ACT to admit the majority of applicants. (Some schools require a test if you have a low GPA or class rank.)”

That leaves about 2/3 of colleges (including most public colleges) still requiring either the SAT or the ACT.  So students need to take the exam.  How important a high score on the exam is depends on the specific college, but in general, higher scores will translate to a higher probability of getting in.  (58% of colleges in one survey reported that the test scores were of considerable importance in admissions decisions.)

Is test prep ethical?

Some people have likened SAT tests to IQ tests, which present test takers with unfamiliar questions in order to determine how well they think.  Preparing for an IQ test invalidates the test, rendering the resulting scores meaningless.  Can the same be said for the SAT and ACT?

First, neither the SAT nor the ACT is attempting to measure IQ or other supposedly static properties of the test taker.  They are intended to measure the student’s preparation for college work and predict how well they will do in college.

For achievement tests, the whole point of the test is to measure how much a student has learned—to measure the total learning that the student has acquired over several years.  Preparation for achievement tests is essential, as what they are measuring is how well prepared the student is.

SAT and ACT tests are neither ability tests nor achievement tests, but a mix of the two concepts.  They measure a combination of what the student knows (vocabulary, math concepts, and so forth) and how well they can solve simple puzzles using that knowledge.  Given that the tests measure knowledge, it is not only ethical for students to prepare for the tests, but essential that they do so in some form.

The ethics question is then reduced to determining whether it is fair for wealthy students to spend more on preparing for the tests than poor students can.  Since the best preparation for the exams is a good education for the preceding 10 years, it would be very difficult to eliminate the effects of wealth.  Indeed, access to a superior education is one possible definition of wealth, independent of more conventional financial measures.

I can only conclude that preparing for the SAT and ACT tests is ethical.

Does test prep help?

The biggest debate seems to be about how coachable the SAT and ACT test scores are.  There is little doubt that students who have had 10 years of excellent education do much better than students who have had 10 years of execrable education.  The debatable question is whether short courses on content or coaching on test-taking techniques make any difference.  There is a multibillion dollar test-preparation industry, so there is a lot of incentive for marketers to sell snake oil.

The best report I’ve found analyzing the actual effectiveness of test preparation is Derek C. Briggs’s paper Preparation for
College Admission Exams, published by the National Association of College Admission Counseling.  It looks at all the published research on the topic and concludes that “Contrary to the claims made by many test preparation providers of large increases of 100 points or more on the SAT, research suggests that average gains are more in the neighborhood of 30 points.”

Of course, even just retaking a test, with no intervening coaching, can improve tests scores (on average about 15 points per section as reported by the College Board).  This report on change in averages can be misleading, since students at the ends of the distribution are likely to move towards the middle on retaking (a phenomenon known as regression to the mean), so that top students should not expect any boost from retaking the test. Briggs estimates the “coaching effect”, how much bigger the gain is from coaching than from retaking the exam without coaching:

• Coaching has a positive effect on SAT performance, but the magnitude of the effect is small.
• The effect of coaching is larger on the math section of the exam (10–20 points) than it is for the critical reading section (5–10 points).
• There is mixed evidence with respect to the effect of coaching on ACT performance. Only two studies have been conducted. The most recent evidence indicates that only private tutoring has a small effect of 0.4 points on the math section of the exam.

Briggs later says “From a psychometric standpoint, when the average effects of coaching are attributed to individual students who have been coached, these effects cannot be distinguished from measurement error. … On the other hand, if marginal college admission decisions are made on the basis of very small differences in test scores, a small coaching effect might be practically significant after all.”

This raises the question of whether 20–30 points is going to make a difference in admissions decisions.  Briggs looked at that question also.  At the low end of the scale, a 20-point difference in SAT score would not affect admissions, but  at the high end (600–750), 40% of surveyed college admissions officers thought a 20-point difference for math or a 10-point difference for critical reading would affect chances, but only 20% thought  a 20-point difference for the writing section would change the probability of admission.

It appears that the effect of short-term test preparation is small, but that admissions officers are looking at differences in scores that are well below the noise level of the tests, so retaking tests in the hopes of getting a higher score randomly could be worthwhile, and test preparation could increase the chance of increasing one’s score enough to affect admissions decisions.

So it looks like doing some test prep may improve scores.

What sort of test prep is most helpful?

Of course, just because some test prep is worthwhile does not mean that any specific course is worthwhile. Briggs distinguishes between student-driven prep (using the example questions provided by the test publishers or studying content and doing sample tests from books) and coaching with a live teacher.

Briggs reports that “No forms of test preparation had statistically significant positive effects on SAT-V scores” and that books, courses, and tutors all had small positive effects on SAT-M.  (Interestingly, use of a computer prep program had a small negative effect.)

Is test prep cost-effective?

Briggs says “Beyond that which occurs naturally during students’ years of schooling, the only free test preparation is no
test preparation at all. This is because all test preparation involves two costs: monetary cost and opportunity
cost.”

The financial costs are easy to analyze. Given the small gains from commercial coaching courses and the roughly similar gains from using a test prep book, there doesn’t seem to be much financial sense to paying for the much more expensive commercial courses.  The books are cheap (and readily available from libraries) so there seems no financial barriers to using them.

The opportunity cost is the time spent on test prep that might more usefully have been spent studying for classes, sleeping, or doing extracurricular activities (like sports, theater, or community service).  Here the analysis is more difficult, but I think favors spending fairly little time on test prep.  Time spent pursuing a passion or serving the community is more likely to improve one’s chances of admission to college than small gains in test scores will.  Of course, time wasted hanging out at the mall or playing video games is unlikely to have any positive effect.

What will we do?

Since my son did very well on the SAT math and critical reading sections when he took them in 6th grade (over 700), I see no reason for him to waste time on test prep for those sections.  He did less well on the writing portion, getting the lowest possible score on the essay, but at the time he had never had instruction in timed essay writing, and had never even heard of the 5-paragraph essay so beloved of SAT graders.  I expect that he will need little prep for the essay writing also, as he will undoubtedly get more practice on the 5-paragraph essay than any sane person could stand in his high school classes.

Powerpoint?

This post is not a list of tips for producing slides to support a presentation (maybe I’ll do a post on that some other time).  Instead, it is a reflection on a pair of essays by Edward Tufte and Jean-Luc Doumont.

I read and enjoyed Edward Tufte’s essay The Cognitive Style of Powerpoint: Pitching Out Corrupts Within. It is a hard-hitting critique of the “pitch culture” that turns all presentations into a series of bullet lists. I wanted to require it for my fall senior design class (which is largely about “soft” skills, such as team formation, management, verbal presentation, and writing), but I had a hard time justifying making the students pay $7 for a 32-page essay. Instead I required the free 6-page excerpt from the essay on Tufte’s web site that catches some of the main ideas, though it is not as well-crafted as the complete essay.  One of the main points that Tufte makes is that the choppy, bullet-list format encouraged by slides is destructive to longer narratives and connected thoughts, and that sentences and paragraphs are not evil.  Not in the excerpt is Tufte’s analysis of the spoof of the Gettysburg Address by Peter Norvig.

Edward Tufte is famous for his self-published books, particularly the first one, Visual Display of Quantitative Information, which is perhaps the best book around on presenting data graphically.  It should be required reading for every scientist, math teacher, science teacher, and journalist.  Edward Tufte also gives one-day workshops based on his books.  I’ve never been to one, but some of the grad students in my department have (Tufte gives a huge student discount: they get the seminar plus four of his books for just the price of the books).  The students report that he gives awesome seminars also, well worth the time and the money.

So Tufte’s credentials as a presenter of data are very, very solid, and people paid a lot of attention to his polemic against PowerPoint.  Perhaps too much so, as his criticism seems mainly directed at the use of slides to replace tech reports, which they clearly cannot replace.

Jean-Luc Doumont has written a good rebuttal to Tufte: “The Cognitive Style of PowerPoint: Slides Are Not All Evil” (Technical Communication, 52(1), 64–70, Feb 2005), which Amazon sells for $6.  Doumont also gives a good seminar on presentation (I’ve heard the one-hour version), and so I respect his opinions also. I got permission from Doumont to distribute his rebuttal to Tufte to my class, but I put it up on a secure server behind password protection. It seems that other teachers either asked for more permission, or have less respect for authors’ copyrights, as the PDF file can be found on-line with a Google search (and not at Doumont’s own website, where it would be if he had truly meant for it to be distributed freely).

Doumont’s main point is that Tufte missed the point of slides:

Three commonsense considerations related to purpose thus invalidate much of Tufte’s case against the use of slides:

• Oral presentations typically have a different purpose than written documents (different even than companion documents).
• Slides in oral presentations are viewed while the presenter is speaking, not read in silence like written documents.
• Tables and graphs, too, may serve a range of purposes, from analysis by oneself to communication to an audience.

The slides should support the speaker, not replace him or her. Both authors agree that (in Doumont’s words) “presentation slides do not double up effectively as [a] presentation handout,” because what is effective as a presentation aid is too terse to be of much use as a standalone document, and a useful standalone document is too wordy to be of much use as a presentation aid. I teach students that the purpose of an oral presentation is as an advertisement for the written document: to make the listeners aware of the ideas and interested enough to want to know more.  There should be just a handful of take-home messages from an oral presentation—trying to pack all the information of a detailed technical paper into a talk results in the listener coming away with nothing.

Doumont also criticizes Tufte for conflating the tool PowerPoint and the slides produced with it—many of the bad things Tufte points out are the fault of the presenters, not of the tool they used. His criticism here is perhaps a little too protective of Microsoft, as some of the common flaws that Tufte points out are indeed encouraged by the tool (PowerPoint provides many very bad templates).  Still, Doumont’s point is well-taken: it is possible to do good presentations with PowerPoint, even if it is not as easy as making bad ones.

Note: to create his own presentations, Doumont does not use PowerPoint, but uses with an idiosyncratic macro package, not a style many people will find easy to copy.  I also use , with the prosper package in , using Adobe Reader to present the resulting PDF files.  This is the only way I’ve found to include decent math formulas in presentations, something I often need to do.  (There are other slide styles, but I’ve been reasonably happy with prosper, and I was not at all happy with the original SliTeX program.)

UC weird pricing for health insurance

The University of California has just finished its “open enrollment” period, when faculty and staff can make changes in their benefit packages for next year.  Once again the University has transferred more of the costs to the employees, with the biggest differences in the health insurance. I used to get HMO insurance for myself and my family for free, but now I need to pay $455.87 a month ($5470 a year) for essentially the same policy.  I think that the University is actually paying more than they used to, but health insurance costs have been soaring, with no apparent change in health care delivery.  One wonders where all that money is ending up.

At some campuses (but not the one I’m at) there are cheaper HMO options: Kaiser and a new Health Net Blue and Gold plan.  Technically, the Blue and Gold plan is available here, but there are very few providers in town, and most aren’t taking new patients.  Since I don’t have a car, a doctor 20 miles away is not really an option.

In an attempt to appear progressive, the University has started  “banding” health insurance payments by employees.  They do this in a really brain-damaged way (the same way they’ve been doing since 2004).  They don’t base it on what people are paid, but on what their “full-time salary rate” is.  I think that means the 9-month rate for faculty, so those who get summer salary don’t pay more, but I’m not sure. Certainly those who are only being paid part time are paying at higher rates. The employee payments don’t go up gradually, but take big jumps at certain salaries.

salary range	Employee monthly payment (HealthNet HMO, self+adult+children)
s≤$47,000	$258.63
$47,000<s≤$93,000	$360.83
$93,000<s≤$140,000	$455.87
$140,000<s	$554.30

Note that this banding results in non-monotonic compensation as a function of salary. If someone gets a raise from $93,000 to $93,001, their compensation doesn’t go up by $1, it goes down by $1139.   Similarly from $47,000 to $47,001 is a loss of $1225.

There is no good reason for this “banding”.  They could have equally well based the employee payments as a percentage of salary.  Why didn’t they? Stupidity? (always a tempting explanation for university administration) Desire to have more complicated computer programs? Listening to consultants? I have a different explanation, based on the following graph:

What fraction of their annual salary does a UC employee have to pay for health insurance premiums (HealthNet HMO, self+adult+children)?

Note that this is really not a progressive scheme.  The low-paid employees spend far more of their income on the health insurance payments than the administrators do.  A flat 5% would have been much simpler to implement and much fairer. Of course, the people at the bottom salary levels can’t pay the huge amounts this chart shows—they have to opt for the much poorer “Core” insurance that results in $3000 deductibles and only 80% coverage above that.

 

Lagrangian Multipliers and HMMs

Where in an undergraduate STEM major is the technique of Lagrangian multipliers for solving constrained optimization problems taught?

Doing a search for it on my campus found a few classes, all of which are upper-division or graduate classes: MATH 140 (Industrial Math), Econ 100 A(Intermediate Microeconomics), Econ 205C (Advanced Macroeconomic Theory III), ISM 206 (Optimization Theory and Applications), Physics 105(Classical Mechanics), CMPS 242 (Machine Learning), CMPS 290C (Advanced Topics in Machine Learning), BME 205 (Bioinformatics: Models and Algorithms). Students in my class had a vague recollection of having heard of Lagrangian multipliers, but had never actually used them (except for a couple of exchange students from Europe).

My personal belief is that such a fundamental, useful technique should be included in the lower-division multi-variable differential calculus class. Now, this isn’t a curmudgeonly complaint about how students nowadays don’t learn things the way we did, back in the good old days of slide rules and tables of logarithms (and, yes, I am just barely old enough to have learned with those tools). I didn’t learn about Lagrangian multipliers until I was a professor (having completed a B.S. and M.S. in math and a Ph.D. in computer science).  I was introduced to the method by an engineering grad student (from India, if I remember correctly), who had had a much more practical math education than mine.  In fact, most of my practical math education came either from my father or was self-taught.  Very little of my math education in school or college turned out to have much lasting value to me, other than allowing me the confidence to learn on my own the math I actually needed.

For those of you completely unfamiliar with Lagrangian multipliers, I recommend the Wikipedia page, Lagrangian multipliers. For bioinformatics students, I will try to present one example, using the technique to derive the Baum-Welch algorithm, which is the expectation-maximization algorithm used for training hidden Markov models (HMMs). These are the notes for one lecture in my graduate class in bioinformatics, coming after we have developed the forward and backward algorithms for HMMs. For those who want to learn this stuff on their own, I recommend Chapter 3 of Biological Sequence Analysis by Durbin, Eddy, Krogh, and Mitchison, though I don’t remember them using Lagrangian multipliers.

[I’m presenting these notes in my blog for two reasons: 1) I wanted to plead for more teachers to teach Lagrangian multipliers, and 2) my presentation of it in class sucked.  In my defense, I had only had four hours of sleep the night before, and live-action math requires more alertness than I had with so much sleep debt.]

Our goal is to choose the parameters of the hidden Markov model to maximize the probability of the HMM generating a particular observed set of data.  The EM (Expectation Maximization) algorithm is an iterative approximation algorithm.  It starts with a setting of the parameters, then chooses a new setting of parameters that increases the probability, and repeats.  What we are working on here is doing a single step: given a setting of the parameters, how do we find a better setting?

The parameters are , the probability of emitting character in state , and , the probability of making a transition to state from state (I’m using the notation of the book here, though it is awkward to mix subscript and function notation, and would have been cleaner to use ). It is common to refer to the parameters as a vector , so that we can concisely refer to functions of $\latex \Theta$. We have the constraints that and $\latex \sum_T a_{ST} = 1$, that is that emission probabilities for a state sum to one, and so do the transition probabilities to a new state.  (There are also constraints that all probabilities are non-negative, but those constraints don’t need to be explicitly represented in solving the problem, as the solution we’ll come up with automatically satisfies that.)

The probability of a sequence is the sum over all paths of length through the HMM of the probability of taking that path and emitting , that is . The products are  a bit messy to take derivatives of, because some of the parameters may be repeated,  but we can simplify them by grouping together repeated factors:

,

where is the number of times and along a particular path and is the number of times and .  Both of these new count variables are specific to a particular path, so should really have the vector as subscripts, but we’ve got too many subscripts already.

If we try to maximize this function without paying attention to the constraints, we find that all the parameters go to infinity—making the parameters bigger makes the product bigger. To keep the constraints relevant, we use Lagrangian multipliers, adding one new variable for each constraint.  I’ll use for the constraints on the emission table of state and for the constraints on the transitions out of state .  The objective function to take derivatives of is thus

Let’s look at the partial derivatives of our objective function with respect to the emission parameters:

If the partials are set to zero, we get

The sum over all paths formula is the expected count of the number of times that state produces character $d$. Note that this happens only  for positions where , so we can rewrite it as the sum over all such positions of the probability of being in state at that position, which is precisely what the forward-backward algorithm computed for us:

We can now impose the constraint , by adding all these equations:

Finally we get the new value of the emission parameters:

We can do a very similar series of operations to get update equations for the transition parameters also:

Note that the optimization we did uses the parameters in the forward-backward algorithm, so we haven’t really found “the optimum”, just a better setting of the parameters, so we need to iterate the whole process, recomputing the forward-backward matrices after resetting all the parameters.