Gas station without pumps

2015 June 19

2015 AP Exam Score Distributions

Filed under: Uncategorized — gasstationwithoutpumps @ 21:36
Tags: , , ,

Once again this year, I’m posting a pointer to 2015 AP Exam Score Distributions:

Total Registration has compiled the following scores from Tweets that the College Board’s head of AP, Trevor Packer, has been making during June. These are preliminary breakdowns that may change slightly as late exams are scored.

I don’t know why I provide this free advertising for Total Registration, as I have no connection with the company, and do not endorse their services.  If the College Board would collect Trevor’s comments themselves, I’d point that page.  The main interest in AP result distributions comes in May, when students are taking the tests, and July when the students get the results.

The official score distributions (still from 2014 as of this posting—new results don’t go up until the Fall) from the College board are at https://apscore.collegeboard.org/scores/about-ap-scores/score-distributions, at least until the College Board scrambles their web site again, which they do every couple of years, breaking all external links.  They post a separate PDF file for each exam, which makes comparison between exams more difficult (deliberately, I believe, since inter-exam comparison is not really a meaningful thing to do).  It is also difficult to get good historical data on how the exam scores have changed over time—College Board probably has it on their website somewhere, but finding stuff in their morass is not easy.

My most popular post this year was once again How many AP courses are too many?, with about 19 views per day.  (It has also come in second over the lifetime of the blog, behind 2011 AP Exam Score Distribution.) The question of how many AP courses seems to come up both in the fall, when students are choosing their schedules, and in the spring, when students are overwhelmed by how many AP courses they took.

There aren’t many exams graded yet (only 11 on the Total Registration site), so I don’t have much to say about the results.  I probably won’t be looking at the exam scores much this year, since my son is no longer eligible to take AP exams, having graduated from high school. I might look at some of the statistics for the AP computer science exam, as I have some interest in seeing whether there are any changes in the number of test takers.  The interesting results (about gender and geography) won’t come out until the fall reports.

Advertisements

2014 September 10

edX finally finds the right market

Filed under: freshman design seminar,home school — gasstationwithoutpumps @ 10:02
Tags: , , ,

I have long been of the opinion that MOOCs are pretty useless for college students but are good for home-schooled students and high school students who don’t have access to higher-level courses in their local schools.  It seems that edX has finally realized that this is an important market with their High School Initiative:

Colleges and universities find that many students could benefit from taking a few extra courses to help close the readiness gap between high school and college. To address this need, edX has launched a high school initiative–an initial collection of 26 new online courses, including Advanced Placement* (AP*) courses and high school level courses in a wide variety of subject areas.

Completion rates will still be low, as a lot of people will sign up on a whim and then not follow through, or will sign up for more than they can handle and be forced to drop some.

AP courses will probably be the most attractive courses, as students can validate their learning with the AP exam, which is widely accepted by colleges as proof of higher-level course work (unlike the “Verified Certificate of Achievement” that edX sells).

The hardest courses to do well on line will be the lab science courses.  Simulated labs are no substitute for real-world labs, as no simulation captures all the phenomena of the real world and few come close to developing lab skills.

There are on-line science courses with real lab components. For example, my son took AP chem through ChemAdvantage.com, which had some rather cleverly designed labs that could be done at home with minimal equipment. Despite the cleverness of the lab design, the lab skills practiced were not exactly the same as would have been practiced in a more traditional lab setting.  And the lab kit was not cheap, costing as much as a community college chem lab course would have (if my son had been able to get into the over-subscribed chem lab course).

I don’t know whether edX has gotten their AP science courses audited by College Board (if not, they’ll probably be forced to remove the AP designation), but the AP audit requires lab time for the AP science courses, and I don’t know which of many mechanism edX is using to provide the required lab content.  Other online AP courses either devise home labs (requiring the purchase of a lab kit) or do weekend or week-long lab intensives in various parts of the country.  These lab intensives can be quite good (if done in college labs with real equipment) or ludicrously overpriced time wasters (if done in hotel ballrooms with crummier equipment and less time than the home lab kits).

The edX AP Physics course, created by Boston University, says “The course covers all of the material for the test, supported by videos, simulations, and online labs.”  So it seems that they have no real labs in AP Physics, but only simulations.  While simulation is a wonderful thing, it does not develop much in the way of real-world lab skills.  I note that in the freshman design course I taught last year, often the only experience that students had had in building anything had been in their high school’s AP Physics courses.  That hands-on experience is very important for developing engineers.

So the edX courses will be valuable for students who have no other access to AP-level material (which is a lot—fewer than 5% of US high schools offer AP Computer Science, for example), but students will still usually be better off finding a community college course or other way to real lab experience for the AP science lab courses.

I wish edX great success on this endeavor, since I have seen first-hand the need for reasonable quality, affordable courses for advanced high school students, which many local high schools cannot provide, because they do not have enough students ready for the course in one place to justify creating and offering the courses.  It is a much more worthy market than trying to compete with brick-and-mortar colleges, which was the initial goal of Coursera, Udacity, and edX.  Udacity has already abandoned that goal in favor of corporate training (again, a reasonable market).  It is good to see the edX is moving in a reasonable direction also.  When will Coursera realize that their original “disruptive” dream was a pipe dream (probably as soon as they’ve burned all the venture capital)?

2014 June 21

2014 AP Exam Score Distributions

Once again this year, I’m posting a pointer to 2014 AP Exam Score Distributions:

Total Registration has compiled the following scores from Tweets that the College Board’s head of AP, Trevor Packer, has been making during June. These are preliminary breakdowns that may change slightly as late exams are scored.

Disclaimer: I have no connection with the company Total Registration, and do not endorse their services.  If the College Board would collect Trevor’s comment themselves, I’d point that page.  The main interest in AP result distributions comes in May, when students are taking the tests, and July when the students get the results.

The official score distributions (still from 2013 as of this posting) from the College board are at https://apscore.collegeboard.org/scores/about-ap-scores/score-distributions, at least until the College Board scrambles their web site again, which they do every couple of years, breaking all external links.  They post a separate PDF file for each exam, which makes comparison between exams more difficult (deliberately, I believe, since inter-exam comparison is not really a meaningful thing to do).  It is also difficult to get good historical data on how the exam scores have changed over time—College Board probably has it on their website somewhere, but finding stuff in their morass is not easy.

Views for my 2011 AP distribution post show the May and July spikes.

Views for my 2011 AP distribution post show the May and July spikes. This has been my most-viewed blog post, which is a bit embarrassing, since it has little original content.

My 2013 AP distribution post has not been as popular, probably because of search engine placement at Google.

My 2013 AP distribution post has not been as popular, probably because of search engine placement at Google.

My most popular post this year was How many AP courses are too many?, with about 10 views per day.  (It has also come in third over the lifetime of the blog, behind 2011 AP Exam Score Distribution and Installing gnuplot—a nightmare.) The question of how many AP courses seems to come up both in the fall, when students are choosing their schedules, and in the spring, when students are overwhelmed by how many AP courses they took.

The one AP exam my son took this year was AP Chemistry, for which only 10.1% got a 5 this year and about 53% pass (3, 4, or 5). We won’t have his score for a while yet, so we’re keeping our fingers crossed for a 5.  He finished all the free-response questions, so he’s got a good shot at it.

The Computer Science A exam saw an increase of 33% in test takers, with about a 61% pass rate (3, 4, or 5). The exams scores were heavily bimodal, with peaks at scores of 4 and at 1.  I wonder whether the new AP CS courses that Google funded contributed more to the 4s or to the 1s. I also wonder whether the scores clustered by schools, with some schools doing a decent job of teaching Java syntax (most of what the AP CS exam covers, so far as I can tell) and some doing a terrible job, or whether the bimodal distribution is happening within classes also.  I suspect clustering by school is more prevalent. The bimodal distribution of scores was there in 2011, 2012, and 2013 also, so is not a new phenomenon.  (Calculus BC sees a similar bimodal distribution in past years—the 2014 distribution is not available yet.) Update 2014 July 13: all score distributions are now available, and Calculus BC is indeed very bimodal with 48.3% 5s, 16.8% 4s, 16.4% 3s, 5.2% 2s, then back up to 13.3% 1s. Calculus AB has a somewhat flatter distribution, but the same basic shape: 24.3% 5s, 16.7% 4s, 17.7% 3s, 10.8% 2s, and 30.5% 1s. Overall calculus scores are up this year.  The 30.5% 1s on Calculus AB indicates that a lot of unprepared students are taking that test.  Is this the “AP-for-everyone” meme’s fault?

Physics B scores were way down this year, and Physics C scores way up—maybe the good students are getting the message that if you want to go into physical sciences, calculus-based physics is much more valuable than algebra-based physics. I expect that the algebra-based physics scores will go up a bit next year when they roll out Physics 1 and Physics 2 in place of Physics B, but that the number of students taking the Physics 2 exam will drop a lot.  I don’t expect a big change in the number of Physics C exam takers—schools that are offering calculus-based physics will not be changing their offerings much just because the College Board wants to have more low-level exams.

AP Biology is still  seeing the nearly normal distribution of scores, with 6.5% 5s and 8.8% 1s, so there hasn’t been a return to the flatter distribution of scores seen before the 2013 test change.

As always, the “easy” AP exams see much poorer average scores than the “hard” ones, showing that self-selection of who takes the exams is much more effective for the harder exams. When College Board and the high-school rating systems push schools to offer AP, the schools generally start by offering the “easy” courses, and push students who are not prepared to take the exams.  As long as we have stupid ratings that look only at how many students are taking the exams, rather than at how many are passing, we’ll see large numbers of failed exams.

2014 April 26

SIR!

Filed under: home school — gasstationwithoutpumps @ 22:33
Tags: , , , ,

Some readers of my blog and e-mail posts have been asking where my son will be going to college.

He filed his “Statement of Intent to Register” (SIR in UC jargon) and paid the deposit for University of California, Santa Barbara.

This post is a partial explanation of why he chose UCSB. I’m somewhat constrained, as I’ve been asked not to detail precisely where he did and did not get accepted. Suffice it to say that the number of acceptances was not different enough from the expected number to reject the null hypothesis that acceptances are random based on the probabilities inferred from the Common Data Set.  (Of course, with a sample size of one, that is not a very strong statement.)

As a family, we’re all pretty happy with UCSB as a choice, despite its reputation as a party school, the conservative community, and the difficulty of reaching it by air from northern California. What sold him on UCSB was the College of Creative Studies (CCS), which seems to be the best honors program in the UC system.

His major will be computer science, but it will be computer science in CCS, rather than computer science in engineering. What this means is that he basically crafts his own degree together with a faculty adviser. In his first quarter, he’ll take a special CCS freshman seminar with the other 10 or so CCS computer science freshmen, during which the instructor will try to assess the current level of expertise of each student and fill any holes they have in their prior learning, to place them in the right CS courses in future quarters.  The class is tiny (usually around 10 students) so the instructor doesn’t have to do one-size-fits-all teaching or advising. Because my son has already had UC-level applied discrete math (through concurrent enrollment at UCSC), he’ll be able to take upper-division courses like formal languages and automata theory right away. In fact, I suspect that he’ll end up skipping almost all the lower-division courses in CS.  He may end up opting to take some of them for review, or so that he’ll have an easy course on his schedule so that he has more time for research or acting, but he won’t be forced into huge lecture classes that have nothing new for him in them.

I looked over the lower-division (first two years) of CS at UCSB and it looks like my son has covered almost all of it already. He’s had several different programming languages (Scratch, C, Scheme, Python, Java, with bits of C++, JavaScript, Logo, assembly language), though he is most proficient now with object-oriented code in Python. One course (CMPSC 56) may have a little new material on exception handling and threading, and he might choose to take something like that to formalize his knowledge—when one learns a subject by reading reference manuals to do particular programming tasks there are sometimes unexpected holes in what you learn.  He’s done a fair amount with threading in Python, but not a lot with exception handling. CMPSC 64, on computer architecture and digital logic also has some new material for him.  The computer architecture will seem fairly simple to him after how deeply he’s been diving into the KL25 ARM Cortex M0+ architecture for programming both PteroDAQ and the light gloves, but some of the combinational and sequential hardware design will new.

One strong plus is that he’ll be able to join a research team his first year—CCS makes a concerted effort to get their students into research groups (in fact, one faculty member he met with when visiting UCSB has already tried to recruit him to a project). The UCSB computer science department is pretty good (their website claims top 10 for grad programs, but even allowing for hype they are probably in the top 20), and the department is fairly large with 32 tenure-track faculty, so there are a lot of different research projects he could join.  Computer engineering is lumped with EE in Electrical and Computer Engineering at UCSB, so there are more faculty and more research projects he could join there.

Another plus of the CCS program is a relaxing of the often bureaucratic nit-picking of general-education requirements. The CCS general-ed requirements are

  1. two courses in fields related to the student’s major, as determined in consultation with a CCS advisor;
  2. eight courses broadly distributed in fields unrelated to the student’s major, as determined in consultation with the advisor. These may be selected from courses offered by the College of Creative Studies, the College of Letters and Science, and the College of Engineering.

One of these courses must fulfill the Ethnicity Requirement: a course that concentrates on the intellectual, social and cultural experience, and history of one of the following groups: Native-Americans, African-Americans, Chicanos/Latinos, Asian-Americans. This course may be selected from a list of courses that fulfill the Ethnicity Requirement offered through the College of Letters and Science, or it may be a College of Creative Studies course that is classified as such.

Students also have to satisfy UC-wide requirements:

The reduction in bureaucratic bean counting means that he can probably satisfy all his general-ed requirements with fun courses in theater, linguistics, physics, math, and so forth.  The only rather arbitrary course is the Ethnicity Requirement, and he can satisfy that with any of several courses, including some theater ones.

One minor problem (shared by almost every college he applied to) is that he gets little relevant credit for his Advanced Placement exams. He’ll probably get 18 credits toward graduation (out of the 180 needed to graduate), but not all the units count towards his major requirements. He gets full credit for the calculus BC, but physics gives only useless non-STEM physics credit for the Physics C exams, the AP CS exam credit is pretty useless, and I’m not sure about chemistry (the page says “Natural Science 1B”, but there does not seem to be such a course—if they mean “CHEM 1B”, then it is useful credit towards his science requirements, assuming he does well enough on the exam in 2 weeks).  Because he is interested in taking some modern physics (quantum mechanics), he’ll probably end up either retaking calculus-based physics or talking his way into the more advanced courses and bypassing the huge lecture courses.

He should also get transfer credit for the community college Spanish courses and the UCSC math courses he has, which could mean another 16–18 credits.  These extra credits will not significantly speed his graduation, but they may give him the flexibility to avoid taking a heavy load some quarter, or to take an internship or study-abroad opportunity without falling behind. One normal benefit to having more credits is getting registration priority, but he already gets that as a CCS student, so that is less of a benefit for him than for others.

One little bonus for us as parents—UCSB is substantially cheaper than the private schools he also applied to, and we have saved enough in his 529 plan that we won’t need to take out any loans and he won’t have to work a meaningless job—he can spend his spare time doing research projects at the University or working on engineering projects for the startup company he and his friends are forming.

2014 January 17

CS commenters need to learn statistics

There was a recent report about how many students were taking AP CS exams, breaking out the information by gender, race, and state, which has been released in a few different forms.  Mark Guzdial’s blog post provides pointers to the data collected by Barbara Ericson.  Some of the comments provided on that post shows an appalling lack of statistical reasoning (like comparing states by subtracting percentages of different things).

So what are the interesting questions to ask of the data and how should they be handled statistically?

Most of the “gee-whiz” statements are about how few people in some group or other took (or passed) the AP CS exam:

  • No females took the exam in Mississippi, Montana, and Wyoming.
  • 11 states had no Black students take the exam: Alaska, Idaho, Kansas, Maine, Mississippi, Montana, Nebraska, New Mexico, North Dakota, Utah, and Wyoming.

Some people pointed out that some of these numbers may not be more than a small sample effect (no one took the exam in Wyoming, so having zero female test takers is not surprising).  How can we best state that a number is interesting?

Generally , this is done by creating a null model—one that computes the probability of different outcomes based on everything except the hypothesis being tested.  Then you look at how surprising the observed outcome is given the null model.   Exactly how the null model is constructed is crucial, as all that the statistical tests tell you is how badly your null model fits the data.

What sort of mathematical model should we be using for assigning probabilities to numbers of test takers (or numbers passing the test)?  One convenient one is a binomial distribution.  The binomial distributions are  a family of distributions over non-negative integers with two parameters N and p.  They are good for modeling the count of a number of independent events each of which occurs with some fixed probability.  If we think of each high school student in a state as having some (small) probability of taking the exam, then the number of exam takers can be modelled as a binomial distribution whose N value is the number of students and p the probability that each one takes the exam.  When N is large (as it would be for the number of high school students in a state) and Np is reasonably large, then the binomial distribution can be approximated by a normal distribution with mean Np and variance Np(1-p), but an even better approximation is to use the Poisson distribution with mean Np, which is what I’ll use here. The probability of zero test takers: P(0)= \binom{n}{0} p^0 (1-p)^{n-0} = (1-p)^n \approx e^{-np}.

So all we need to set the parameters of our null model is an expected number of test takers based on everything except what we wanted to test.  For example, if we wanted to test whether black test takers were under-represented in Maine, we would need a model that predicted how many black students would take the test, perhaps using the probability that students in Maine would take the test independent of race and the fraction of students in Maine that are black.  For Maine, there were 161 test takers, and 0 black test takers.  I don’t know the racial mix of high school students in Maine, but Wikipedia gives the black fraction of the whole state population as 1.03%.  Thus the expected number of black test takers is 1.658, and we can use e^{-1.658} as the probability of seeing zero black test takers by chance.

UPDATE: 2014 Feb 1.  Some values in the following table corrected, due to clerical errors in copying from spreadsheet (I’m not sure which I hate worse, spreadsheets or HTML tables—they’re both awful formats).

state # test takers state % black expected black test takers under-rep p<
Alaska  21  4.27%  0.897  0.41
Idaho 6 47  0.95%  0.086 0.447  0.92 0.64
Kansas 12 47  6.15%  0.738 2.891  0.48 0.056
Maine  161  1.03%  1.658  0.19
Mississippi 2 1  37.3%  0.746 0.373  0.47 0.69
Montana 0 11  0.67%  0 0.074  1 0.93
Nebraska 12 46  4.50%  0.540 2.070  0.58 0.126
New Mexico 7 57  2.97%  0.208 1.693  0.81 0.184
North Dakota 1 9  1.08%  0.011 0.097  0.99 0.91
Utah 11 103  1.27%  0.140 1.308  0.87 0.27
Wyoming 2 0  1.29%  0.026 0  0.97 1

Even before we do a correction for having 51 hypotheses (50 states plus District of Columbia), none of these “no black students” states shows significant under-representation of black students. In fact, it would have been significantly surprising if the test taker in North Dakota had been black. None of the states had so few students that a black test taker would have been surprising (except Wyoming).

One can do similar computations to show that the lack of women in Mississippi, Montana, and Wyoming is not surprising.  Montana looks surprising if treated as a single hypothesis (p<0.004), but not after multiple-hypothesis correction (E-value=0.21). Even combining all three states (which increases the number of hypotheses enormously and would call for a stronger multiple-hypothesis correction), the under-representation of women in those states is not statistically significant.

There are states that do have significant under-representation of women: for example, Utah had 103 test takers, only 4 of whom were women. With an expected number of about 51.5, this is p<1.4E-16. Even with 51× multiple hypothesis correction, this under-representation is hugely significant.  Looking nationwide, total counts were 5485 female test takers out of 29555 total test takers.  That’s p< 1.4E-1677. The highest percentage of female test takers was in Tennessee, with 73 out of 251, which is  p< 2.6E-7, again highly significant.

Tennessee also had a high proportion of black test takers with 25 out of 251.  With an expected number of 42.12, this is p<0.003 (still significantly under-represented).  To see if black students were under-represented nationwide, one would have to add up the expected numbers for each state and see how the actual number compared with the expected number.  (I’m certain that the under-representation is hugely significant since even the states with high numbers of black test takers are under-represented,  but I’m too lazy to do the multiplication and addition needed.)

The case can clearly be made for female and black students being under-represented, though pointing to the states with 0 female or 0 black test takers is not the way to do it. (From a marketing standpoint, rather than a statistical one , shouting “no black test takers in these states”, “no female test takers in these other states” may be exactly the right way to get attention, even though the real story about blacks and females is in the states where there were enough test takers to say something about them after dividing them into subgroups.)

A case could also be made for some states having far fewer CS AP test takers than others.  One would need to come up with an expected number of test takers from some model (for example, by state population as a share of national population, or by number of total AP test takers in state as share of national total AP test takers).  The second model would correct for state-to-state differences in age distribution or in popularity of AP exam taking in general.  One could also base predictions on some other STEM test, such as AP Calculus, if one wanted to control for different amounts of STEM instruction in different states.

Let’s look at the states with no black test takers again, to see if they are significantly under-represented in CS.  There were 29555 AP CS tests taken nationwide and 3,824,691 AP tests nationwide total, so we would expect the CS tests taken in a state to be 0.77% of the total for the state.

state #  CS test takers # all test takers expected CS test takers p < E-value
Alaska 21 4570 35.31 0.0066 0.34
Idaho 6 47 9723 75.13 6.3E-25 3.3E-4 3E-23 1.7E-4
Kansas 12 47 15339 118.53 5.95E-36 6.25E-14 3E-34 3.2E-12
Maine 161 14051 108.58 0.9999
Mississippi 2 1 9032 69.79 1.23E-27 3.5E-29 6E-26 1.8E-27
Montana 0 11 4868 37.62 4.59E-17 3.4E-7 2E-15 1.7E-5
Nebraska 12 46 11117 85.91 1.9e-23 1.7E-6 1E-21 8.8E-7
New Mexico 7 57 13365 103.28 3.7E-35 4.7E-7 2E-33 2.4E-5
North Dakota 1 9 2295 17.73 3.7E-7 0.018 2E-5 0.91
Utah 11 103 35721 276.03 2.4E-101 5.6E-23 1E-99 2.8E-21
Wyoming 2 0 2050 15.84 1.9E-5 1.3E-7 0.00096 6.7E-6

Of these eleven states, eight appear to be under-represented in CS test takers (Maine is significantly over-represented in CS test takers).  When I do the multiple-hypothesis correction for having 51 different “states” (including the District of Columbia), the mild under-representation in Alaska and North Dakota is no longer significant, but the other nine eight are.

So the zero black AP CS test takers for the nine states can be fairly confidently attributed to the lack of AP CS test takers, and in Maine to the shortage of black students.  For Alaska, the lack of black AP CS test takers is probably due to the shortage of AP CS test takers in the state.

One can generalize the techniques here to any method of predicting the mean number of students in some category, to see whether the observed number is significantly smaller than the predicted number.  When the predicted number is small, even 0 students may not be statistically significant under-representation.

Next Page »

Blog at WordPress.com.

%d bloggers like this: