Gas station without pumps

2011 February 11

Science Fair judging

Filed under: Science fair — gasstationwithoutpumps @ 16:04
Tags: , ,

Science Fair season is now in full swing.  I’ve judged two big school science fairs already this year: an elementary school fair with about 160 projects and a middle school fair with about 100 projects, and I still have the County Science Fair to judge in March.

I talked a colleague and our lab group (we both have small labs in similar fields, so our lab groups meet together to get critical mass for discussions and journal clubs) into helping the judging of elementary school.  None of them had judged a science fair before, and one foreign student had never even seen a science fair before, but they all had fun interviewing the kids, evaluating the quality of the science, and checking that the kids had done the work themselves. I urge other scientists, engineers, and faculty in STEM fields to take some time this month and next to help schools judge their science fairs.  Around here there are usually more than enough computer engineers as county judges (Seagate is a major sponsor of the County Science Fair), but there are always shortages in the behavioral sciences and biological sciences, because of the popularity of those fields for student projects.

I’ve been judging at the county science fairs since 1998 (though I missed 2002 and 2004 because of sabbatical), and I’ve even judged at the state science fair (in 2010).  Over the years I’ve seen a lot of different projects (and a lot of the same project over and over—I really don’t need to see another lemon battery or the Stroop effect again).  While it is theoretically possible for a child to actually learn some electrochemistry and do a good “lemon battery” project, or to do something interesting with measuring reactions times in the presence of cognitive dissonance, I have yet to see it, and a student doing such a project would have an enormous barrier to overcome in convincing a judge that their project is not the same as the 1000s of other lemon battery and Stroop effect projects.

Project selection is probably the hardest part of science fair for children, their parents, and their teachers—harder even than doing the projects.  There are a lot of web sites (like Science Buddies) intended to help students choose projects, replacing classic sources like the Amateur Scientist column of Scientific American (written by  C.L. Strong 1952–1978, Jearl Walker 1978–1988, Shawn Carlson up to 2001).  A lot of the Amateur Scientist columns taught how to build some sort of scientific instrumentation cheaply, that could then be used for a number of different investigations, but many “science fair” web sites provide cookie-cutter projects that result in kids just going through the motions of doing a project, without thinking about what they are doing or caring much what happens.

Parents and teachers (and to a lesser extent the kids themselves) always want to know what the judges are looking for.  This varies from fair to fair, depending on the instructions the judges are given, the amount of experience they have had, and their own beliefs about what constitutes a good project.  Regional and state fairs usually have their instructions to judges on their web sites, for judges to read ahead of time.  Teachers and parents can also read them, to know what instructions the judges have been given, and how much weight is supposed to be given to different aspects of the project.  Parents and kids usually put too much emphasis on the flashiness of the final poster board, and not enough on the underlying science and keeping a contemporaneous lab notebook.  Teachers tend to teach a simplified view of the scientific method, emphasizing the final half or third of the process (the experimental validation of a hypothesis).  I’ll talk more about proper construction of a hypothesis later.

One thing that I’ve found difficult to communicate to kids, parents, and teachers is that science fair judging, like any award process that has to select a few “winners” out of a large pool of contestants, has an inherent random element.  Just like federal funding, the best project doesn’t always win.  As science fair judges (and as panelists judging grant proposals for NSF or NIH), we do our best to pick out those projects that are really excellent, but it is not possible to do so perfectly.  Some people get very bitter if their kids don’t win, accusing the fairs of being rigged.  I have seen no evidence to support accusations of any sort of systematic bias in the fairs I’ve participated in.  I have occasionally seen a judge use an inappropriate criterion (like a student’s gender, ethnicity, or school) to favor one student over another, but other judges have usually explained that science fairs are not about spreading awards around evenly, but rewarding those students who do excellent work, whoever and wherever they are.

So if all the judges are trying to pick the best projects, why do some projects get ranked “incorrectly”?  What are the sources of the randomness?  There are many.

One of the biggest problems is that there are so many projects that no one judge really sees more than a small fraction of them.  The best in that small sample may be the best overall or may be far from the best, but the judge has little way of telling. There are many different methods used to try to combine the judgments of different judges.  The most common, and poorest, method is to ask for some sort of numerical score from each judge for each project, and average those scores.  Even if a judge manages to give scores consistently to all the projects he or she sees (which is not a trivial task), there is almost no consistency between judges, so a project’s score may depend more on which judges rated it than on its quality (the same problem plagues NIH review panels, and they’ve not come up with an adequate solution there, despite years of trying and much more at stake). Attempts to correct for different calibrations of judges by using the mean and standard deviation of the individual judges score (the “average Z-score” method) overcorrect, because they are based on the assumption that every judge sees a sample that reflects the overall population.

The bigger fairs tend to have the judges get together after the evaluation of individual projects and argue for particular projects, often going back as a group to look at top projects and hear what makes these particular projects stand out.  This (like the NIH panel meetings) helps correct for different calibrations of the individual judges, but adds a different random element: how persuasive is the advocate for a particular project.  Sometimes they are more interested in seeing a good project in their field win than in seeing the best project win.  Sometimes they overvalue a project in another field, because they are not aware of the flaws in the method or the lack of originality (these problems are even bigger at NIH panels!).

There is also little time for judges to really dig into a project at any depth, to see how thoroughly the kids understood what they were doing, where the ideas came from, and what the kids did with the ideas.  A judge may have 10–40 projects to judge in a few hours, so rarely gets more than 5–10 minutes with any one project.  Some projects only require a minute or two to be seen as routine copies of standard projects off the web, but politeness to the kids requires that judges spend about as much time with them as with the excellent projects (in the younger grades, it is rarely the kids’ fault if their projects are poorly chosen, and having a good experience at science fair one year can lead them to try a more ambitious project the next year). The short judging time means that the judges see only a limited view of the project, and different  judges may get different views, and so reach very different conclusions about the overall value of the project.

The expertise of judges at a science fair varies enormously.  At the regional and state levels, just about everyone judging is a professional engineer or scientist, but at the school fairs it is often parents and teachers doing the judging, sometimes unconsciously using information from outside the science fair to judge the effort of the kids. Even when outside expert judges are used, there is often a mismatch between the field of the project and the skills of the judge.  I have seen projects that none of the available judges really had the expertise to evaluate critically (there was a high-school project last year that came very close to being a PhD project in biochemistry, and my son once did a project on fractals with derivation of the fractal dimension that none of the sixth-grade judges that year had the math background to follow). It helps to have a few judges with very wide interests in science, to try to fill in the holes where there is a lack of specific experts (I’ve done that a few times at the county science fair).  But the unavailability of judges with specific expertise does serve as an additional source of randomness in the judging.

In some science fairs, the applicability of a result my trump the quality of the science.  I have noticed that at the California State Science fair, a reasonable environmental technology or health application will often beat a strong pure science project.  Engineering projects with a veneer of hypothesis often have a better chance than true hypothesis-driven research, and both do much better than discovery research (which science fairs, like NIH funding panels, don’t seem to be aware exists).  The Santa Cruz County Science Fair does a better job, I think, of selecting good science and engineering projects than the state science fair does.  The focus at the county is more on whether the science is good than on whether it pretends to solve an environmental problem (there are separate awards for particularly good environmental projects).  Still, judges tend to be more impressed by applied science than pure science.  Students have to make the judges care about the project, and that is easier to do with applied projects.

One of the big reasons that some parents and some schools reject science fairs is that they feel there is too much parental involvement—that the parents are doing science fair projects for the kids, and that this isn’t fair.  I’ve seen a little of that at school fairs, but almost none at the county level.  I suspect that there may be regional variations in this, as I have heard of bigger problems with parents doing projects in areas where parents measure their worth by what colleges their kids get into.  It certainly is not the dynamic here, and only a few clueless parents do their kids’ work.  Judges pick this up pretty fast, when the kids don’t understand the project and can’t talk about it reasonably.  The kids are often pretty honest and tell the judges when their parents have built the apparatus and written the report.  It is usually the projects near the bottom of the pool that have too much parental involvement, though, so regional and state fairs don’t have to deal much with it.

A more common fairness concern locally is that it is often the “faculty brats” (children of university professors) who win at science fairs.  Some attribute this to easier access to fancy lab equipment, but I don’t think that is the reason for the difference.  First, judges usually give bonus points for homemade equipment and clever use of common items, so fancy lab equipment usually raises the bar: students have to do much more impressive things if they have fancy tools to work with. Second, any student who needs access to fancier tools can try to get mentoring arranged by the County Office of Education. There are quite a few students otherwise unaffiliated with the university who have gotten mentoring and lab access through this system, and more could do so.  (The teachers know about it, but they don’t always pass the information on to parents and students in a timely way.)

The reason that faculty brats do well has, I think, more to do with their education than simple access. As I mentioned in an earlier blog about my own upbringing, engineers and scientists do a lot of the science education at home.  The children of scientists may be 5 or 6 years ahead of their age mates in science education, which is naturally reflected in the sophistication of their projects.  Furthermore, the university faculty may put a high value on science as an activity, supporting and encouraging their children to take on 3-month, 6-month, or even multi-year science fair projects, which end up being much more impressive than the 1-month (and often really only 1-week) projects that a lot of kids attempt.

My son’s projects for the past few years have involved writing computer programs using free software on an old iMac (now about 6 years old) and demoing the programs on a 10-year-old Titanium Mac Laptop. The projects weren’t impressive for the equipment used, but for the amount of effort he put in and how much he learned about computer science and math in the process of doing the projects. In mentoring my son on his projects I do give him some help in understanding the fundamental concepts of the science, but the program design and implementation is all his own. In some areas (like GUI tools and web pages) he is now more advanced than me anyway, and my involvement in those parts is limited to discussing with him what features he wants from a GUI-builder and helping him install PyGUI and PyObjC (which did not install properly the first time, because easy_install was not updated when Python 2.7 was installed).  I’ve ended up giving him less help than I give many grad students, because he is already a better programmer than some of them, and more capable of finding the information he needs in documentation on the web.

So as a judge I’m aware that some kids can do amazing things and I try very hard to find out whether the kids did the project or are serving as fronts for a parent or mentor.  I do raise the bar higher for kids who use fancier equipment, but in the end the decision comes down to who has done the work and produced the best science.

Advertisements

17 Comments »

  1. Kevin:

    Thanks for your thoughtful post and particularly for sharing it w all of us Westlake judges. I enjoyed talking to the students enormously; assigning a numerical score to their efforts far less so. I wish that we had been able to discuss the projects collectively as, it seems, you do at the higher levels. I think of two projects particularly — both were effectively presented by the students who were terribly enthusiastic and clearly knew there subjects well. One however was completed w considerable help from the Sci faculty parents, the other w far less adult assistance. The latter suffered from several inadequacies (eg insufficient replication) while the former was near perfect; according to our scoring protocol I has no choice but to assign higher scores to the project w more (and fairly sophisticated) parental assistance. I want to emphasize that the students responsible for each of these were very impressive, showing excellent understanding. I also felt that the help received w the “fancier” project wasn’t unreasonable. But the two either shouldn’t be compared directly or should have been scored as more or less dead even. Ah well! Perhaps I’ll have a little more sympathy for the reviewers when I get the results from my last proposal.

    Comment by Peter Nelson — 2011 February 11 @ 20:39 | Reply

  2. Kevin:

    Thank you for such a thorough and truthful view of science fairs and their inherently difficult judging issues. There is one area of judging ‘variable’ that you had not mentioned in this post, however I believe was touched on in the previous one. That is concerning the practice of parents in the role of science fair judges. Not only do parents (facultied or otherwise) sometimes have considerable input into the student’s projects, but in some cases are indeed also judges in the very fair their child is competing in! While this practice does not necessarily have to cause problems (if handled appropriately), my son (now a college student) became the unfortunate victim of parental judge ‘battering’ and inappropriate advocacy activities at a county level fair several years ago. Indeed, the parental judge’s son was granted Grand Champ status and proceeded to the Intel fair. Supposedly, the county fair administration at the time was ‘not aware’ that the parent judge should have recused himself when his son became in the running for champion honors. My opinion: NO parents should be permitted to judge in ANY level of science fair where their own children are in competition. Thanks for continuing your informative blog. I read it with great interest.

    Comment by Donna Sherman — 2011 February 13 @ 12:52 | Reply

    • I have judged at the County Science Fair for many years, and continued to do so even after my son started entering the science fair. I have been careful to recuse myself from any judging that might involve him (which means both the category he is in and overall decisions for his division). I have never judged at a school fair in which he was participating.

      I think that it is possible for parents to serve as judges, but it does require a careful firewall.

      I have seen only one instance of inappropriate advocacy at the county science fair, and it was not from a parent, but from a judge who turned out to also be the mentor for the student whose work he was championing.

      Comment by gasstationwithoutpumps — 2011 February 13 @ 15:04 | Reply

      • I agree with your comment of a need for a ‘careful firewall’. Most definitely such parameters had not been in place for the county fair experience I described previously. I suppose, had we wanted to ‘make waves’ and cause hassles, we could have complained to the Intel ISEF governs and had the student prohibited from participation at the international level….but, to what purpose? My son was honored as the Reserve Champ and would have then been granted the trip to Intel. Hopefully the local admin learned from their ‘mistake’, as it was. I had not even considered the possibility of a student’s mentor acting as judge…. :O

        Comment by Donna Sherman — 2011 February 13 @ 17:49 | Reply

  3. My son has never done a school science fair (his school didn’t have them). But he was an active 4-H member, and exhibited science projects at the county and state fair levels for many years. 4-H judging gives ample time for the judge to discuss the project with the student, often up to half an hour. Fair judges usually have some expertise in the category they are judging, and at the state fair level, are usually university profs, since 4-H is run through the state university extension service. And at the state fair level, judging was done in a small group setting, and usually lasted around two hours. Students had to explain their project to the judge and the other 4-H winners in their grouping, answer questions, ask questions about the other projects, and talk about any problems/challenges etc. they had along the way, as well as talk about how the project could be expanded for future study/research. Granted, most of these projects were not pure science research, but the judging process itself was always an integral and valuable part of the learning process.

    Comment by Jodi S. — 2011 February 15 @ 05:47 | Reply

    • I have heard good things about 4H from other parents around the country, and I know that there are 4H groups in the underwater remotely-operated vehicle contest for which I’m coaching a team for the high school, but the few 4H projects I’ve seen at the County Fair have not impressed me much.

      The nearest 4H group is about 8 miles away, and it has never seemed worth that long and hilly a bike ride.

      Comment by gasstationwithoutpumps — 2011 February 15 @ 07:40 | Reply

      • 4-H has been a great fit for my son because the clubs are multi-aged, and boy/girl. Lots of caring adults. Lots of different opportunities, both social and academic. The clubs are run by the kids. Our kids use Robert’s Rules and it’s great watching a group of early elementary kids running a meeting by themselves. County fair projects are going to vary widely. The best of the best move on to the state fair, where you are going to see a lot of wonderful work- though most non-livestock exhibits are going to be more practical in nature, as opposed to pure research. But we’ve had kids in our club who have constructed machines for specific tasks on their farms, worked on plant breeding and done wildlife survey work.

        Comment by Jodi Summit — 2011 February 15 @ 10:34 | Reply

  4. […] Science Fair judging (gasstationwithoutpumps.wordpress.com) […]

    Pingback by California State Science Fair judging « Gas station without pumps — 2011 May 5 @ 08:17 | Reply

  5. […] Science Fair judging […]

    Pingback by Blogoversary « Gas station without pumps — 2011 June 5 @ 10:52 | Reply

  6. […] Science Fair judging […]

    Pingback by Blog year in review « Gas station without pumps — 2012 January 1 @ 14:17 | Reply

  7. Could you give more consideration to formation of hypotheses?

    In my limited experience, science fair projects start with a question formed neither from observation nor from some fact reported in a book the kids have read. In early (but not THAT early) years, they employ anything they can measure and say “boys are better than girls at…”, or vice versa, chosen at random. These directional hypotheses are based on air! Since their teachers are okaying this, I did not know how to deal with it.

    Comment by Pat Dolan — 2012 January 25 @ 08:21 | Reply

    • It is certainly true that many elementary and middle-school teachers have no idea what “hypothesis” really means in a scientific context. That is probably worth a post.

      Comment by gasstationwithoutpumps — 2012 January 25 @ 13:50 | Reply

      • I think in early education the word “hypothesis” is translated into “guess”. The oversimplification for the sake of comprehension leads to misunderstanding of the concept of what is an informed hypothesis.

        Comment by Kim Miller — 2012 February 4 @ 07:26 | Reply

  8. […] year, about this time, I posted some notes on judging, which may be worth re-reading.  This year, I want to talk more about what things I see as common […]

    Pingback by Science fair time again « Gas station without pumps — 2012 February 6 @ 20:53 | Reply

  9. […] Science fair judging (gasstationwithoutpumps.wordpress.com) […]

    Pingback by More science fair advice « Gas station without pumps — 2012 February 15 @ 14:25 | Reply

  10. […] Science Fair judging […]

    Pingback by How to be a Good Science Fair Judge « Gas station without pumps — 2012 March 6 @ 20:48 | Reply

  11. […] Science Fair judging […]

    Pingback by Second Blogoversary « Gas station without pumps — 2012 June 2 @ 18:15 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at WordPress.com.

%d bloggers like this: