Gas station without pumps

2016 July 6

Outcomes assessment

Filed under: Uncategorized — gasstationwithoutpumps @ 21:47
Tags: , , , ,

In Confessions of a Community College Dean: The Whole and the Sum of Its Parts, “Dean Dad” wrote about outcomes assessment:

The first was a discussion on campus of the difference between the “course mapping” version of outcomes assessment and the “capstone” version.  Briefly, the first version implies locating each desired student outcome in a given class—“written communication” in “English composition,” say—and then demonstrating that each class achieves its role.  The idea is that if a student fulfills the various distribution requirements, and each requirement is tied to a given outcome, then the student will have achieved the outcomes by the end of the degree.

Except that it doesn’t always work that way.  Those of us who teach (or taught) in disciplines outside of English have had the repeated experience of getting horrible papers from students who passed—and even did well in—freshman comp.  For whatever reason, the skill that the requirement was supposed to impart somehow didn’t carry over.  Given that the purpose of “general education” is precisely to carry over, the ubiquity of that experience suggests a flaw in the model.  The whole doesn’t necessarily equal the sum of the parts.

In a “capstone” model, students in an end-of-sequence course do work that gets assessed against the desired overall outcomes.  Can the student in the 200 level history class write a paper showing reasonable command of sources?  The capstone approach recognizes that the point of an education isn’t the serial checking of boxes, but the acquisition and refinement of skills and knowledge that can transfer beyond their original source.  

I have certainly experienced the phenomenon of students doing well in freshman writing courses, but being unable to write reasonably in upper-division and graduate engineering courses—indeed, that is why I insisted on UCSC’s computer engineering curriculum requiring a tech writing course 30 years ago (and teaching it for about 14 years). I continue to teach writing-intensive courses—my current main course, Applied Electronics for Bioengineers, requires about 5–10 pages of writing from each pair of partners each week (though that load will drop in half next year, as the course is split into two quarters). The writing level of students increases noticeably during the quarter, though a number of students continue to have problems with organization, with paragraph structure, with grammatical sentences, and with punctuation (particularly commas).

But evaluating writing just once in a capstone course is no solution—that just invites a lowering of standards so that the failure rate is not too high. Nor can one guarantee that capstones will necessarily be a good check of all the desired outcomes. Indeed, one of the capstone options for the bioengineering degree at UCSC does not involve major writing—the results are presented orally, and oral presentations are frequent in the course.

I recently wrote an evaluation of the “Program Learning Outcomes” (PLOs) for the bioinformatics program at UCSC (and refused to write one for the bioengineering program—it was hard enough getting the info needed for the bioinformatics assessment).  The assessment falls more in the “various distribution requirements” camp than in the “capstone” camp.  We did not have much trouble showing that the PLOs were assessed thoroughly, largely because the PLOs were chosen to be things that the faculty really cared about and included in their course designs, rather than “wouldn’t it be nice if students magically acquired this” outcomes.

Here is the report (minus any attachments):
Program Learning Outcome Assessment
Bioinformatics BS program
Spring 2016
The bioinformatics program was asked to assess at least one of our Program Learning Outcomes (PLOs) [https://www.soe.ucsc.edu/departments/biomolecular-engineering/programs/bs-bioinformatics]:

A bioinformatics student completing the program should

  • have a detailed knowledge of statistics, computer science, biochemistry, and genetics;
  • be able to find and use information from a variety of sources, including books, journal articles, and online encyclopedias;
  • be able to design and conduct computational experiments, as well as to analyze and interpret data;
  • be able to apply their knowledge to write programs for answering research questions in biology;
  • be able to communicate problems, experiments, and design solutions in writing, orally, and as posters; and
  • be able to apply ethical reasoning to make decisions about engineering methods and solutions in a global, economic, environmental, and societal context.

Because the program graduates so few students, it did not seem very productive to examine the output of just the most recent graduating class—we would need a decade’s worth of output to get statistical significance, and any information about changes in the curriculum would be lost in viewing such a long time scale. Instead, we asked the faculty in our department who teach the required courses of the major how they assess the students for the objectives that they cover.

A Google form was used to collect the information.  The faculty were prompted

Please complete a separate form for each course that you teach that is a required or elective course for the Bioinformatics major (select from list below).  Only courses that are required (or are part of a small list of constrained choices) matter here, since we are looking for guarantees that all students are meeting the PLOs, not that there is some elective path that would cover them.

Please provide a sentence or two describing how your course(s) provide evidence that the student has met the outcome.  Be explicit (what assignment in what course provides the evidence)!  Your course does not have to provide evidence for all the PLOs—one or two PLOs supported strongly in a course is more convincing.

Responses were collected for 7 courses: BME 80G Bioethics, BME 110 Computational Biology Tools, BME 130 Genomes, BME 185 Technical Writing for Bioengineers, BME 205 Bioinformatics models & algorithms, BME 211 Computational Systems Biology, and BME 230/L Computational Genomics.  Each of these courses is required, except BME 185 (bioinformatics students may take CMPE 185) and only one of BME 211 and 230/L is required. Our hope is that all the PLOs are assessed thoroughly in these courses, so that we do not need to rely on courses outside our control for outcome assessment.

The responses to the questions are attached as a CSV file and will be summarized here [I’m not including the attachments in this blog version]. Despite the prompt, faculty did not always explain exactly how the outcome was assessed.

detailed knowledge of statistics, computer science, biochemistry, and genetics

All courses except BME 80G (bioethics) test some aspect of this objective. Most of the assignments in most the courses depend heavily on this content knowledge, and the faculty are convinced that this objective is being adequately addressed.  Note that we did not include the courses from other departments that actually teach the fundamental material—just the courses within our department that rely on it.

able to find and use information from a variety of sources, including books, journal articles, and online encyclopedias;

All the courses rely on students gathering information from a variety of sources, with different levels of search and different levels of interpretation needed in each course. All courses have at least one assignment that assesses students’ ability to use information from a variety of sources, and most have several.  Again, because of the pervasive nature of the objective in all our courses,  the faculty have no concern that the outcome is being inadequately assessed.

able to design and conduct computational experiments, as well as to analyze and interpret data;

All the courses except Bioethics require some data analysis, and several require computational experiments, but only BME 211 and 230/L have the students doing extensive design of the experiments.

able to apply their knowledge to write programs for answering research questions in biology;

BME 80G (Bioethics) and BME 110 (Bioinformatics tools) do not require programming, and BME 185 (Technical writing) has minimal programming, but all the other courses require writing computer programs, and the programming tasks are all directly related to research questions.  In BME 211 and BME 230/L the questions are genuine open research questions, not just classroom exercises.

able to communicate problems, experiments, and design solutions in writing, orally, and as posters; and

All courses except BME 110 (Bioinformatics tools) require written reports, and several of the courses require oral presentation. Only BME 185 (Technical writing) requires poster presentation, so we may want to institute a poster requirement in one of the other courses, to provide more practice at this form of professional communication, as posters are particularly important at bioinformatics conferences.

able to apply ethical reasoning to make decisions about engineering methods and solutions in a global, economic, environmental, and societal context.

BME 80G (Bioethics) is specifically focused on this PLO and covers it thoroughly, with all the assessments in the course testing students’ ability to apply ethical reasoning.  There is also coverage of research and engineering ethics in BME 185 (Technical Writing).  Although most of the courses do not teach ethics, the writing assessment in each of the courses holds students to high standards of research citation and written acknowledgement of collaboration.

Overall, the faculty feel that the PLOs are more than adequately assessed by the existing courses, even without looking at assessments in obviously relevant courses for the objectives from outside the department (such as AMS 132 for statistical reasoning). Because so many of the objectives are repeatedly assessed in multiple courses, they see no point to collecting portfolios of student work to assess the objectives in yet another process.

Only poster presentation and ethical reasoning are assessed in only one course, and practical research ethics is assessed in almost every course, leaving only poster presentation as a skill that might need to be reinforced in improvements to the curriculum.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: