Gas station without pumps

2016 July 6

Outcomes assessment

Filed under: Uncategorized — gasstationwithoutpumps @ 21:47
Tags: , , , ,

In Confessions of a Community College Dean: The Whole and the Sum of Its Parts, “Dean Dad” wrote about outcomes assessment:

The first was a discussion on campus of the difference between the “course mapping” version of outcomes assessment and the “capstone” version.  Briefly, the first version implies locating each desired student outcome in a given class—“written communication” in “English composition,” say—and then demonstrating that each class achieves its role.  The idea is that if a student fulfills the various distribution requirements, and each requirement is tied to a given outcome, then the student will have achieved the outcomes by the end of the degree.

Except that it doesn’t always work that way.  Those of us who teach (or taught) in disciplines outside of English have had the repeated experience of getting horrible papers from students who passed—and even did well in—freshman comp.  For whatever reason, the skill that the requirement was supposed to impart somehow didn’t carry over.  Given that the purpose of “general education” is precisely to carry over, the ubiquity of that experience suggests a flaw in the model.  The whole doesn’t necessarily equal the sum of the parts.

In a “capstone” model, students in an end-of-sequence course do work that gets assessed against the desired overall outcomes.  Can the student in the 200 level history class write a paper showing reasonable command of sources?  The capstone approach recognizes that the point of an education isn’t the serial checking of boxes, but the acquisition and refinement of skills and knowledge that can transfer beyond their original source.  

I have certainly experienced the phenomenon of students doing well in freshman writing courses, but being unable to write reasonably in upper-division and graduate engineering courses—indeed, that is why I insisted on UCSC’s computer engineering curriculum requiring a tech writing course 30 years ago (and teaching it for about 14 years). I continue to teach writing-intensive courses—my current main course, Applied Electronics for Bioengineers, requires about 5–10 pages of writing from each pair of partners each week (though that load will drop in half next year, as the course is split into two quarters). The writing level of students increases noticeably during the quarter, though a number of students continue to have problems with organization, with paragraph structure, with grammatical sentences, and with punctuation (particularly commas).

But evaluating writing just once in a capstone course is no solution—that just invites a lowering of standards so that the failure rate is not too high. Nor can one guarantee that capstones will necessarily be a good check of all the desired outcomes. Indeed, one of the capstone options for the bioengineering degree at UCSC does not involve major writing—the results are presented orally, and oral presentations are frequent in the course.

I recently wrote an evaluation of the “Program Learning Outcomes” (PLOs) for the bioinformatics program at UCSC (and refused to write one for the bioengineering program—it was hard enough getting the info needed for the bioinformatics assessment).  The assessment falls more in the “various distribution requirements” camp than in the “capstone” camp.  We did not have much trouble showing that the PLOs were assessed thoroughly, largely because the PLOs were chosen to be things that the faculty really cared about and included in their course designs, rather than “wouldn’t it be nice if students magically acquired this” outcomes.

Here is the report (minus any attachments):
Program Learning Outcome Assessment
Bioinformatics BS program
Spring 2016
The bioinformatics program was asked to assess at least one of our Program Learning Outcomes (PLOs) []:

A bioinformatics student completing the program should

  • have a detailed knowledge of statistics, computer science, biochemistry, and genetics;
  • be able to find and use information from a variety of sources, including books, journal articles, and online encyclopedias;
  • be able to design and conduct computational experiments, as well as to analyze and interpret data;
  • be able to apply their knowledge to write programs for answering research questions in biology;
  • be able to communicate problems, experiments, and design solutions in writing, orally, and as posters; and
  • be able to apply ethical reasoning to make decisions about engineering methods and solutions in a global, economic, environmental, and societal context.

Because the program graduates so few students, it did not seem very productive to examine the output of just the most recent graduating class—we would need a decade’s worth of output to get statistical significance, and any information about changes in the curriculum would be lost in viewing such a long time scale. Instead, we asked the faculty in our department who teach the required courses of the major how they assess the students for the objectives that they cover.

A Google form was used to collect the information.  The faculty were prompted

Please complete a separate form for each course that you teach that is a required or elective course for the Bioinformatics major (select from list below).  Only courses that are required (or are part of a small list of constrained choices) matter here, since we are looking for guarantees that all students are meeting the PLOs, not that there is some elective path that would cover them.

Please provide a sentence or two describing how your course(s) provide evidence that the student has met the outcome.  Be explicit (what assignment in what course provides the evidence)!  Your course does not have to provide evidence for all the PLOs—one or two PLOs supported strongly in a course is more convincing.

Responses were collected for 7 courses: BME 80G Bioethics, BME 110 Computational Biology Tools, BME 130 Genomes, BME 185 Technical Writing for Bioengineers, BME 205 Bioinformatics models & algorithms, BME 211 Computational Systems Biology, and BME 230/L Computational Genomics.  Each of these courses is required, except BME 185 (bioinformatics students may take CMPE 185) and only one of BME 211 and 230/L is required. Our hope is that all the PLOs are assessed thoroughly in these courses, so that we do not need to rely on courses outside our control for outcome assessment.

The responses to the questions are attached as a CSV file and will be summarized here [I’m not including the attachments in this blog version]. Despite the prompt, faculty did not always explain exactly how the outcome was assessed.

detailed knowledge of statistics, computer science, biochemistry, and genetics

All courses except BME 80G (bioethics) test some aspect of this objective. Most of the assignments in most the courses depend heavily on this content knowledge, and the faculty are convinced that this objective is being adequately addressed.  Note that we did not include the courses from other departments that actually teach the fundamental material—just the courses within our department that rely on it.

able to find and use information from a variety of sources, including books, journal articles, and online encyclopedias;

All the courses rely on students gathering information from a variety of sources, with different levels of search and different levels of interpretation needed in each course. All courses have at least one assignment that assesses students’ ability to use information from a variety of sources, and most have several.  Again, because of the pervasive nature of the objective in all our courses,  the faculty have no concern that the outcome is being inadequately assessed.

able to design and conduct computational experiments, as well as to analyze and interpret data;

All the courses except Bioethics require some data analysis, and several require computational experiments, but only BME 211 and 230/L have the students doing extensive design of the experiments.

able to apply their knowledge to write programs for answering research questions in biology;

BME 80G (Bioethics) and BME 110 (Bioinformatics tools) do not require programming, and BME 185 (Technical writing) has minimal programming, but all the other courses require writing computer programs, and the programming tasks are all directly related to research questions.  In BME 211 and BME 230/L the questions are genuine open research questions, not just classroom exercises.

able to communicate problems, experiments, and design solutions in writing, orally, and as posters; and

All courses except BME 110 (Bioinformatics tools) require written reports, and several of the courses require oral presentation. Only BME 185 (Technical writing) requires poster presentation, so we may want to institute a poster requirement in one of the other courses, to provide more practice at this form of professional communication, as posters are particularly important at bioinformatics conferences.

able to apply ethical reasoning to make decisions about engineering methods and solutions in a global, economic, environmental, and societal context.

BME 80G (Bioethics) is specifically focused on this PLO and covers it thoroughly, with all the assessments in the course testing students’ ability to apply ethical reasoning.  There is also coverage of research and engineering ethics in BME 185 (Technical Writing).  Although most of the courses do not teach ethics, the writing assessment in each of the courses holds students to high standards of research citation and written acknowledgement of collaboration.

Overall, the faculty feel that the PLOs are more than adequately assessed by the existing courses, even without looking at assessments in obviously relevant courses for the objectives from outside the department (such as AMS 132 for statistical reasoning). Because so many of the objectives are repeatedly assessed in multiple courses, they see no point to collecting portfolios of student work to assess the objectives in yet another process.

Only poster presentation and ethical reasoning are assessed in only one course, and practical research ethics is assessed in almost every course, leaving only poster presentation as a skill that might need to be reinforced in improvements to the curriculum.

2015 December 26

Syllabi for splitting Applied Electronics into two courses

In order to split the Applied Electronics for Bioengineers course into two courses, as I suggested in Considering splitting Applied Electronics course, I need to fill out course approval forms to get the courses approved by the Committee on Educational Policy.  They’ve changed the forms this year, so that there are now three documents needed:

If I were requesting a general-education code for the course, I would also have to fill out one of the thirteen general-education forms (corresponding to the 13 possible general-education codes for a course at UCSC), listed at the bottom of the supplemental sheet.

The supplemental sheet was simplified this year, by pushing all the general-education forms out to separate forms, but the requirement for a course syllabus is new.  Basically, the supplemental sheet asks more or less the same questions as before, rephrased to “where on the course syllabus …?”  The “learning outcomes” question is new, as it reflects a relatively new bureaucratic approach to curriculum design.  The learning outcomes make a lot more sense at the course level than at the degree level, where the administration has been pushing for them.

Here are my first more-or-less complete drafts of my sample syllabi for the split course:

The split course is a pair of 4-credit courses, representing a total time of  about 250 hours (240–263 hours), 140 of which are contact hours (3.5 hours of lecture and 3.5 hours of lab a week).  I’m thinking in terms of MWF lectures (70 minutes each) and TTh labs (105 minutes each).  That should be easier to schedule than the 7-unit BME 101/L these courses will replace, which takes about 220 hours (210–232 hours), 95 of which are contact hours (3.5 hours of lecture and 6 hours of lab a week). The increased contact hours should result in students learning more, as many of our students are not very efficient at learning on their own.

One thing I’ll have to decide is whether to require all bioengineering majors to take both courses, or whether BME 51A is enough for the biomolecular concentration. For the bioelectronics and assistive technology: motor concentrations, both parts are needed both for the content and for the lab experience.  But for the biomolecular and assistive technology: cognitive/perceptual concentrations, the courses are mainly there to teach engineering design practices.  The assistive technology: cognitive/perceptual concentration relies on software courses for design content, and so BME 51A is probably enough for them, but there are very few design courses for the biomolecular concentration, as biomolecular lab work is very slow, and a full-year capstone sequence is barely enough for one iteration of one prototype.

2015 June 4

Last lab of Spring 2015

Filed under: Circuits course — gasstationwithoutpumps @ 22:50
Tags: , , , ,

I spent all day today in the lab for the electronics class, from around 9:30 a.m. until the last students packed up and left at 8:30 p.m.  This was the last lab of the quarter, and I had decided to stay until all the students had left.

Most of the students got their EKG boards soldered up and working today (there may be a few who left without demoing their working boards, and the last group at 8:30 p.m. had just had some more wiring errors pointed out to them).

I’m amazed sometimes at how basically competent designers can be very careless in their wiring, rushing through the placement and soldering without carefully checking each connection. The result is a 10-minute savings in wiring time, and a 4-hour or more cost in debugging and resoldering time.

Tomorrow in class I plan to go over a couple of 1-transistor amplifier designs, but that shouldn’t take the whole time.I’ll give them some pointers to companies that sell parts and kits that might be of use to them: Digikey, Mouser, Jameco, Sparkfun, Adafruit Industries, Itead Studio, Seeedstudio, Smart Prototyping, Elecrow, OSH Park, Makershed, … . And I’ll be sure to mention some local resources: Santa Cruz Electronics and Idea Fab Labs.

I also hope to remind the students of some of the goals of the course, and try to see whether the goals have been met.  I quote from the supplemental form for the course renaming that was approved this spring (effective next year).

The Program Learning outcomes for the bioengineering program are as follows:
A bioengineering student completing the program should

  • have a broad knowledge of science and engineering disciplines including biology, chemistry, physics, mathematics, statistics, and computer science; [Not relevant to this course]
  • be able to apply their broad knowledge to identify, formulate, and solve engineering design problems; [Students passing BME 101L will be able to design simple amplifiers and RC filters for a variety of sensor-interfacing applications.]
  • be able to find and use information from a variety of sources, including books, journal articles, online encyclopedias, and manufacturer data sheets; [Students passing BME 101L will be able to find and read data sheets for a number of analog electronics parts.]
  • be able to design and conduct experiments, as well as to analyze and interpret data; [Students passing BME 101L will be able to measure signals with multimeters, oscilloscopes, and data-acquisition devices,  plot the data, and fit non-linear models to the data.]
  • be able to communicate problems, experiments, and design solutions in writing, orally, and as posters; [Students passing BME 101L will be able to write coherent design reports for electronics designs with block diagrams, schematics, and descriptions of design choices made.] and
  • be able to apply ethical reasoning to make decisions about engineering methods and solutions in a global, economic, environmental, and societal context. [Not relevant to this course]

So tomorrow I plan to ask where the students feel that they are able to design simple amplifiers and RC filters, whether they can find and read data sheets for analog parts, whether they can measure signals with multimeters, oscilloscopes, and data acquisition devices, whether they can plot the data and fit non-linear models to it, and whether they can write coherent design reports.

I had some unofficial goals for the course also: to turn a few of the students into electronics hobbyists, to encourage a few to declare the bioelectronics concentration of bioengineering, to teach some tool-using, maker skills (calipers, micrometer, soldering iron, …), and to make all of them better at attacking problems by dividing into subproblems with clear interfaces between the subproblems.  I’ll ask about those things also.

I’ll also want some detailed suggestions for the course.  (So far I’ve gotten one: fume extractors for the lab for use when soldering.)  Some things I’m curious about include

  • Should the first amplifier lab (the low-power audio amp lab) be changed to use a single power supply and solder up the board, so that the board can be used as a preamp for the class-D power amp lab later?  We could then also do an emitter follower (common collector) class-A amplifier using the preamp board.  If they solder up a pre-amp, then we could eliminate soldering the instrumentation amp for the blood pressure lab.
  • Should I redesign the prototyping board to have more room for resistors and no instrumentation amp slot, making it more suitable for the preamp lab and the EKG lab?  A new custom board is still cheaper than something like the $4 perma-proto boards from Adafruit.
  • Should I switch from 18-turn trimmer pots to 3/4-turn trimmers with shafts?  The ones with shafts tend to be easier to turn, but not as precise and the multi-turn worm gear pots.  There are 3/4-turn trimmer pots that play nicely with a breadboard, though they take up a bit more space than the worm-gear trimmers we used this year.
  • Are there tools or parts that almost no one used?
  • Are there tools or parts that should be added to the lab kit? If so, at what price do they stop being attractive?
  • Should students buy oscilloscope and voltmeter probes, like they do at UCSB, rather than having to deal with broken probes or probes locked inconveniently to equipment?
  • Should there be more practice questions in the book? (currently I have very few, with almost all the questions being part of prelab assignments)
  • Does there need to be a “what you are already expected to know” section or chapter, to review material that students are supposed to know already?
  • Which labs took up too much time for the amount of learning achieved? How can they be streamlined?
  • How much time did the course really take total for the quarter?
  • What suggestions do students have for more fun labs?

2015 January 6

New quarter, lots of work

Filed under: Uncategorized — gasstationwithoutpumps @ 21:21
Tags: , , , ,

The Winter quarter started yesterday, and my schedule is going to be even fuller than I thought.

I only have two classes this quarter, both 2-unit courses. One course is the freshman design seminar, the other is the senior thesis writing course, so I’m getting students at both the beginning and the end of their undergraduate experience. Two 2-unit courses sounds like a pretty light load, but the courses are more work for me than most 2-unit courses.

For the freshman design seminar, I’m having to create content as I go—we have some lab access this year that we didn’t have last year, and I want to get the students building stuff as soon as I can. But that means my having to build the stuff, to make sure it is feasible, and to figure out how to digest it down to the point where students can have success in the design and construction. I’ll probably be spending more time on the course than any of the students—they’re supposed to have 3.5 hours a week in class or lab and 2.5 hours a week on their homework—I’m expecting to spend more than that prepping for class, and even more on grading.

At the end of last quarter, I filled out a form on the library web site asking for an information session for the course, and I filled it out again yesterday.  Usually the librarians are pretty good about responding to the requests, and I was wondering why I hadn’t heard from them, so I sent e-mail to a couple of the librarians I’d worked with in the past—it seems that the forms on their web page weren’t entering the data into the request system (reason still not known), and they’d never gotten my requests to schedule the information session.  They forwarded my request on to other librarians (a different team handles lower-division information sessions, which worries me a bit, because I don’t want the sort of pablum they usually give freshmen—the ones I’m used to working with know that I want solid training on search techniques), but I’ve not heard back from that team yet. It’s a good thing that I haven’t figured out my schedule for the quarter yet, as I’ve no idea when they’ll be available.

I’m also still waiting to hear back from the engineering lab staff about what training the students need to be able to use the tools in the fab lab, and how I can get access.  I probably need to go and talk with them in person—I’ve gotten no responses to my e-mail requests.

Today I asked the bioengineering undergrads (by e-mail) for volunteers to lead lab tours of the labs they work in, and to explain to the freshmen how they managed to join the lab. My explanation of how to join labs never seems as convincing to the freshmen as hearing directly from juniors and seniors that are in labs. The lab tours are always rather cool, because there is a lot of interesting research going on by bioengineering undergrads here. So far, I’ve had one faculty member tell me he’s assigned some students to do the tour of his lab, but they haven’t contacted me to schedule it yet. Tomorrow, I may ask the seniors doing thesis research if any of them have time to help out.

The senior writing class meets for 3.5 hours this week, but for the rest of the quarter we’ll be meeting only 1.75 hours a week (Wednesdays 5–6:45) as a group, mainly to go over common problems and for them to practice presentations with an audience. I was expecting about 12 students in the class, but I have 19, six of whom had not taken the prerequisite tech-writing course. This pisses me off a bit, since I don’t have time in a 2-unit course to teach everything that should be covered in the 5-unit writing course. But the fault is not entirely theirs—the tech-writing class has been full with students not able to get in every quarter for the past few years.

I had to do some last-minute restructuring of the thesis-writing class because of the size.  The last time I’d taught the class (2 years ago), I read each student’s draft thesis 5 times, providing four rounds of detailed feedback before the final draft was evaluated. I won’t be able to do that this year. Instead, I’ve broken the class into three groups, who turn in their papers out of phase (6 this week, 7 next week, 6 the week after, then repeat).  This plan results in only 3 rounds of feedback before the final version, not 4, but still has me reading a thesis (50-to-100 pages) every day.

I will be meeting with each thesis writer individually for 20 minutes a week, for them to practice their elevator talk, to discuss their research with me, and to discuss their writing.  In some cases in the past, I could not understand their projects from their writing, and it took several rounds of discussion before I realized what they were trying to do, so that I could help them word it comprehensibly.  (In at least one case, the student had misunderstood the statistics so badly that what they were claiming as exciting results were all indistinguishable from chance.) Nineteen students at 20 minutes each is another 6.3 contact hours.

Another difference from the last time I taught the course is that most of the students are in the second quarter of the three-quarter thesis project, rather than in the last quarter. This was a deliberate rescheduling on my part, because I was appalled at how many students had been working for over 20 weeks and written absolutely nothing.  In some cases they hadn’t even properly done their background research, and found that they’d wasted most of the 20 weeks on rather useless stuff that didn’t address their real research questions.  (I did send some politely worded e-mail messages to the faculty who had been “supervising” them, though I wanted to scream at them for their incompetence as advisers.)  Since then, I’ve also gotten much more careful when signing the independent study forms as the undergraduate director and program chair (the forms call for both signatures, but for the bioengineering program, both signatures are mine) to make sure that the students are writing something every quarter.

In any case, the student will not have finished theses at the end of the writing course, but almost-finished ones, with just a few results to fill in next quarter and a little discussion.

So for the two small classes this quarter, I have 11.6 contact hours, 12 hours of grading, and probably 4–5 hours of prep work each week.  I also have 2 hours a week scheduled for office hours, 1.5 for meetings with the department manager, and 2 hours for the department research seminar. So I’m up to 17 hours a week of scheduled meetings and 12 hours of grading—and that doesn’t count the 2–3 hours a week I’ll need to spend with grad students and 1–2 with the nanopore research group, nor the extra advising load that will happen this quarter as all the sophomores try to declare their majors.. So figure at least 22 hours scheduled and 12 hours grading, before we get to my administrative duties.

Today was spent almost entirely on administrative duties—both the department and the bioengineering program are having to do self studies this year for WASC accreditation next year, and I’m stuck writing the whole self study for the bioengineering program and the undergrad portion for the department. I had written a draft of the bioengineering self study over the two-week break while campus was closed, but I only found the form with the prompts for the self study today.  None of what I wrote was wasted, but they wanted a whole lot more bureaucratic bullshit about “Program Learning Outcomes”, a top-down management approach that ensures that every faculty member will run screaming when asked to do anything about the curriculum.  (Actually, our faculty are much more subtle than that—they just nod their heads and disappear.)

I was particularly annoyed by the loaded question in one prompt:

Overall, how has program assessment (including all steps: defining the program learning outcomes, developing the curriculum matrix or rubrics, interpreting the findings) been used to guide improvement of the program? Provide at least one example since the last review of an improvement made to some aspect of the program’s curriculum or course effectiveness.

My current draft answer (which I’m afraid will have to be toned down a bit) is

This prompt assumes that the Program Learning Outcome process has some beneficial effect on improving the program, for which there is no empirical evidence in the bioengineering program.  The improvements in the program have come despite the time wasted on the PLO process, not because of it. The extensive curricular changes described in Section 2 considered data from senior exit interviews, careful thought about what concentrations were uniquely offerable at UCSC, what made pedagogic sense, and what courses and resources were available.  The overly bureaucratic PLO process, particular the curriculum matrix and “rubrics”, took time away from thinking about and discussing the curriculum and pedagogy, diverting it to satisfying arbitrary bureaucratic requirements.

As you might gather, I’m pretty pissed about the PLO process, which calls for an annual report on assessment of one of the Program Learning Outcomes.  I had to make up the outcomes myself last year (none of the faculty were interested), and I’ll have to do all the “assessment” and writing of the report this year for two programs (none of the other faculty are interested). If the process served to trigger discussion about curriculum or pedagogy among the faculty, it might be worthwhile, but it has had the opposite effect, making faculty even less willing than usual to engage in discussions of the curriculum. So I’m stuck writing bullshit reports for administrators who’ll probably never read them, when I’d much rather be teaching, advising students, or fixing problems in the curriculum with other faculty.  (Or even, gasp, doing some research!)

Note: there have been some substantial improvements to the curriculum since the last review—I spent countless hours last year meeting with other faculty (one-on-one or in small groups) to completely overhaul the bioengineering curriculum.  I’m pretty pleased with the result (and students who’ve compared the old and the new curriculum wish that I’d overhauled it a couple years earlier, so that they could have done the new curriculum). But it really was the case that the “Program Learning Outcomes” was a distraction and a sideshow that cut into the time I had to think about and fix the curriculum.

Anyway, after meeting with the BME chair and department manager this morning, to set the agenda for this Friday’s monthly BME faculty meeting, I spent most of the rest of the day trying to wrestle the draft of the bioengineering self-study into shape so that I could share it with the bioengineering faculty, while answering the loaded questions of the administrators. (I’d shared the first draft and gotten feedback from only two faculty out of the thirty—both of them instructors, not tenure-track faculty—two of the best teachers in the School of Engineering.)  I’m unlikely to get much out of the rest of the faculty—I have no carrot or stick to encourage compliance (the program chair of the bioengineering program comes with 0 resources, not even a course buyout for hundreds of hours of work on the curriculum and the self-study).

I had been planning to spend the afternoon doing a first draft of the undergraduate self-study for the bioinformatics program, but the amount of extra work needed on the bioengineering one took up my whole afternoon. I spent a lot of it asking staff for the data that the administrators wanted—much of the data they asked for was not available and will take the staff several days to try to compile. Some of the prompts were particularly irksome:

Provide a brief description of the learning outcomes assessment process, including a multiyear assessment plan, references to assessment instruments provided in Appendix III (e.g., a capstone rubric), and a summary of the annual assessment findings regarding each of the program learning outcomes (as many as have been
assessed to date). Comment on what the indirect evidence from the undergraduate major (UCUES) survey, such as students’ self-reported competency levels and satisfaction with educational experience, indicates in terms of the strengths and weaknesses of  the program. How do measures of direct evidence of student learning agree with indirect measures?                           

What is the UCUES survey data I’m supposed to analyze?  Well, after trawling through my e-mail I found a promise that I’d get the data in December. I never did, so I asked. Oh, they might have that by next week, maybe.  So much for trying to get the self-study written before the thesis-a-day grading starts tomorrow!  The “assessment instruments in Appendix III?” There aren’t any—I only wrote the bloody outcomes last year under duress, and there haven’t been any “assessment instruments” (by which they probably mean bullshit surveys and meaningless statistical analysis of portfolios using made-up “rubrics”). So, summary of annual assessment findings: “there haven’t been any and they wouldn’t mean squat even if there were some”.

I do have several years’ worth of portfolios from graduating seniors as well as notes from exit interviews (both portfolios and exit interviews are requirements for graduation), and I’ve actually read a lot of the student theses in depth. That’s useful to do, and a lot of the ideas for the curriculum overhaul came from discussing the curriculum with graduating seniors at the exit interviews, but turning the portfolios and interviews into an “assessment instrument” using their 50-page guide to the process?—pure, unadulterated educrap.


%d bloggers like this: