Gas station without pumps

2016 September 29

GRE Analytic Writing favors bullshitters

Filed under: Uncategorized — gasstationwithoutpumps @ 22:33
Tags: , , , ,

My son recently took the GRE exam to apply for grad school in computer science.  The test has changed since I took it in 1973, but it still looks a lot like the SAT exam, which has also changed since I took it in 1970.  The multiple-choice section is still primarily 9th and 10th grade material, so it is a bit surprising that only 5.5% of CS students, 11.4% of physics students, and 15.3% of math students get 170, the highest possible score, on the quantitative reasoning section. [All data in this post from https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf]

The “quantitative reasoning” questions are primarily algebra and reading simple graphs, so the banking and finance students do best with 15.5% getting 170. The scores would be more valuable for STEM grad school admissions if they included some college-level math (calculus, ODEs, combinatorics, statistics, … ), but the general GRE has always been based on an extremely low math level.

The verbal scores are perhaps less surprising, with philosophy being the only major with over 3% getting a 170 (5.1%), and with some of the education and business majors doing the worst—except for computer science, where 8% get the lowest scores (130–134), with the next worst major being accounting with 2.7% having 130–134.  I wonder how much of the difference here is due to the number of native and non-native speakers, as computer science certainly attracts a lot more foreign students than most majors.

I was most interested in looking at the “Analytical Writing” scores, since I’ve not seen much correlation between them and the quality of student writing on the grad school applications I’ve reviewed over the last decade.  I was interested in two things: the mean score and the fraction that got 4.5 or better (the fraction getting 5.5 or better is too small to be worth looking at).  Again computer science and electrical engineering stand out as having extremely low means and small fractions of students having 4.5 or better.  I have not found any analyses of the GRE scores that separate native speakers of English from non-native ones—I wonder how much of the effect we see here is due to being native speakers and how much is due to curricular differences.

Here is the table of all the broad categories in the order that ETS provided them:

Subject

Mean writing

%ile ≥4.5

Computer and Information Sciences

3.1

8.8

Electrical and Electronics

3.1

6.7

ENGINEERING

3.3

12.6

Civil

3.3

13.2

Industrial

3.3

9.8

Mechanical

3.3

12.3

PHYSICAL SCIENCES

3.4

17.3

Accounting

3.4

12.3

Banking and Finance

3.4

10.7

Natural Sciences ─ Other

3.5

14.8

Materials

3.5

19.4

BUSINESS

3.5

15.2

Other

3.5

14.7

Agriculture, Natural Res. & Conservation

3.6

18.0

Mathematical Sciences

3.6

21.0

Chemical

3.6

21.6

Early Childhood

3.6

16.0

Student Counseling and Personnel Srvcs

3.6

17.3

Business Admin and Management

3.6

17.8

Health and Medical Sciences

3.7

19.0

Chemistry

3.7

23.8

Other

3.7

23.1

Other

3.7

21.3

Arts ─ Performance and Studio

3.7

24.3

Administration

3.7

21.9

Elementary

3.7

21.3

Special

3.7

19.5

Other

3.7

23.7

LIFE SCIENCES

3.8

21.3

Biological & Biomedical Sciences

3.8

26.0

Earth, Atmospheric, and Marine Sciences

3.8

25.4

Physics and Astronomy

3.8

26.8

Economics

3.8

27.8

Sociology

3.8

28.2

EDUCATION

3.8

23.9

Curriculum and Instruction

3.8

21.4

Evaluation and Research

3.8

23.6

SOCIAL SCIENCES

3.9

29.1

Psychology

3.9

26.6

Higher

3.9

29.7

Anthropology and Archaeology

4.0

34.7

Foreign Languages and Literatures

4.0

37.2

Secondary

4.0

33.9

Political Science

4.1

42.9

ARTS AND HUMANITIES

4.1

40.8

Arts ─ History, Theory, and Criticism

4.1

38.5

History

4.1

40.4

Other

4.1

38.6

English Language and Literature

4.2

45.2

Philosophy

4.3

52.7

 OTHER

Architecture and Environmental Design

3.4

13.1

Communications and Journalism

3.7

23.3

Family and Consumer Sciences

3.7

20.7

Library and Archival Sciences

4.0

34.3

Public Administration

3.8

23.7

Religion and Theology

4.2

46.5

Social Work

3.6

16.7

The table is more interesting in sorted order (say by %ile ≥4.5 on Analytical Writing):

Subject

Mean writing

%ile ≥4.5

Electrical and Electronics

3.1

6.7

Computer and Information Sciences

3.1

8.8

Industrial

3.3

9.8

Banking and Finance

3.4

10.7

Mechanical

3.3

12.3

Accounting

3.4

12.3

ENGINEERING

3.3

12.6

Architecture and Environmental Design

3.4

13.1

Civil

3.3

13.2

Other

3.5

14.7

Natural Sciences ─ Other

3.5

14.8

BUSINESS

3.5

15.2

Early Childhood

3.6

16.0

Social Work

3.6

16.7

PHYSICAL SCIENCES

3.4

17.3

Student Counseling and Personnel Srvcs

3.6

17.3

Business Admin and Management

3.6

17.8

Agriculture, Natural Res. & Conservation

3.6

18.0

Health and Medical Sciences

3.7

19.0

Materials

3.5

19.4

Special

3.7

19.5

Family and Consumer Sciences

3.7

20.7

Mathematical Sciences

3.6

21.0

Other

3.7

21.3

Elementary

3.7

21.3

LIFE SCIENCES

3.8

21.3

Curriculum and Instruction

3.8

21.4

Chemical

3.6

21.6

Administration

3.7

21.9

Other

3.7

23.1

Communications and Journalism

3.7

23.3

Evaluation and Research

3.8

23.6

Other

3.7

23.7

Public Administration

3.8

23.7

Chemistry

3.7

23.8

EDUCATION

3.8

23.9

Arts ─ Performance and Studio

3.7

24.3

Earth, Atmospheric, and Marine Sciences

3.8

25.4

Biological & Biomedical Sciences

3.8

26.0

Psychology

3.9

26.6

Physics and Astronomy

3.8

26.8

Economics

3.8

27.8

Sociology

3.8

28.2

SOCIAL SCIENCES

3.9

29.1

Higher

3.9

29.7

Secondary

4.0

33.9

Library and Archival Sciences

4.0

34.3

Anthropology and Archaeology

4.0

34.7

Foreign Languages and Literatures

4.0

37.2

Arts ─ History, Theory, and Criticism

4.1

38.5

Other

4.1

38.6

History

4.1

40.4

ARTS AND HUMANITIES

4.1

40.8

Political Science

4.1

42.9

English Language and Literature

4.2

45.2

Religion and Theology

4.2

46.5

Philosophy

4.3

52.7

Note that all the fields that call for precise, mathematical reasoning do poorly on this test, but those which call for fuzzy, emotional arguments with no mathematical foundation do well—the test is designed to favor con men. I believe that this is partly baked into the prompts (see the pool of issue topics and, to a lesser extent, the pool of argument topics), partly the result of having the writing being done entirely without access to facts (benefitting those who BS over those who prefer reasoning supported with well-sourced facts), and partly the result of having graders who are easily swayed by con men.

I believe that most of the graders are trained in the humanities, and so are more swayed by familiar vocabulary and rhetoric.  If ETS had science and engineering professors doing the grading (which they would have a hard time getting at the low rates they pay the graders), I think that the writing scores would come out quite different.

Of course, there are curricular differences, and science and engineering faculty are mostly not paying enough attention to their students’ writing (and I can well believe that CS and EE are the worst at that). But I don’t think that even engineering students who do very, very good engineering writing will necessarily score well on the GRE analytical writing test, which seems to favor rapid writing in only one style.

I will continue to give relatively little weight to Analytical Writing GRE scores in graduate admissions. The untimed essays that the students write for the applications are much closer to the sort of writing that they will be expected to do in grad school, and so much more indicative of whether their writing skills are adequate to the job. I will continue to interpret low GRE scores as a warning sign to look more closely at the essays for signs that the students are not up to the task of writing a thesis, but high GRE writing scores are not a strong recommendation—I don’t want grad students who are good at bull-shitting.

2015 February 25

Freshman design seminar writing notes

Along with the senior-thesis writing course this quarter, I’m also teaching a freshman design seminar. Many of the problems in their first design reports are similar to the problems I see in senior theses (Senior thesis pet peeves, More senior thesis pet peeves, and Still more senior thesis pet peeves). I hope that by catching them early, I can squelch the problems.

Here are some things I saw in the first design report turned in by the freshmen:

  • Every design document should have a title, author, and date. If the document is more than one page log, it should have page numbers.
  • Passive voice should be used very sparingly—use it to turn sentences around to pull the object into the first position, when that is needed to get a smooth old-info-to-new-info flow.  Sometimes you can use it to hide the actor, when you really don’t know who did something, but that should be very rare.
  • Errors in schematics, programs, block diagrams, and other low-redundancy representations are very serious.  In the circuits class, any error in a schematic triggers an automatic REDO for the assignment.  I’m not as harsh in the freshman design class, but there is no notion of “just a little mistake” in a schematic.
  • The battery symbol is not the right way to show a voltage source that is not a battery.  Use the power-port symbol, to indicate connect to a power supply that is not included in the schematic, or include the Arduino board from which you are getting power as a component in the schematic.
  • Bar charts are not appropriate for all that many data representations in the physical and biological sciences.   If you have 2-D data, use a scatter diagram.  A bar should only be used when the area of the bar communicates the quantity of something that is labeled in discrete classes.  (And even then a single point is often clearer.)
  • Captions on figures should be about a paragraph long.  Remember that people generally flip through a paper looking at the pictures before deciding whether to read it.  If the figures and captions are mysterious, they’ll give up without ever reading the paper.  A lot of academic authors, when writing a paper for publication, start by choosing the figures and writing the captions.  Those figures and captions then form the backbone of the paper, which is written to explain and amplify that backbone.
  • In academic writing, figures are treated as floating insertions, not fixed with respect to the text.  Therefore, it is correct to refer to the figures by name, “Figure 1”, but not by location (“above” or “below”). Every figure in a paper should be referred to explicitly by name in the main body of the text, and the floating insertion put near where the first reference to the figure occurs.
  • Citations in modern scholarly works are not done as footnotes—those went out of style 50 or 60 years ago, and only high school teachers still use that style.  Modern papers put all the citations at the end (in any of several different styles, usually specific to a particular journal).  I have a slight preference for reference lists that are sorted by author, rather than by order cited in the paper, and I have a preference towards high-redundancy reference list formats rather than minimalist ones, but I don’t have a particular style that I recommend.
  • There is no point to saying “web” in a citation—if something comes from the web, then give me the URL (or DOI). For material that is only on the web (not citable as a journal article), you must give the URL or DOI.
  • When typing numbers, never start them with a decimal point—use a leading zero to prevent the easily missed leading decimal point. Even better is to follow the engineering convention of using numbers between 1 and 1000 with exponents of 10 that are multiples of three.  That is, instead of saying .01, or even 0.01, say 10E-3.  The advantage is that the powers of 1000 have prefix names, so that .01A becomes 10mA.  Don’t worry about significant figure meanings, because engineers express significance explicitly, not through imprecise sig-fig conventions.  That is, and engineer would say 10mA±2mA, not 1.E-2A (which a physicist would interpret as 10mA±5mA) or 1.0E-2A (which a physicist would interpret as 10mA±0.5mA).
  • In describing where components are in a schematic diagram, “before” and “after” don’t make much sense.  I have no idea what you mean if you say that a resistor is before an LED. When engineers use “before” or “after” it is generally in an information-flow sense.  For example, you may filter before amplifying or amplify before filtering, but if a resistor and capacitor are in series, neither is “before” the other.
  • Students use “would” in many different ways, but mostly incorrectly, as if it were some formal form of “was” or “will be”, while it is actually a past subjunctive form of the modal auxiliary “will”.  There are many correct uses of “would” in general English, but in technical writing, it is usually reserved for “contrary to fact” statements. When a student writes “I would grow bacteria for 2 days”, I immediately want to know why they don’t.
  • The pronoun “this” is very confusing, as the reader has to work out what antecedent is meant. A lot of effort can be saved if “this” and “these” are not used as pronouns but only as demonstrative adjectives modifying a noun. This usage is much easier for people to follow, as the noun helps enormously in figuring out the antecedent.  If you can’t figure out what noun to use, then your reader has no hope of understanding what you meant by “this”.
  • “First” is already an adverb and needs no -ly. The same is true of “second”, “third”, and “last”.  For some reason, no one makes the mistake with “next”, which follows the same pattern of being both an adjective and an adverb.  I wonder why that is?

2013 April 11

Reading student writing

Filed under: Uncategorized — gasstationwithoutpumps @ 20:16
Tags: , , , , ,
I’m spending this quarter reading senior theses: five drafts each of 13 theses.  None of these students are working for me—I’m just running the class in which they are trying to convert what they’ve done for the past year into something resembling a thesis.  About half the class had not written anything on their projects before taking this senior-thesis seminar—a serious dereliction of duty on the part of their faculty supervisors, who should have been requiring a draft at the end of each quarter of work.
I meet with each student weekly (for about half a hour, though I seem to run over more often than not) in addition to the 1 ¾ hour weekly class meeting and all the reading and scrawling on drafts.  I’m currently spending over 14 hours a week on this 2-unit course (my light teaching quarter, as I had 8 units in the Fall and 7 in the Winter), and the amount of time will probably go up as the drafts get longer and more complete.
I know that a lot of MOOC-proponents are pushing automatic grading of papers as a cost-effective way to handle classes with over 1000 students.  Quite frankly, the idea appalls me—I can’t see any way that computer programs could provide anything like useful feedback to students on any sort of writing above the 1st-grade level.  Even spelling checkers (which I insist on students using) do a terrible job, and what passes for grammar checking is ludicrous nonsense.  And spelling and grammar are just the minor surface problems, where the computer has some hope of providing non-negative advice.  But the feedback I’m providing covers lots of other things like the structure of the document, audience assessment, ordering of ideas, flow of sentences within a paragraph, proper topic sentences, design of graphical representation of data, feedback on citations, even suggestions on experiments to try—none of which would be remotely feasible with the very best of artificial intelligence available in the next 10 years.
Providing good feedback on the student theses requires a good understanding of what the students are talking about (which I have gotten mainly from hearing years of research talks by their supervisors, since none are working on subjects within my areas of expertise) plus an understanding of what makes good technical writing.  Either one without the other is nearly useless, which is why students who worked on their thesis drafts as part of a tech writing course last quarter are not much better off than those who didn’t—the tech writing instructor knew none of the content, and so could not see when the ideas were in the wrong order, misstated, or otherwise badly presented. Misuse of jargon and incorrect presentation of data were also missed. The main advantage for the students who wrote a draft for the tech writing course is that they have more complete draft to start from, with a few of the surface errors already removed.
If even expert tech writing instructors with decades of experience can’t produce good enough feedback on student writing, what hope is there that automated programs can do anything useful?
I’m not alone in my thinking that automated feedback on student writing is an incredibly stupid idea. John Warner, in his post 22 Thoughts on Automated Grading of Student Writing, wrote


5. I don’t know a single instructor of writing who enjoys grading.

6. At the same time, the only way, and I mean the only way, to develop a relationship with one’s students is to read and respond to their work. Automated grading is supposed to “free” the instructor for other tasks, except there is no more important task. Grading writing, while time-consuming and occasionally unpleasant, is simply the price of doing business.

7. The only motivations for even experimenting [with], let alone embracing, automated grading of student writing are business-related.


12. The second most misguided statement in the New York Times article covering the EdX announcement is this from Anant Argawal, “There is a huge value in learning with instant feedback. Students are telling us they learn much better with instant feedback.” This statement is misguided because instant feedback immediately followed by additional student attempts is actually antithetical to everything we know about the writing process. Good writing is almost always the product of reflection and revision. The feedback must be processed, and only then can it be implemented. Writing is not a video game.

14. The most misguided statement in the Times article is from Daphne Koller, the founder of Coursera: “It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right.”

15. I’m sorry, that’s not misguided, it’s just silly.

…22. The purpose of writing is to communicate with an audience. In good conscience, we cannot ask students to write something that will not be read. If we cross this threshold, we may as well simply give up on education. I know that I won’t be involved. Let the software “talk” to software. Leave me out of it.

I pulled out the points that resonated most for me, but I recommend reading the whole of John Warner’s post.

2011 November 26

Criteria for reasoning

Filed under: Uncategorized — gasstationwithoutpumps @ 00:37
Tags: , , , ,

In her post Evaluating Thinking: Why These Criteria?, Mylène describes the criteria she provides her students for what makes a good reasoning process.  She started with a list from the Foundation for Critical Thinking:

  • Clarity
  • Precision
  • Accuracy
  • Significance
  • Relevance
  • Logic
  • Fairness
  • Breadth
  • Depth

and modified it to fit better with her class and her students (an intro physics course for people training to be technicians, so students from the middle of the distribution in terms of intelligence and prior education).  Her modified list is about the same length but more specific to physical models:

  1. clarity
  2. precision
  3. internal consistency
  4. connection to our model
  5. connection to our experience/intuition
  6. seamless chain of cause and effect
  7. consideration of other equally valid perspectives
  8. credible sources

She describes her reasons for selecting this particular list of concepts to teach to her students, and I recommend reading her blog post for the detailed and thoughtful process she used.  I’m interested in coming up with a similar list of concepts for the different group of students that I teach: grad students and seniors in engineering programs at an R1 university.

The students I’m dealing with have a much higher level of education (selected from the top 1/8th of the high school population, then further selected by sticking with an engineering major for at least 3 years) and probably a higher average intelligence.  I don’t get as much of the “magical thinking, begging of questions, and conclusions that don’t follow from their premises” that has driven her to tackle this educational problem.

I do, however, see a lot of bad writing that suffers from these problems.  I present to my students the “4Cs” of technical writing:

The 4 Cs are arranged as a cross, because clarity and correctness are often in opposition to each other, as are conciseness and completeness.  Good technical writing requires finding the right balance of these opposing ideals for the audience and purpose of any particular document.

Note that the first two points on Mylène’s list correspond to one of the major axes of the 4Cs, the clarity-correctness axis.  Her remaining points mainly address ways to assess correctness when the “right answer” is not known, though “consideration of other equally valid perspectives” gives at least a nod to completeness.

The “conciseness” goal often gives students the most trouble, but is not addressed in her list.  There are two parts to it:

  • Clear, direct sentences. Too often, students imagine that academic writing requires vague, passive constructions full of inflated diction and complex sentence structures.  Bad academic writing is used as the model.  This obfuscation (deliberate or accidental) is generally a writing problem and not a reasoning problem, though consistent use of vague phrasing may indicate an underlying lack of comprehension.
  • Putting in just what is relevant. Often students do a dump of all their notes on a subject, most of which is irrelevant to the audience and the paper that they are writing. I would have suggested to Mylène that she needs to restore “relevance” to her list, but I suspect that it is covered somewhat more specifically by her points 4 and 5: “connection to our model” and “connection to our experience/intuition”.  What may be missing is “connection to the problem we are trying to solve”.

Completeness is sometimes a problem also, mainly because students tend to write to their faculty adviser or teacher (generally the worst choice of audience, among the many possible).  Because the faculty member knows more than they do about the problem (usually), students leave out the statement of the problem, its significance, and work done on the problem by others in the past.  It is not unusual to send a student away from an advancement to candidacy exam or even a thesis defense telling them that they have a nice bit of research (proposed or done), but that Chapter 1 is missing and they need to write it before they can move forward. Faculty advisers are often too close to the research to see the absence of the background information and explanation of significance, so it is essential for thesis committee members to look for this hole.

For senior design reports and theses, I usually get the lack of problem statement and background corrected by draft 3 or 4,which is one of the benefits to having the instructor for the course which focuses on students finishing their research and the presentation of it not be the faculty member who has been supervising the research.

I suspect that defining the problem and explaining its importance is missing from the exercises that Mylène gives her students, as they are generally working on well-defined problems posed for them, and not coming up with new research topics or design problems that need to be justified. I’m sure that she can come up with ideas that are relevant to technician-trainees for practicing this skill, perhaps tying in it with her “technical reading”  training (see, for example, her post Reading Comprehension Techniques: Review).

The “seamless chain of cause and effect” should not be a huge problem for my students, but they often have a hard time getting their understanding into readable form.  Choppy writing that throws together ideas without explicitly connecting them is common.  I usually refer students to Technical Writing and Professional Communication for Non-Native Speakers of English by Thomas Huckin and Leslie Olsen (see my post Getting an A on a paper in school, for example). The chapters on focus and flow, particularly the explanation of the heuristic of connecting sentences with “old information→new information”, are among the best I’ve seen for explaining to students how to avoid choppy writing.  I think that in many cases the apparent lack of a “seamless chain of cause and effect” is not an underlying problem with reasoning, but an inability to connect ideas in writing.  (Though that perception may be a result of differences between my students and hers.)  In any case, it is probably more productive for me to concentrate on getting students to present their reasoning more clearly, rather than on trying to fix underlying problems in reasoning that may not even exist. (If the underlying reasoning is the problem, then fixing the flow in the writing should make it more apparent.)

This post ends at an awkward point:  I’ve looked over Mylène’s list and identified places where it does not quite meet the pedagogical needs of my students, but I’ve not come up with my own list.  I suspect I need 4: one for each of the 4 Cs, but I’ll leave that as an idea for a later post.

%d bloggers like this: