Gas station without pumps

2016 September 29

GRE Analytic Writing favors bullshitters

Filed under: Uncategorized — gasstationwithoutpumps @ 22:33
Tags: , , , ,

My son recently took the GRE exam to apply for grad school in computer science.  The test has changed since I took it in 1973, but it still looks a lot like the SAT exam, which has also changed since I took it in 1970.  The multiple-choice section is still primarily 9th and 10th grade material, so it is a bit surprising that only 5.5% of CS students, 11.4% of physics students, and 15.3% of math students get 170, the highest possible score, on the quantitative reasoning section. [All data in this post from https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf]

The “quantitative reasoning” questions are primarily algebra and reading simple graphs, so the banking and finance students do best with 15.5% getting 170. The scores would be more valuable for STEM grad school admissions if they included some college-level math (calculus, ODEs, combinatorics, statistics, … ), but the general GRE has always been based on an extremely low math level.

The verbal scores are perhaps less surprising, with philosophy being the only major with over 3% getting a 170 (5.1%), and with some of the education and business majors doing the worst—except for computer science, where 8% get the lowest scores (130–134), with the next worst major being accounting with 2.7% having 130–134.  I wonder how much of the difference here is due to the number of native and non-native speakers, as computer science certainly attracts a lot more foreign students than most majors.

I was most interested in looking at the “Analytical Writing” scores, since I’ve not seen much correlation between them and the quality of student writing on the grad school applications I’ve reviewed over the last decade.  I was interested in two things: the mean score and the fraction that got 4.5 or better (the fraction getting 5.5 or better is too small to be worth looking at).  Again computer science and electrical engineering stand out as having extremely low means and small fractions of students having 4.5 or better.  I have not found any analyses of the GRE scores that separate native speakers of English from non-native ones—I wonder how much of the effect we see here is due to being native speakers and how much is due to curricular differences.

Here is the table of all the broad categories in the order that ETS provided them:

Subject

Mean writing

%ile ≥4.5

Computer and Information Sciences

3.1

8.8

Electrical and Electronics

3.1

6.7

ENGINEERING

3.3

12.6

Civil

3.3

13.2

Industrial

3.3

9.8

Mechanical

3.3

12.3

PHYSICAL SCIENCES

3.4

17.3

Accounting

3.4

12.3

Banking and Finance

3.4

10.7

Natural Sciences ─ Other

3.5

14.8

Materials

3.5

19.4

BUSINESS

3.5

15.2

Other

3.5

14.7

Agriculture, Natural Res. & Conservation

3.6

18.0

Mathematical Sciences

3.6

21.0

Chemical

3.6

21.6

Early Childhood

3.6

16.0

Student Counseling and Personnel Srvcs

3.6

17.3

Business Admin and Management

3.6

17.8

Health and Medical Sciences

3.7

19.0

Chemistry

3.7

23.8

Other

3.7

23.1

Other

3.7

21.3

Arts ─ Performance and Studio

3.7

24.3

Administration

3.7

21.9

Elementary

3.7

21.3

Special

3.7

19.5

Other

3.7

23.7

LIFE SCIENCES

3.8

21.3

Biological & Biomedical Sciences

3.8

26.0

Earth, Atmospheric, and Marine Sciences

3.8

25.4

Physics and Astronomy

3.8

26.8

Economics

3.8

27.8

Sociology

3.8

28.2

EDUCATION

3.8

23.9

Curriculum and Instruction

3.8

21.4

Evaluation and Research

3.8

23.6

SOCIAL SCIENCES

3.9

29.1

Psychology

3.9

26.6

Higher

3.9

29.7

Anthropology and Archaeology

4.0

34.7

Foreign Languages and Literatures

4.0

37.2

Secondary

4.0

33.9

Political Science

4.1

42.9

ARTS AND HUMANITIES

4.1

40.8

Arts ─ History, Theory, and Criticism

4.1

38.5

History

4.1

40.4

Other

4.1

38.6

English Language and Literature

4.2

45.2

Philosophy

4.3

52.7

 OTHER

Architecture and Environmental Design

3.4

13.1

Communications and Journalism

3.7

23.3

Family and Consumer Sciences

3.7

20.7

Library and Archival Sciences

4.0

34.3

Public Administration

3.8

23.7

Religion and Theology

4.2

46.5

Social Work

3.6

16.7

The table is more interesting in sorted order (say by %ile ≥4.5 on Analytical Writing):

Subject

Mean writing

%ile ≥4.5

Electrical and Electronics

3.1

6.7

Computer and Information Sciences

3.1

8.8

Industrial

3.3

9.8

Banking and Finance

3.4

10.7

Mechanical

3.3

12.3

Accounting

3.4

12.3

ENGINEERING

3.3

12.6

Architecture and Environmental Design

3.4

13.1

Civil

3.3

13.2

Other

3.5

14.7

Natural Sciences ─ Other

3.5

14.8

BUSINESS

3.5

15.2

Early Childhood

3.6

16.0

Social Work

3.6

16.7

PHYSICAL SCIENCES

3.4

17.3

Student Counseling and Personnel Srvcs

3.6

17.3

Business Admin and Management

3.6

17.8

Agriculture, Natural Res. & Conservation

3.6

18.0

Health and Medical Sciences

3.7

19.0

Materials

3.5

19.4

Special

3.7

19.5

Family and Consumer Sciences

3.7

20.7

Mathematical Sciences

3.6

21.0

Other

3.7

21.3

Elementary

3.7

21.3

LIFE SCIENCES

3.8

21.3

Curriculum and Instruction

3.8

21.4

Chemical

3.6

21.6

Administration

3.7

21.9

Other

3.7

23.1

Communications and Journalism

3.7

23.3

Evaluation and Research

3.8

23.6

Other

3.7

23.7

Public Administration

3.8

23.7

Chemistry

3.7

23.8

EDUCATION

3.8

23.9

Arts ─ Performance and Studio

3.7

24.3

Earth, Atmospheric, and Marine Sciences

3.8

25.4

Biological & Biomedical Sciences

3.8

26.0

Psychology

3.9

26.6

Physics and Astronomy

3.8

26.8

Economics

3.8

27.8

Sociology

3.8

28.2

SOCIAL SCIENCES

3.9

29.1

Higher

3.9

29.7

Secondary

4.0

33.9

Library and Archival Sciences

4.0

34.3

Anthropology and Archaeology

4.0

34.7

Foreign Languages and Literatures

4.0

37.2

Arts ─ History, Theory, and Criticism

4.1

38.5

Other

4.1

38.6

History

4.1

40.4

ARTS AND HUMANITIES

4.1

40.8

Political Science

4.1

42.9

English Language and Literature

4.2

45.2

Religion and Theology

4.2

46.5

Philosophy

4.3

52.7

Note that all the fields that call for precise, mathematical reasoning do poorly on this test, but those which call for fuzzy, emotional arguments with no mathematical foundation do well—the test is designed to favor con men. I believe that this is partly baked into the prompts (see the pool of issue topics and, to a lesser extent, the pool of argument topics), partly the result of having the writing being done entirely without access to facts (benefitting those who BS over those who prefer reasoning supported with well-sourced facts), and partly the result of having graders who are easily swayed by con men.

I believe that most of the graders are trained in the humanities, and so are more swayed by familiar vocabulary and rhetoric.  If ETS had science and engineering professors doing the grading (which they would have a hard time getting at the low rates they pay the graders), I think that the writing scores would come out quite different.

Of course, there are curricular differences, and science and engineering faculty are mostly not paying enough attention to their students’ writing (and I can well believe that CS and EE are the worst at that). But I don’t think that even engineering students who do very, very good engineering writing will necessarily score well on the GRE analytical writing test, which seems to favor rapid writing in only one style.

I will continue to give relatively little weight to Analytical Writing GRE scores in graduate admissions. The untimed essays that the students write for the applications are much closer to the sort of writing that they will be expected to do in grad school, and so much more indicative of whether their writing skills are adequate to the job. I will continue to interpret low GRE scores as a warning sign to look more closely at the essays for signs that the students are not up to the task of writing a thesis, but high GRE writing scores are not a strong recommendation—I don’t want grad students who are good at bull-shitting.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: