Back in May, I read a blog post (Life in College Matters for Life After College) that pointed to the Gallup-Purdue Index Report 2014. I finally got the time to download the report and look at it.
The report has a rather ridiculous interpretation of copyright on its copyright page: “It is for your guidance only and is not to be copied, quoted, published, or divulged to others.” This is particularly ridiculous for a report that they are distributing for free—I think that they have a piece of boilerplate that they put on all their reports, written by lawyers who want to claim far more that copyright law really provides. They’ve got deeper pockets than me though, so the threat is effective—I won’t directly quote them in my blog, but just summarize what I see as the main points. If I mangle their message, they have only their own over-zealous lawyers to blame.
What the report is ostensibly about is whether college prepares students for an “engaging” job and a good life. They were looking for whether students were engaged in their jobs and at five measures of well-being that Gallup has used in other studies: purpose, social, financial, community, and physical. They were also looking at how attached alumni were to their alma maters (which, of course, is primarily what Purdue was interested in, as that determines how much money they can extract from alumni).
Basically, they started with the assumption that the point of college is to get a “great job” and a “great life” (a debatable point, but a widely held belief). They then tried to determine what produced these outcomes, by interviewing 30,000 graduates. Note that they did not interview those who quit or were kicked out of college—they were only considered those that college thought had succeeded. It might be interesting for them to look at the outcomes for those who dropped or failed out also, to see whether the things they think mattered in college also affected the students who left without a degree. (I suspect that the effects would be even stronger, because of the higher variance in the outcomes, but guessing about sociology is not one of my strengths.)
Their main result was that it didn’t really matter much where people went to college (other than that results were consistently worse at for-profit schools)—what mattered is what they encountered there. Having an inspiring professor who cared about them, excited them about learning, and encouraged them doubled the odds of their being engaged at work after college. Internships in which they applied their learning, multiple-term projects, and being extremely active in extracurricular activities also doubled the odds of their being engaged at work.
(They use the term “odds” rather than “probability” consistently, so I’m not sure if they mean the probability p or the odds ratio 
I think that UCSC does manage to provide some engaging faculty—most of the students I talked to in senior exit interviews had at least one faculty member who excited them about learning (but that’s fairly common—63% of graduates reported that in the Gallup-Purdue survey). I don’t know that we do as well at providing professors who show that they care about the students or providing mentors who encourage students to pursue their dreams—those are hard to provide at scale, as they rely on matching personalities as well as having enough faculty time to spend. Indeed, in the Gallup survey only about 27% of graduates felt that professors cared about them as a person and only 22% felt they had a mentor who encouraged them, so we’re not alone in finding this difficult to supply. I suspect that students doing senior theses get more mentoring than those doing group projects, but a lot depends on the student and whoever is supervising the work.
One thing that the Jack Baskin School of Engineering at UCSC is doing right—all the students in bioengineering, computer engineering, electrical engineering, and computer game design are required to do 2-quarter or 3-quarter-long capstone projects. (That alone should be a 1.8× on the odds of being engaged at work, and only 32% of students in the survey reported having that experience.) Our students do not do so well on the “extreme extracurricular activity”, though, as few engineering students feel they have time for much in the way of extracurriculars. Internships are something that UCSC could be much better at—there is a huge industry base only 40 miles away in Silicon Valley, but students are left on their own for finding internships, and not very many do.
The two strongest predictors of engagement were not really what the college did, but what students thought about the college: if they thought “the college prepared me well for life outside college” or that the college was “passionate about the long-term success of its students”. These raised the odds of engagement at work by 2.6× and 2.4× respectively. Causality is not clear here, as these attitudes may have resulted from their engagement at work, rather than being causes of it.
The report is very sloppy about confounding variables: they report that women are more engaged at work than men, and that arts, humanities, and social science majors are more engaged than science or business majors. But they don’t seem to have done anything to determine which of the two highly correlated variables is the causal one here: gender or major. Their sample is large enough that they should have been able to get at least a strong hint, despite the correlation.
One unsurprising result: those who took out large loans as students were much less likely to be thriving in all 5 areas of well-being than those who took out small loans or no loans. Since financial well-being is one of the areas, and large loans make it difficult to achieve financial well-being, this is hardly a surprising result. It would have been more interesting if they had reported differences in just the other four areas—did the large loans have any effects other than the obvious financial one? They’ve got the data, but they didn’t do the analysis (or they’re not sharing it in the free report, which seems more likely—I’m sure they’ll share it for a hefty consulting fee).
Given that there was almost no difference in well-being based on public vs. private or selective vs. non-selective colleges, the big negative correlation of large loans with well-being sounds like a strong argument to go to a college you can afford, rather than taking out large loans. (Again, the report did not attempt to look at confounding variables for the for-profit schools—how much of their poor performance was due to the large loans they encouraged their students to take out?)
The results for alumni attachment were much stronger than for well-being or job engagement, probably because the background level of alumni attachment was fairly low—only about 18% of college graduates were emotionally attached to their colleges by the criteria used in the poll. The biggest drivers for emotional attachment were whether they felt the college had prepared them well and whether they felt it was passionate about the long term success of the students. Again, I question the causality here—it seems likely that those who are emotionally attached are more likely to hold these beliefs, irrespective of what the college actually did.
I’m also confused by their “odds” again, where they report 48% of a group being emotionally attached as 6.1× the odds of another group where 2% are emotionally attached. I don’t see how they computing their “odds”—it is a very odd computation indeed! Update: perhaps the odds they mean are 
Although the basic conclusion of the study seem reasonable to me (that what happens to you in college is more important than where you go to college, and that large loans make you miserable), the survey seems rather sloppily done, confusing correlation with causality, not attempting to disentangle confounding variables, and doing some sort of arithmetic that seems completely inconsistent so that the “odds” they report are incomprehensible. Also, they asked few questions and every question they asked seemed to have about the same effect on the odds, so I don’t know whether the survey was actually measuring anything (no negative controls).
I’d hesitate to invest money or make academic planning decisions based on this report. I think that Purdue wasted a lot of money on a load of crap (unless they got a private report with a lot better data and analysis).
Gas station without pumps 