A lot of academic bloggers are commenting on the recent hype about digital badges (as I did yesterday), but some of them seem to have gone on into bashing many fields of academic research as well.
For example, Aaron Barlow, the executive editor of the Academe Blog (an opinion column published by the AAUP, but not reflecting official AAUP positions), wrote a post Badges! Two: Why Data Is Never Enough. In it he misinterprets Gödel’s proof and admiringly quotes Esther Quintero:
The irony, of course, is that this notion is actually contrary to the scientific process. Being data-driven is only useful if you have a strong theory by which to navigate; anything else can leave you heading blindly toward a cliff.
Now perhaps Quintero’s quote has been taken out of context and misapplied, but as stated, it is simply incorrect and denigrates discovery science. A lot of big data science projects are not based on “strong theory”, but on the notion that unbiased collection of data can lead to a lot of hypothesis generation—hypotheses that could not have been conceived if the theory had to come before the data was collected. The notion of a strong theory to test works well in model-driven fields like physics, but not in fields that study more complicated systems, like biology.
Bioinformatics is a field that is not driven by strong theories but by data. We have found it very valuable to have large amounts of high-throughput data—looking at every base of a genome, every mRNA molecule that is expressed, every methylation of cytosines, DNA from every bacterium in an environmental sample, … . Analysis of such big data sets is often difficult, but can lead to some surprising discoveries—co-incidences that no one would have thought to look for in hypothesis-based research.
It is true that bioinformatics with big data usually leads to hypothesis generation, not confirmation, which usually requires more focused experimentation. In large part this is because the huge number of “hypotheses” that are being examined in discovery research will inevitably lead to a some looking very well-supported by the data used to discover them, when they are really just co-incidences or experimental artifacts.
Generally, bioinformaticians try to report their results with robust estimates of the false positive rate—in essence saying things like “here are 50 interesting things I found in the data, but 5–10 of them are probably wrong”. The 50 or so interesting things are usually pulled from a pool of thousands or millions of possibilities. More targeted experiments, generally using methods that will make different errors than the methods used for the original data, are then done to try to figure out which of the hypotheses are real finds and which are just coincidences.
But the big data is needed to come up with the hypotheses to test, and Quintero and Barlow’s disdain for collecting data is an insult to many scientists.
One irony of Barlow’s article is that he uses hoary drunk-looking-for-his-keys-under-the-streetlight analogy to support his argument, but that analogy applies equally well to the theory-first error he makes about using data. Limiting your search, either by what data is easy to collect or by what theories you find easy to believe, is going to severely limit what conclusions you can reach.
Of course, I’m in agreement with some of what Barlow has to say. For example,
Real education is a great deal more than reaching numerical benchmarks
The rubrics for writing exams, no matter how flexible we want to make them, always seem to drag the writers back to the five-paragraph theme. Why? Because they always focus on things that can easily be counted (paragraphs, sentences, spelling errors, sentence types, punctuation variety, supporting points, etc.). They have to, or the conclusions reached will be different for different readers.
The problem with that? Someone who writes brilliantly, but in another fashion, will always fail–even when the reader knows the writing is better than anything encompassed in the grading rubric. The reader isn’t given the freedom of deciding on his or her own, but only to make a decision based on the ‘countables’ of the rubric.
If Barlow had limited himself to talking about the low quality of the data available to educators and how poorly the data measured what we were really interested in, I’d have been posting a positive summary of his post. But his jump to bashing data-driven fields as a whole makes his post unacceptable to me.