Dan Mangan is no stranger to using scientific references and metaphors to communicate larger meanings about life. His lovely track About as helpful as you can be without being any help at all uses a quote from science illustrator Charlie Harper to beautiful effect (and was a previous Song of the Week). The song that follows it directly on the album (2011’s Oh Fortune) is called How Darwinian and pivots on the lines I should know better by now/there’s only so much to go around.
The A-side of his new 7″ Radicals, released today, works in the other direction. It’s an observation about life that actually has special significance for science. The song is heavy on guitars and crunchier and louder than usual for the normally melodic and almost symphonic singer-songwriter. After a few quiet verses, the song’s title begins as a quiet refrain and then swells into a noisy passionate plea: We Want To Be Pleasantly Surprised, Not Expectedly Let Down.
Yes, that is correct, expectedly let down. I really like how the phrase captures the ongoing tension in science between wanting and hoping for significant results (that’s what most publishable and what brings recognition for doing new and exciting work) and knowing deep down that the most rigorous approach is to always assume the null result: to be expectedly let down.
Waiting to be pleasantly surprised is part of what leads to publication bias: only publishing the surprising and positive results, while ignoring studies that show no effect (or even negative effects) for interventions or drug treatments. It’s an especially troubling issue once those positive results are then collected into the meta-analyses. And yet a January 2012 study in BMJ found that only 32% of the meta-analyses they looked at discussed or investigated possible issues with publication bias.
It also contributes to the ongoing problem discouraging the publication of studies that replicate and check previous research. There is rarely anything pleasantly surprising about replication studies, despite how important they are. Ed Yong wrote an excellent examination of the replication problem in psychology for Nature that included the following:
One reason for the excess in positive results for psychology is an emphasis on “slightly freak-show-ish” results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. “High-impact journals often regard psychology as a sort of parlour-trick area,” he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. “When we review papers, we’re often making authors prove that their findings are novel or interesting,” he says. “We’re not often making them prove that their findings are true.”
As Yong goes on to explain, there are many factors that contribute to these issues including problems with scientific publishing and funding models. The simplest explanation, though, is that scientists are people, and like Mangan notes we all want to be pleasantly surprised, not expectedly let down. It’s not a bad thing to keep in mind when reading new and exciting results or working to dig through and publish new research.
If you want to hear from more you can also check out What Happens Next, a new documentary about Dan Mangan, getting its Canadian national debut on CBC television October 20 and, for music fans everywhere, available streamed that afternoon from the CBCMusic site.
Ahmed, I., Sutton, A. J., & Riley, R.D. (2012). Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: A database survey. BMJ, 344.