I’ve always thought the Reproducibility Project represented an incredibly naive approach to the scientific method. This excellent news piece in Science sums up many of the reasons why. As Richard Young says in the piece, “I am a huge fan of reproducibility. But this mechanism is not the way to test it.” Here’s why:
1) Reproducibility in science is not achieved by having a generic contract research organization replicate a canned protocol, for good reason: cutting edge experiments are often very difficult and require specialized skills to get running. Replication is instead achieved by other labs in the field who want to build on the results. Sometimes this is done using the same protocol as the original experiment, and sometimes by obtaining similar results in a different system using a different method.
2) For this reason, I don’t have much confidence that the results obtained by the Reproducibility Project will accurately reflect the state of reproducibility in science. A negative result could mean many things — and most likely it will reflect a failure of the contract lab and not an inherent problem with the result. Contrary to the claims of the projects leaders, the data produced by the Project will probably not be useful to people who are serious about estimating the scope of irreproducibility in science. At its worst, it could be extremely misleading by painting an overly negative picture of the state of science. It’s already been damaging by promoting a too-naive view of how the process of successful science actually works.
3) As the Science piece points out, there is a much better, cheaper, and scientifically sensible way to achieve better reproducibility. If many papers out there are suspect because they lack proper controls, don’t use validated reagents, fail to describe methods adequately, or rely on flawed statistics, then we don’t need to spend millions of dollars and thousands of hours of effort trying to repeat experiments. We need to make sure editors and reviewers require proper controls, reagents, statistics, and full methods descriptions.
It’s worth reading the full article, but below the fold are some salient quotes: Continue reading “The Cancer Reproducibility Project is Incredibly Naive, Probably Useless, and Potentially Damaging”
This week, Science for the People meets the authors of three big books that use stunning images to tell intriguing stories about the history of science. We’ll discuss evolution and the building of the fossil record with invertebrate palaeontologist Paul Taylor, author of A History of Life in 100 Fossils. Archivist Julie Halls shares stories of unheralded ingenuity from her book Inventions that Didn’t Change the World. We will also learn about attempts to map the world in three dimensions from independent conservator Sylvia Sumira, author of Globes: 400 Years of Exploration, Navigation, and Power.
The Finch and Pea 