Scientific Method in Decline?

Jonah Leher in The New Yorker about the slipperiness of the scientific method:

“The Truth Wears Off: Is There Something Wrong With The Scientific Method?”

The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.

The piece, dressed up in a bit of mysticism, is essentially a description of some well known (but too rarely acknowledged) biases in science: Unconscious selection of favorable data; the tendency to publish only positive results, and the effects of randomness.

It’s an important point, one that was first taught to me in a physics class when we learned about Millikan’s efforts to measure the charge of the electron – a classic case of selection bias.

Lehrer quotes some scientists in his article who suggest that this is science’s dirty secret, one that researchers are ashamed of. But why should we be ashamed of this?

Science is a human enterprise. Mistakes get made. Biases exist. And yet, amazingly, science still works, which is really the only justification for its existence. Science is still the most powerful approach for manipulating and predicting the physical world, period. No other philosophy comes close. With all of its flaws, using science we still manage to build nuclear reactors, create glow-in-the-dark fish, find cancers using NMR, build superconducting materials, send robots to Mars, and track the spread of new flu viruses.

Given this track record, scientists have no need to be ashamed that, even in the absence of fraud, science is imperfect. Feynman puts it more eloquently:

The scientist has a lot of experience with ignorance and doubt and uncertainty, and this experience is of very great importance, I think. When a scientist doesn’t know the answer to a problem, he is ignorant. When he has a hunch as to what the result is, he is uncertain. And when he is pretty damn sure of what the result is going to be, he is still in some doubt. We have found it of paramount importance that in order to progress, we must recognize our ignorance and leave room for doubt. Scientific knowledge is a body of statements of varying degrees of certainty — some most unsure, some nearly sure, but none absolutely certain. Now, we scientists are used to this, and we take it for granted that it is perfectly consistent to be unsure, that it is possible to live and not know. But I don’t know whether everyone realizes this is true. Our freedom to doubt was born out of a struggle against authority in the early days of science. It was a very deep and strong struggle: permit us to question — to doubt — to not be sure. I think that it is important that we do not forget this struggle and thus perhaps lose what we have gained.

“The Value of Science,” address to the National Academy of Sciences (Autumn 1955)

Lehrer ends his piece with a misleading statement:

We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

The uncertainty of science doesn’t mean that we’re simply left with an arbitrary choice of what to believe. You still must follow the evidence, which, as Feynman points out, needs to be weighed because “scientific knowledge is a body of statements of varying degrees of certainty.” (This is something sorely neglected in most science reporting that portrays each new paper as a sensational breakthough.)

A recent result, especially one supported by just a handful of studies (such as the effectiveness of a drug), gets less weight than something that has been accumulating multiple lines of evidence for decades (the mass of an electron, the common ancestry of humans and chimps). Yes, in some rapidly changing fields, some results that are later overturned end up in textbooks. Other results stand the test of time (the structure of amino acids, the genetic code).

We shouldn’t be ashamed of varying degrees of uncertainty in science. I’ll leave xkcd with the last word:

Author: Mike White

Genomes, Books, and Science Fiction

7 thoughts on “Scientific Method in Decline?”

  1. Strange how we perpetuate the ‘scientific method’ fallacy – that science is a step-by-step process that results in a truth. It doesn’t work that way, has never worked that way, and won’t work that way in the future. Rather, it’s a methodology; a set of values by which we assess a method to gain confidence in its having produced useful observations. We then use that to gain confidence in ideas in how those observations might be related to one another.

    Lehrer’s article relies on the subtle but mistaken notion amongst the public (and, in my experience, a lot of scientists) that science is a series of ‘Eureka!’ moments that provides proof of hypotheses. Science journalism thrives on this image by its very nature. Instead, science is a steady gradualism, where human biases are worn away through shifts in social paradigms. That means it can take a long time for a theory to evolve as those biases shift and the good ol’ ‘filing draw’ is opened and revised.

    Sorry about the rant – every time I see ‘scientific method’ I swear a genetically engineered fairy somewhere dies. 🙂 Great article.

    1. The focus on “truth” has always been troubling. Science is about predicting. We strive to build models that are more and more predictive. At some level the “truth” is irrelevant. For example, the concepts of quantum mechanics are non-intuitive, weird, and hard to believe. They may not be true, but to startling degrees of accuracy, the universe behaves as if they are true. Practically speaking, that is what matters. The scientific method is unrivaled as a way of building progressively more accurate and predictive models.

      What Lehrer sees as “decline” is simply the process of the models getting more accurate. It is also a bit naive to claim that there is a problem with “replicability”. The studies highlighted in his article almost certainly adapted and improved methods over time (perhaps including less bias), leading to more accurate results.

      1. He picked examples to show his pet point – some hot new effect that ends up being bogus. There are many other examples of how new but rough ideas develop over time in a different way, and successfully become core, well-established results.

    2. Rant away!

      Scientific gradualism doesn’t sell newspapers, so it’s unfortunate but not surprising that there is so much misrepresentation of how science works in the media.

      Although it’s not always obvious, whenever I say or write “the scientific method” it is usually with the same ironic attitude I have when I say “the central dogma”, although

      1. Historically, it is easy to get the impression of EUREKA moments. As time passes our resolution gets worse and we only see the peaks, not the gradual slopes leading up to them. If you chop a century up into 25 year chunks, progress can look like leaps. If you chop it up into 1 year chunks, the progress in pretty gradual. This alone can create the illusion that progress has stalled. Partner it with the increasing public awareness of those individual steps due to newer forms of media, and the problem gets worse.

        Its a bit like how historians will note that no one in Rome ever knew that the Roman Empire fell, even though it is pretty obvious from our perspective.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s