Why are so many non-reproducible experiments so highly cited? Part of the problem may be a growing cultural change in biology: not everyone does experiments now. More and more, biologists are divided into experimentalists and computational biologists. (I hesitate to say theorists, because computational biologists don’t theorize about biology any more than experimental biologists.) The reason for this division is because, thanks to the growing availability of big data sets, it is possible to learn something new by analyzing already available data.
This is a positive development, but the risk is that we create a class of biologists who don’t understand the subtleties of the experiments that produced the data they work with. In the worst cases, I’ve seen talks during which a computational biologists puts up an image of a gel and says ‘I don’t know what this means, but my collaborator say this is validation.’ Not being an experimentalist is no excuse for not understanding experiments.
Physicists understand this, as demonstrated by one of my favorite Feynman stories. Here it is, courtesy of the American Institute of Physics oral history archive. Feynman is discussing traces of particle paths on photographic plates with some cyclotron experimentalists:
They told me that they had discovered an anti-proton. So, I know it’s impossible because it’s a 384 MEV cyclotron with which they say they discovered the negative proton, and that’s not enough energy. Same difficulty as with Schein. And you know, if you go to a magic show, and you have a half belief that magic is possible, then you don’t understand a number of the phenomena that you see. But if you know it’s impossible, then you keep working until you find the explanation.
The same way, I knew it was impossible, whereas they thought it might be possible because we don’t know enough. So I said, “All right, let me see.” So they got their underlings. It wasn’t Lawrence, it was somebody lower level. I don’t remember who — like me, I was low level too — and it was the same situation as Schein and Bethe but, say, another notch below. It was very amusing; it was the same situation exactly. In order to show me the plates that they had, they had to use the same kind of instruments, same dark room, same group of people looking, only this time it was smaller fry, you know.
So it was the same game, and I had learned exactly what to do from Bethe, on each track to find out what was the matter. And they had a very very similar situation. They said, “We have a whole lot of them,” and then they couldn’t find the lot when they were looking for them. They had good ones. Anyway, they had one. Finally. It focused on the one. It was the most beautiful track for a proton, for an anti-proton, you have ever seen — clean, clear, it curved the wrong way.
But I know there are no anti-protons. It was one of these great victories; it was wonderful. See, I deduced that the only way that this track curved the way it did was because it must be a proton; therefore, it must be going the other way. You know what the negative looked like? The wrong curvature. It’s obvious. I know it’s a proton, so it must be coming from the other side. It is obvious that it has to be coming backwards through the chamber.
So I say, “You must have some matter around this somewhere.” “No, none whatsoever. This chamber is just a very thin glass wall chamber with nothing around it.” “Well,” I said, “I haven’t seen the design, but it usually is necessary to hold the upper and lower plates together on such a chamber, to keep the pressure, you know.” They said, “Oh, yes, we have four thin bolts, only so and so much in diameter, that hold the two plates, you see.” So they had a plate there, and I said, “Well, right here” — I put my pencil down outside the picture –- “there must be one of those bolts.” So they got the drawing out, of the cloud chamber, and they put it over the picture, to fit, and I had my pencil right on a bolt.
What happened was, the proton hit the bolt that holds the top chamber, and scattered backward. So I demolished the anti-proton before it was published, before any rumors had got very far. You see, they’d said, “We have others,” and so on, but the whole thing just collapsed in exactly the same matter. They had been completely convinced by the one perfect track, and the others, they were hopeful-thinking, and they weren’t really good evidence. And so it was easy to demolish the other ones, easier than it was to demolish this one.
The Finch and Pea 