Archive for June, 2011

Error-correcting in science

Monday, June 27th, 2011

Carl Zimmer has a column in Sunday’s NY Times about the ways in which science does and does not succeed in self-correcting when incorrect results get published:

Scientists can certainly point with pride to many self-corrections, but science is not like an iPhone; it does not instantly auto-correct. As a series of controversies over the past few months have demonstrated, science fixes its mistakes more slowly, more fitfully and with more difficulty than Sagan's words would suggest. Science runs forward better than it does backward.

One of the controversies inspiring this column is the scientist who failed to replicate a controversial clairvoyance study and were unable to publish their results, because the journals only publish novel work, not straight-up replications. I offered my opinion about this a while ago: I think that this is a bad journal policy, and that in general attempts to replicate and check past work should be valued at least a bit more than they currently are.

On the other hand, it’s possible to make way, way too much of this problem. The mere fact that people aren’t constantly doing and publishing precise replications of previous work doesn’t mean that previous work isn’t checked. I don’t know much about other branches of science, but in astrophysics the usual way this happens is that, when someone wants to check a piece of work, they try to do a study that goes beyond the earlier work, trying to replicate it in passing but breaking some new ground as well. (A friend of mine pointed this out to me off-line after my previous post. I knew that this was true, but in hindsight I should have stressed its importance in that post.)

For instance, one group claims to have shown that the fine-structure constant has varied over time, then another group does an analysis, arguably more sensitive than the original, which sees no such effect. If the second group can successfully argue that their method is better, and would have found the originally-claimed effect had that effect been real, then they have in effect refuted that claim.

(I should add that in this particular example I don’t have enough expertise to say that the  second group’s analysis really is so much better as to supersede the original; this is just meant as an example of the process.)

The main problem with Zimmer’s article is a silly emphasis on the unimportant procedural matter of whether an incorrect claim in a published paper has been formally retracted in print. This is sort of a refrain of the article:

For now, the original paper has not been retracted; the results still stand.

[T]he journal has a longstanding policy of not publishing replication studies … As a result, the original study stands.

Ms. Mikovits declared that a retraction would be "premature” … Once again, the result still stands.

It’s completely understandable that a journalist would place a high value on formal, published retraction of error, but as far as the actual progress of science is concerned it’s a red herring. Incorrect results get published all the time and are rarely formally retracted. What happens instead is that the scientific community goes on, learns new things, and eventually realizes that the original result was wrong. At that point, it just dies away from lack of attention.

One example that come to mindis the Valentine’s Day monopole.  This was a detection of a magnetic monopole, which would be a Nobel-Prize-worthy discovery if verified. It was never verified. As far as I know, the original paper wasn’t formally retracted, and I’m not even sure whether it’s known what went wrong with the original experiment, but it doesn’t matter: subsequent work has shown convincingly that this experiment was in error, and nobody believes it anymore.

(By the way, this is also an illustration of the fact that getting something wrong doesn’t mean you’re a bad scientist. This was an excellent experiment done by an excellent physicist. If you’re going to push to do new and important science, you’re going to run the risk of getting things wrong.)

Of course, there is some value in a formal retraction, especially in cases of actual misconduct. And when an incorrect result is widely publicized in the general press, it would be nice if the fact that it was wrong were also widely publicized. But a retraction in the original scholarly journal would be of precisely no help with the latter problem — the media outlets that widely touted the original result would not do the same for the retraction.

Quantum silliness from the BBC

Friday, June 3rd, 2011

Since I regularly express my envy at the quality of science journalism in England, I thought I should mention that the BBC has a particularly silly piece about quantum mechanics on its web site.

The experiment described in the article looks perfectly nice, but the results definitely do not “bend” any rules of quantum physics, nor does they “pull back the veil” on quantum reality. The results found by the experiment are precisely what we “knew” they must be all along — that is, they’re exactly what is predicted by standard quantum mechanics. Of course, that doesn’t mean the experiment isn’t worth doing — it’s always nice to test our theories, and learning better techniques for manipulating quantum systems is a great thing to do. But there are precisely no philosophical implications about the nature of reality here. As John Baez said a long time ago

Newspapers tend to report the results of these fundamentals-of-QM experiments as if they were shocking and inexplicable threats to known physics. It's gotten to the point where whenever I read the introduction to one of these articles, I close my eyes and predict what the reported results will be, and I'm always right. The results are always consistent with simply taking quantum mechanics at face value.

The new result fits perfectly into this category.