Matt Yglesias uses the Reinhart-Rogoff economics fiasco as a springboard to talk about When to Take Empirical Evidence Seriously:
So when you look at a purported empirical finding you need to ask not only how strong is the evidence, but what’s a reasonable prior assessment before looking at the new empirical data. A correlation is never sufficient to establish a causal relationship, but it might be good evidence for the existence of one or else it might not. That depends, in part, on how strong your theory is.
Basically, he’s advocating the position, often attributed to Sir Arthur Eddington, that you should never believe an experiment until it’s been confirmed by a theory. That position is usually thought of as at least partly a joke, but, as I’ve mentioned before, it’s really just an expression of sound Bayesian reasoning. New evidence causes you to update your prior beliefs. Your new level of belief in any given proposition is determined by both your prior belief and the evidence. It’s not just OK to take your prior beliefs into account; it’s mandatory.
I’m glad to see that Yglesias understands this. He attributes his understanding in part to Nate Silver’s book, so I guess Silver is also sound on this subject. (I haven’t read Silver’s book, so I can’t confirm this, but it doesn’t surprise me.)
Ted, should we be applying a litmus tests like this to the Planck anomalies? http://spaceinimages.esa.int/Images/2013/03/Planck_enhanced_anomalies
Of course! We should interpret all results using Bayesian reasoning (otherwise known as “correct reasoning”). In particular, we definitely should include the low prior probability of any of the exotic cosmological models required to explain the anomalies when deciding what we think of them.