Month: May 2015

I’ve been reading some news coverage about the now-retracted paper published in Science, which purported to show that voters’ opinions on same-sex marriage could be altered by conversations with gay canvassers. Some of the things the senior author, Donald Green, said in one article struck me as very odd, from my perspective in the natural sciences. I wonder if the culture in political science is really that different?

Here’s the first quote:

“It’s a very delicate situation when a senior scholar makes a move to look at a junior scholar’s data set,” Dr. Green said. “This is his career, and if I reach in and grab it, it may seem like I’m boxing him out.”

In case you don’t have your scorecard handy, Dr. Green is the senior author of the paper. There’s only one other author, a graduate student named Michael LaCour. LaCour did all of the actual work on the study (or at least he said he did — that’s the point of the retraction).

In physics, it’s bizarre to imagine that one of the two authors of a paper would feel any delicacy about asking to see the data the paper is based on. If a paper has many authors, then of course not every author will actually look at the data, but with only two authors, it would be extremely strange for one of them not to look. Is it really that different in political science?

Later in the same article, we find this:

Money seemed ample for the undertaking — and Dr. Green did not ask where exactly it was coming from.

“Michael said he had hundreds of thousands in grant money, and, yes, in retrospect, I could have asked about that,” Dr. Green said. “But it’s a delicate matter to ask another scholar the exact method through which they’re paying for their work.”

Delicacy again! This one is, if anything, even more incomprehensible to me. I can’t imagine having my name on a paper presenting the results of research without knowing where the funding came from. For one thing, in my field the funding source is always acknowledged in the paper.

In both cases, Green is treating this as someone else’s work that he has nothing do do with. If that were true, then asking to see the raw data would be presumptuous (although in my world asking about the funding source would not). But he’s one of only two authors on the paper — it’s (supposedly) his work too.

It seems to me that there are two possibilities:

1. The folkways of political scientists are even more different from those of natural scientists than I had realized.
2. Green is saying ridiculous things to pretend that he wasn’t grossly negligent.

I don’t know which one is right.

 

I learned via Peter Coles of this list of ways that scientists try to spin results that don’t reach the standard-but-arbitrary threshold of statistical significance. The compiler, Matthew Hankins, says

You don’t need to play the significance testing game – there are better methods, like quoting the effect size with a confidence interval – but if you do, the rules are simple: the result is either significant or it isn’t.

…

The following list is culled from peer-reviewed journal articles in which (a) the authors set themselves the threshold of 0.05 for significance, (b) failed to achieve that threshold value for p and (c) described it in such a way as to make it seem more interesting.

The list begins like this:

(barely) not statistically significant (p=0.052)
a barely detectable statistically significant difference (p=0.073)
a borderline significant trend (p=0.09)
a certain trend toward significance (p=0.08)
a clear tendency to significance (p=0.052)
a clear trend (p<0.09)
a clear, strong trend (p=0.09)
a considerable trend toward significance (p=0.069)
a decreasing trend (p=0.09)
a definite trend (p=0.08)
a distinct trend toward significance (p=0.07)

And goes on at considerable length.

Hankins doesn’t provide sources for these, so I can’t rule out the possibility that some are quoted out of context in a way that makes them sound worse than they are. Still, if you like snickering at statistical solecisms, snicker away.

I would like to note one quasi-serious point. The ones that talk about a “trend,” and especially “a trend toward significance,” are much worse than the ones that merely use language such as “marginally significant.” In the latter case, the authors are merely acknowledging that the usual threshold for “significance” (p=0.05) is arbitrary. Hankins says that, having agreed to play the significance game, you have to follow its rules, but that seems like excessive pedantry to me. The “trend” language, on the other hand, suggests either a deep misunderstanding of how statistics work or an active attempt to mislead.

Hankins:

For example, “a trend towards significance” expresses non-significance as some sort of motion towards significance, which it isn’t: there is no ‘trend’, in any direction, and nowhere for the trend to be ‘towards’.

This is exactly right. The only thing a p-value does is tell you about the probability that results like the ones you saw could have occurred by chance. Under that hypothesis, a low p-value occurred due to a chance fluctuation and will (with high probability) revert to higher values if you gather more data.

The “trend” language suggests, either deliberately or accidentally, that the results are marching toward significance and will get there if only we can gather more data. But that’s only true if the effect you’re looking for is really there, which is precisely what we don’t know yet. (If we knew that, we wouldn’t need the data.) If it’s not there, then there will be no trend; rather, you’ll get regression to more typical (higher / less “significant”) p-values.

 

Last week, Nature ran a piece under the headline Quantum physics: What is really real? I obnoxiously posted this on Facebook (because I’m too old and out of touch to be on any of the hipper social media sites):

I have now read the piece, and I can report that there’s no need for a recantation. As expected, Nature is making grandiose claims about quantum mechanics and the nature of reality that go beyond anything supported by evidence.

Nature writes pretty much the same story every couple of years. The main idea behind all of these articles is the question of whether the quantum mechanical wavefunction describes the way a system really is  or merely our knowledge of the system. In philosophy-of-science circles, these two points of view are sometimes known as the psi-ontic and psi-epistemic stances. More specifically, all three of these articles have to do with a theorem published (in one of the Nature journals) by Pusey et al. that claims to provide an experimental way of distinguishing between these possibilities. After Pusey et al. published this theoretical result, others went ahead and performed the proposed experimental tests, leading to the (claimed) conclusion that the wavefunction describes actual reality, not merely our knowledge.

You should of course be skeptical of any claim that an experimental result reveals something about the deep nature of reality. Sure enough, if you dig down just a little bit, it becomes clear that these results do no such thing. The Pusey et al. theorem proves that a certain class of psi-epistemic theories make predictions that differ from the predictions of standard quantum mechanics. The subsequent experiments confirmed the standard predictions, so they rule out that class of theories.

The problem is that ruling out a specific class of psi-epistemic theories is not the same thing as ruling out the psi-epistemic point of view as a whole. We now know that that class of theories is wrong, but that’s all we know. To make matters worse, the class of theories ruled out by these experiments, as far as I can tell, does not contain any theories that any proponents of psi-epstemicism actually believe in. The theories they tested are straw men.

In particular, the most prominent proponents of the psi-epistemic point of view are the advocates of something called quantum Bayesianism (QBism). QBism is an interpretation of quantum mechanics, as opposed to an alternative theory — that is, it makes predictions that are identical to those of standard quantum mechanics. There is, therefore, no experimental result that would distinguish QBism from psi-ontic versions of quantum mechanics.

Not all psi-epistemicists are QBists, of course, but as far as I can tell even the others never advocated for any theories in the class considered by Pusey et al. If I’m wrong about that, I’d be interested to know.