AAS journals going electronic-only

June 2nd, 2014 by Ted Bunn

The American Astronomical Society just announced that they’ll stop producing paper copies of their journals. The Society publishes some of the leading journals in astronomy and astrophysics —  the several different flavors of Astrophysical  Journal (main journal, letters, supplement series) and the Astronomical Journal — so they’re not exactly a bit player.

The days when people actually looked things up in paper copies of journals are long gone, so this change makes a lot of sense to me. One good consequence: if there’s still a stigma associated with online-only journals (i.e., the notion that something online-only can’t be a “real” journal), the conversion of high-profile journals to online-only has to help combat it.

I’ve heard people say that paper copies are the best way to create a permanent archive of the scholarly record — maybe in 100 years, nobody will be able to read all the electronic copies that are out there. Maybe that’s right, but I doubt it. It’s true that old digital information eventually becomes practically unreadable — I threw out a bunch of floppy disks not too long ago, for instance — but the reason I lost that information is  because it’s material that I never tried to preserve in a useful form. Whatever future changes in data storage technology come along, I bet that we can update our electronic scholarly journals accordingly.

The AAS has offered electronic-only subscriptions for a while now, at about 60% the cost of a full (paper+electronic) subscription. The price is not bad compared to other journals, and the profits go to benefit the Society, which I think is a good thing to do. Still, it’s hard for me to see what value the journal is supplying that justifies the costs. The most important thing a journal does is provide peer review, but the actual peer reviewers do it for free.

Replication

May 21st, 2014 by Ted Bunn

I heard (via Sean Carroll) about this piece in Science headlined “Replication Effort Provokes Praise—And ‘Bullying’ Charges.” It’s about efforts to replicate published results in certain areas of psychology.

In general, I think that publication bias and dodgy statistics are real problems in science, so I’d bet that lots of results, particularly those that are called “significant” because they clear the ridiculously weak threshold of 5%, are wrong. Apparently lots of people, particularly in certain parts of psychology, are worried about this. I think it’s great for people to try to replicate past results and find out. (Medical researchers are on the case too, particularly John Ioannidis, who claims that “most published research findings are false.”)

The most striking part of the Science piece is the “bullying” claim. It seems ridiculous on its face for a scientist to complain about other people trying to replicate their results. Isn’t that what science is all about? But I can understand in part what they’re worrying about. You can easily imagine someone trying to replicate your work, doing something wrong (or perhaps just different from what you did), and then publicly shaming you because your results couldn’t be replicated. For instance,

Schnall [the original researcher] contends that Donnellan’s effort [to replicate Schnall's results] was flawed by a “ceiling effect” that, essentially, discounted subjects’ most severe moral sentiments. “We tried a number of strategies to deal with her ceiling effect concern,” Donnellan counters, “but it did not change the conclusions.” Donnellan and his supporters say that Schnall simply tested too few people to avoid a false positive result. (A colleague of Schnall’s, Oliver Genschow, a psychologist at Ghent University in Belgium, told Science in an e-mail that he has successfully replicated Schnall’s study and plans to publish it.)

The solution, of course, is for Donnellan to describe clearly what he did and how it differs from Schnall’s work. The readers can then decide (using Bayesian reasoning, or as I like to call it, “reasoning”) whether those differences matter and hence how much to discount the original work.

The piece quotes Daniel Kahneman giving an utterly sane point of view:

To reduce professional damage, Kahneman calls for a “replication etiquette,” which he describes in a commentary published with the replications in Social Psychology. For example, he says, “the original authors of papers should be actively involved in replication efforts” and “a demonstrable good-faith effort to achieve the collaboration of the original authors should be a requirement for publishing replications.”

If the two groups work in good faith to do a good replication, it’ll make the final results easier to interpret. If the original group refuses to work with people who are trying to replicate their results, well, everyone is entitled to take that into account when performing (Bayesian) reasoning about whether to believe the original results.

 

Dust or not?

May 17th, 2014 by Ted Bunn

Following the recent rumor, some more useful information has been coming out about questions that some people are raising about whether the BICEP experiment really has seen signs of gravitational waves from inflation in the polarization of the cosmic microwave background radiation. The Washington Post has by far the best news article I’ve seen on the subject: it actually quotes people on the record, rather than repeating vague anonymous speculation.

The original rumor seems to be generally true, in the sense that it accurately described some criticisms that cosmologists were making about the BICEP analysis. The rumor does seem to have exaggerated and/or oversimplified things, and of course whether those criticisms are valid or not remains to be seen.

The best place I know of to get the technical details is this talk by Raphael Flauger. (Unfortunately, the video doesn’t show the slides as he’s talking, so if you want to follow it, download the slides first and try to follow along as he talks.) He argues that the dust models used by the BICEP team are inaccurate for a few reasons, mostly having to do with problems associated with the reason in the original rumor: the BICEP team appears to have used an image in a slide from a talk for part of their model, and they seem (he claims) to have misinterpreted what was in that slide. In addition (he claims), there are other errors associated with digitizing the image rather than using the real data (which BICEP doesn’t have access to). Flauger further claims that when you use a different (better?) dust model, the possible contribution of dust to what BICEP saw gets significantly larger, possibly large enough to explain their signal.

If BICEP has offered a detailed, technical rebuttal to this criticism, I haven’t seen it yet.

My personal assessment, based on obviously incomplete information: Flauger’s arguments seem to me to need serious consideration. BICEP needs to supply a detailed response. As of now, I don’t know whether he’s right or not, but my view has changed somewhat since the original rumor. The available information now does seem to me sufficient to substantially lower my own estimate of the probability that BICEP has seen primordial gravitational waves. I was fairly skeptical all along, but now I’m more skeptical. If you must know, I’d put the probability significantly below 50%.

 

Rumors

May 13th, 2014 by Ted Bunn

The story so far:

  • BICEP2 announces a detection of B modes in the cosmic microwave background (CMB) polarization on large angular scales. If this result is correct, it’s very strong evidence that inflation happened in the very early Universe and is a really big deal. But that “if” part is important: we shouldn’t place too much confidence in this result until it’s independently confirmed.
  • In the blog Résonaances, Adam Falkowski publishes a rumor that an error had been found in the BICEP2 analysis.
  • Various science news outlets pick up the story (particularly this one and this one). They ask the BICEP2 people what they think, and the BICEP2 people vehemently stand by their results.

So what are we supposed to think?

The key claim in the Résonaances post is that the BICEP2 team made an error in modeling Galactic dust. This is potentially important, as an important part of the analysis is testing to make sure that the signal seen in the data is due to the CMB and not to boring, nearby sources such as dust.

Résonaances:

To estimate polarized emission from the galactic dust, BICEP digitized an unpublished 353 GHz map shown by the Planck collaboration at a conference.  However, it seems they misinterpreted the Planck results: that map shows the polarization fraction for all foregrounds, not for the galactic dust only (see the “not CIB subtracted” caveat in the slide). Once you correct for that and rescale the Planck results appropriately, some experts claim that the polarized galactic dust emission can account for most of the BICEP signal.

This looks to me like it might be at least partially true.

There is not a definitive map of polarized Galactic dust emission, so the BICEP team had to cobble together models of dust from different sources. They did so in several different ways: section 9.1 of their paper lists six different dust models. One of these models is based on data from the Planck satellite. It appears that they created the model using a digitized image of a slide from a talk by the Planck people, because the relevant data hadn’t been released in any other form. (Footnote 33 of the paper is the evidence for this last statement, in case you want to check it out.) The evidence does seem to me to support Falkowski’s statement: the image in question explicitly says “not CIB subtracted,” meaning that the data that went into that image includes other stuff besides what the BICEP team wanted. This does seem like a flaw in the construction of this particular model.

But it seems to me that Falkowski greatly overstates the significance of this flaw. For one thing, this is just one of six dust models used in the analysis. It was regarded as in some sense the “best” of them, but the more important point is that the other models yielded similar results. The BICEP team’s claim, as I understand it, is that the entire analysis, taking into account all the models, makes it implausible that dust is the source of the signal. Even if you throw out this model, I don’t think that that claim is significantly weakened.

As I’ve said before, I don’t think that the BICEP team has made a thoroughly convincing case that what they’ve seen can’t be foreground contamination. I think we need more data to answer that question. But even if Falkowski has correctly identified an error in the analysis, I don’t think that it changes the level of doubt all that much.

In the past, I’ve found Résonaances to be a good source of information, but I can’t say I’m thrilled with the way Falkowski handled this.

Animal magnetism

May 8th, 2014 by Ted Bunn

Interesting piece in Nature:

Interference from electronics and AM radio signals can disrupt the internal magnetic compasses of migratory birds, researchers report today in Nature1. The work raises the possibility that cities have significant effects on bird migration patterns.

That’s from a news item. The actual paper is here (possibly paywalled).

There’s strong evidence that some animals (birds, sharks, and bacteria, among others) respond to the Earth’s magnetic field, but the mechanisms by which they sense the field are still quite uncertain in many cases. Physics Today did a nice overview of this about six years ago. I think it’s fascinating that such a simple question remains unsolved.

The new result appears to be that robins do poorly at orienting themselves to Earth’s magnetic field when they’re in an environment with human-generated radio frequency electromagnetic fields, but when you shield them from those fields, they get better. Here’s a figure from the paper:

The dots around the two blue circles show the way the birds oriented themselves when they were inside of a grounded metal shield. The two red circles show what happened when the shield was not grounded. In each case, the arrow at the center is the average of all the directions, and the dashed circle shows the threshold for a significant deviation from random (5% significance, I believe). The graphs below are the field strengths with and without grounding, as functions of frequency.

These results barely exceed the 5% threshold, but the paper gives results of other similar experiments that show the same pattern.

Although the experiment seems to have been well-designed, I have to admit I’m skeptical, for a familiar reason: you should never believe an experiment until it’s been confirmed by a theory. I find it hard to imagine a mechanism for birds to sense magnetic fields that would be disrupted by the weak, low-frequency fields involved here.

The authors acknowledge this:

Any report of an effect of low-frequency electromagnetic fields on a biological system should be subjected to particular scrutiny for at least three reasons. First, such claims in the past have often proved difficult to reproduce. Second, animal studies are commonly used to evaluate human health risks and have contributed to guidelines for human exposures. Third, “seemingly implausible effects require stronger proof”.

Here’s what they say about mechanisms:

The biophysical mechanism that would allow such extraordinarily weak, broadband electromagnetic noise to affect a biological system is far from clear. The energies involved are tiny compared to the thermal energy, kBT, but the effects might be explained if hyperfine interactions in light-induced radical pairs or large clusters of iron-containing particles are involved.

The “tiny compared to the thermal energy” part is the really puzzling thing. If these electromagnetic fields are having an effect inside the system, they must do it by something absorbing photons (since that’s all an electromagnetic field is). But the energy of a photon at these frequencies is tiny in comparison to the thermal energy sloshing around a biological system anyway, so how could there be an effect?

The first of these two mechanisms seems to refer to one of the proposed mechanisms for magnetoreception in birds, which Wikipedia describes as follows:

According to one model, cryptochrome, when exposed to blue light, becomes activated to form a pair of two radicals (molecules with a single unpaired electron) where the spins of the two unpaired electrons are correlated. The surrounding magnetic field affects the kind of this correlation (parallel or anti-parallel), and this in turn affects the length of time cryptochrome stays in its activated state.

I think that this mechanism involves tiny energy differences between quantum states of a system, depending on how the electron spins are oriented. If the energy differences are tiny enough, then I guess low-frequency EM fields could disrupt the effect. But if the energy differences are that small, then I don’t understand why normal thermal fluctuations don’t mess it up all the time. I guess that for this mechanism to work, the electrons have to be shielded from thermal fluctuations, but external EM fields could still sneak in and mess them up. I guess that might be possible, but I’d want to see the details.

I completely don’t get what the authors are talking about when they refer to “large clusters of iron-containing particles”. I can’t see any conceivable way such particles could be affected by weak oscillating fields of the sort described here.

I have no idea whether you should believe this result or not. I hope that others will attempt to replicate it. If it’s real, it’s got to be a big clue about the interesting puzzle of how birds feel magnetic fields.

What are you going to do with that?

April 21st, 2014 by Ted Bunn

Check out the new blog by the noted cosmologist Lloyd Knox. Once per month, he’ll interview  someone with a degree in physics about their career choices and experiences. The first interview, with a medical device physicist, is up.

I think this is a great idea. People don’t have a clear sense of the variety of things that you can do with a physics education. I frequently have parents of prospective students ask me, in a worried tone, whether their children will be able to get jobs after majoring in physics. The answer, of course, is yes: people with physics degrees do very well in the job market. The American Institute of Physics has lots of statistics on this, such as these:


 

(both from this document). But the statistics don’t give a good sense of the variety of things people do with their physics degrees, so Lloyd’s plan to let people tell their stories sounds great.

 

Important If True

March 18th, 2014 by Ted Bunn

Some 19th-century skeptic is supposed to have said that all churches should be required to bear the inscription “Important If True” above their doors. (Google seems to credit Alexander William Kinglake, whoever he was.) That’s pretty much what I think about the big announcement yesterday of the measurements of  cosmic microwave background polarization by BICEP2.

This result has gotten a lot of news coverage, which is fair enough: if it holds up, it’s a very big deal. But personally, I’m still laying quite a bit of stress on the “if true” part of “important if true.” I don’t mean this as any sort of criticism of the people behind the experiment: they’ve accomplished an amazing feat. But this is an incredibly difficult observation, and at this point I can’t manage to regard the results as more than an extremely exciting suggestion of something that might turn out to be true.

Incidentally, I have to point out the most striking quotation I saw in any of the news reports. My old friend Max Tegmark is quoted in the New York Times as saying

I think that if this stays true, it will go down as one of the greatest discoveries in the history of science.

A big thumbs-up to Max for the first clause: lots of people who should know better are leaving that out (unless nefarious editors are to blame). But the main clause of the sentence is frankly ludicrous. It’s natural (and even endearing) that Max is excited about this result, but this isn’t natural selection, or quantum mechanics, or conservation of energy, or the existence of atoms, to name just a few of the “greatest discoveries in the history of science.”

I’ll say a bit about why this result is important, then a bit about why I’m still skeptical. Finally, since the only way to think coherently about any of this stuff is with Bayesian reasoning, I’ll say something about that.

Important

I’m not going to try to explain the science in detail right now. (Other people have.)  But briefly, it goes like this. For about 30 years now, cosmologists have suspected that the Universe went through a brief period known as “inflation” at very early times, perhaps as early as 10-35 seconds after the Big Bang.. During inflation, the Universe expanded extremely — almost inconceivably — rapidly. According to the theory, many of the most important properties of the Universe as it exists today originate during inflation.

Quite a bit of indirect evidence supporting the idea of inflation has accumulated over the years. It’s the best theory anyone has come up with for the early Universe. But we’re still far from certain that inflation actually happened.

For quite a while now, people have known about a potentially testable prediction of inflation. During the inflationary period, there should have been gravitational waves (ripples in spacetime) flying around. Those gravitational waves should leave an imprint that can still be seen much later, specifically in observations of  the cosmic microwave background radiation (the oldest light in the Universe). To be specific, the polarization of this radiation (i.e., the orientation of the electromagnetic waves we observe) should vary across the sky in a way that has a particular sort of geometric pattern. In the jargon of the field, we should expect to see B-mode microwave background polarization on large angular scales.

That’s what BICEP2 appears to have observed.

If this is correct, it’s a much more direct confirmation of inflation than anything we’ve seen before. It’s very hard to think of any alternative scenario that would produce the same pattern as inflation, so if this pattern is really seen, then it’s very strong evidence in favor of inflation. (The standard metaphor here is “smoking gun.”)

If True

(Let me repeat that I don’t mean the following as any sort of criticism of the BICEP2 team. I don’t think they’ve done anything wrong; I just think that these experiments are hard! It’s pretty much inevitable that the first detection of something like this would leave room for doubt. It’s very possible that these doubts will turn out to be unfounded.)

One big worry in this field is foreground contamination. We look at the microwave background through a haze of nearby stuff, mostly stuff in our own Galaxy. An essential part of this business is to distinguish the primordial radiation from these local contaminants. One of the best ways to do this is to observe the radiation at multiple frequencies. The microwave background signal has a known spectrum — that is, the relative amplitudes at different frequencies are fixed — which is different from the spectra of various contaminants.

The (main) data set used to derive the new results was taken at one frequency, which doesn’t allow for this sort of spectral discrimination. The authors of the paper do use additional data at other frequencies, but I’ll be much happier once those data get stronger.

I should say that the authors do give several lines of argument suggesting that foregrounds aren’t the main source of the signal they see, and at least some other people I respect don’t seem as worried about foregrounds as I am, so maybe I’m wrong to be worried about this. We will get more foreground information soon, e.g., from the Planck satellite, so time will tell.

There are other hints of odd things in the data, which may not mean anything. Matt Strassler lays out a couple. One more thing someone pointed out (can’t immediately track down who): the E-type polarization significantly exceeds predictions in precisely the region (l=50 or so) where the B signal is most significant. The E signal is larger / easier to measure than the  B signal. Is this a hint of something wrong?

I’m actually more worried about the problem of “unknown unknowns.” The team has done an excellent job of testing for a wide variety of systematic errors and biases, but I worry that there’s something they (and we) haven’t thought of yet. That seems unfair: how can I ding them for something that nobody’s even thought of? But nonetheless I worry about it.

The solution to that last problem is for another experiment to confirm the results using different equipment and analysis techniques. That’ll happen eventually, so once again, time will tell.

(Digression: I always thought it odd that people mocked Donald Rumsfeld for talking about “unknown unknowns.” I think it was the smartest thing he ever said.)

 What Bayes has to say

This section is probably mostly for Allen Downey, but if you’re not Allen, you’re welcome to read it anyway.

My campaign to rename “Bayesian reasoning” with the more accurate label “correct reasoning” hasn’t gotten off the ground for some reason, but the fact remains that Bayesian probabilities are the only coherent way to think about situations like this (and practically everything else!) where we don’t have enough information to be 100% sure.

This paper is definitely evidence in favor of inflation.

P1 = P(BICEP2 observes what it did | inflation happened)

is significantly greater than

P2 = P(BICEP2 observes what it did | inflation didn’t happen)

so your estimate of the probability that inflation happened should go up based on this new information.

The question is how much it should go up. I’m not going to try to be quantitative here, but I do think there are a couple of observations worth making.

First, all the stuff in the previous section goes into one’s assessment of P2. Without the possibility of foregrounds or undiagnosed systematic errors messing things up, the P2 would be extremely tiny. Your assessment of how likely those problems are is what determines your value of P2 and hence the strength of the evidence.

But there’s more to it than just that. “Inflation” is not just a theory; it’s a family of theories. In particular, it’s possible for inflation to have happened at different energy scales (essentially, different times after the Big Bang), which leads to different predictions for the B-mode amplitude. The amplitude BICEP2 detected is very close to the upper limit on what would have been possible, based on previous information. In fact, in the simplest models, the amplitude BICEP2 sees is inconsistent with previous data; to make everything fit, you have to go to slightly more complicated models. (For the cognoscenti, I’m saying that you seem to need some running of the spectral index to make BICEP2′s amplitude consistent with TT observations.) That makes P1 effectively smaller, reducing the strength of the evidence for inflation.

What I’m saying here is that the tension between BICEP2 and other sources of information makes it more likely that there’s something wrong.

Formally, instead of talking about a single number P1, you should talk about

P1(r,…) = P(BICEP2 observes what it did | r, …).

Here r is the amplitude of the signal produced in inflation and … refer to the additional parameters introduced by the fact that you have to make the model more complicated.

Then the probability that shows up in a Bayesian evidence calculation is the integral of P1(r,…) times a prior probability on the parameters. The thing is that the values of r where P1(r,…) is large are precisely those that have low prior probability (because they’re disfavored by previous data). Also, the more complicated models (with those extra “…” parameters) are in my opinion less likely a priori than simple models of inflation.

So I claim, when properly integrated over the priors, P1 isn’t as large as you might have thought, and so the evidence for inflation isn’t as high as it might seem.

Of course, it’s hard to be quantitative about this. I could make up some numbers, but they’d just be illustrative, so I don’t think they’d add much to the argument.

 

Correlation is correlated with causation

February 21st, 2014 by Ted Bunn

My old friend Allen Downey has a thoroughly sensible post about correlation and causation.

It is true that correlation doesn’t imply causation in the mathematical sense of “imply;” that is, finding a correlation between A and B does not prove that A causes B. However, it does provide evidence that A causes B. It also provides evidence that B causes A, and if there is a hypothetical C that might cause A and B, the correlation is evidence for that hypothesis, too.

If you don’t understand what he means by this, or if you don’t believe it, read the whole thing.

I do think he’s guilty of a bit of rhetorical excess when he says that “the usual mantra, ‘Correlation does not imply causation,’ is true only in a trivial sense.” I think that the mantra means something quite specific and valid, namely something like “correlation does not provide evidence for causation that’s as strong as you seem to think.” One often sees descriptions of measured correlations that imply that the correlation supports the hypothesis of causation to the exclusion of all others, and when that’s wrong it’s convenient to have a compact way of saying so.

But that’s a small quibble, which gets even smaller if I include the next phrase in that quote from Allen,

The point I was trying to make (and will elaborate here) is that the usual mantra, “Correlation does not imply causation,” is true only in a trivial sense, so we need to think about it more carefully [emphasis added].

I couldn’t agree more, and everything Allen goes on to say after this is 100% correct.

 

 

Calamity?

February 20th, 2014 by Ted Bunn

Sean Carroll has a post about a report by Laurence Yaffe on the state of funding for theoretical particle physics in the US, or more specifically in the Department of Energy, which historically has been the big funding source in this field. They both use the word “calamity” to describe the situation, but as far as I can tell the evidence cited doesn’t support that conclusion.

(For what it’s worth, I see that Peter Woit views the situation the same way I do. For some people, that will increase my credibility; for others it’ll decrease it.)

Yaffe’s abstract:

A summary is presented of data obtained from a grass-roots effort to understand the effects of the FY13 and FY14 comparative review cycles on the DOE-funded portion of the US high energy theory community and, in particular, on graduate students and postdoctoral researchers who are beginning their careers. For a sample comprised of nearly all of the larger groups undergoing comparative review, total funding declined by an average of 23%, with numerous major groups receiving reductions in the 30–55% range. Funding available for postdoc or graduate student support declined over 30%, with many reductions in the 40–65% range. The total number of postdoc positions in this large sample of theory groups is declining by over 40%. The impacts on young researchers raise grave concerns regarding continued U.S. leadership in high energy theory.

Carroll:

Obviously this is unsustainable, unless as a society we make the decision that particle physics just isn’t worth doing. But hopefully things can be rectified at least a bit, to restore some of that money.

But the striking thing about Yaffe’s report is that it says precisely nothing about the total level of DOE funding in this field. What it says instead is that existing large particle theory research groups have had their funding cut. Is that because the funding is going away or because it’s going to other, smaller groups? As far as I can tell, Yaffe and Carroll assume the former, but they provide no evidence for it.

DOE did recently change their procedure for evaluating grants in various ways. According to Yaffe,

Three years ago, the Office of High Energy Physics (OHEP) within the Department of Energy made significant changes in how university-based research proposals are reviewed, switching to a comparative review process and synchronizing all new grants. Overt goals included decreasing the effects of historical inertia on funding levels for different groups, ensuring a level playing field, and moving to a start date for grants mid-way through the federal fiscal year by which time, it was hoped, Congressional funding decisions would normally be known.

The first two of those goals, it seems to me, pretty much say that the DOE is aiming to redistribute funds away from previously-large research groups (those that have benefitted from “historical inertia”). Yaffe gathered data on large research groups and showed they got smaller, precisely as you’d expect. So it’s not at all clear to me that the alarmist response to this information is warranted.

What we really need to know is simply how much funding high energy theory is getting in comparison with past years. That information isn’t as easy to find as you might think. The most recent DOE budget request does show a drop in high energy theory funding, but a more modest one than Yaffe’s figures, and in any case that wouldn’t have shown up in the figures yet. Over the few previous years, things seem pretty stable. Of course, “stable” in nominal terms is a modest de facto decline in real terms, but nothing like the proclaimed “calamity.”

I’m not a particle theorist, and I don’t deal with DOE, so I haven’t paid close attention to DOE funding levels over the years. It’s certainly possible that I’m missing something here. If anyone knows what it is, I’d be interested to hear.

Notes:

  1. One can take the view that the government has no business funding pure curiosity-driven research like particle theory anyway. I don’t agree with that view, although I do understand it.
  2. One can take the more moderate view that, even if the government should be funding things like particle theory, previous funding levels were too high and so a cut isn’t a “calamity.” I don’t have that great a sense of what funding levels are like in particle theory, so it’s hard for me to say for sure what I think about that. In general, I think we should be funding more science, not less, but then as a (modest) beneficiary of government research grants, I would think that, wouldn’t I?
Update: Joanne Hewett’s comment on Sean’s post is by far the most informative thing I’ve seen on this subject.

 

Nature on p-values

February 18th, 2014 by Ted Bunn

Nature has a depressing piece about how to interpret p-values (i.e., the numbers people generally use to describe “statistical significance”). What’s depressing about it? Sentences like this:

Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm.

If it’s really true that “most scientists” think this, then we’re in deep trouble.

Anyway, the article goes on to give a good explanation of why this is wrong:

But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is.

The main point here is the standard workhorse idea of Bayesian statistics: the experimental evidence gives you a recipe for updating your prior beliefs about the probability that any given statement about the world is true. The evidence alone does not tell you the probability that a hypothesis is true. It cannot do so without folding in a prior.

To rehash the old, standard example, suppose that you take a test to see if you have kuru. The test gives the right answer 99% of the time. You test positive. That test “rules out” the null hypothesis that you’re disease-free with a p-value of 1%. But that doesn’t mean there’s a 99% chance you have the disease. The reason is that the prior probability that you have kuru is very low. Say one person in 100,000 has the disease. When you test 100,000 people, you’ll get  roughly one true positive and 1000 false positives. Your positive test is overwhelmingly likely to be one of the false ones, low p-value notwithstanding.

For some reason, people regard “Bayesian statistics” as something controversial and heterodox. Maybe they wouldn’t think so if it were simply called “correct reasoning,” which is all it is.

You don’t have to think of yourself as “a Bayesian” to interpret p-values in the correct way. Standard statistics textbooks all state clearly that a p-value is not the probability that a hypothesis is true, but rather the probability that, if the null hypothesis is true, a result as extreme as the one actually found would occur.

Here’s a convenient Venn diagram to help you remember this:

(Confession: this picture is a rerun.)

If Nature‘s readers really don’t know this, then something’s seriously wrong with the way we train scientists.

Anyway, there’s a bunch of good stuff in this article:

The irony is that when UK statistician Ronald Fisher introduced the P value in the 1920s, he did not mean it to be a definitive test. He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: worthy of a second look.

Fisher’s got this exactly right. The standard in many fields for “statistical significance” is a p-value of 0.05. Unless you set the value far, far lower than that, a very large number of “significant” results are going to be false. That doesn’t necessarily mean that you shouldn’t use p-values. It just means that you should regard them (particularly with this easy-to-cross 0.05 threshold) as ways to decide which hypotheses to investigate further.

Another really important point:

Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. “P-hacking,” says Simonsohn, “is trying multiple things until you get the desired result” — even unconsciously.

I didn’t know the term P-hacking, although I’d heard some of the others. Anyway, it’s a sure-fire way to generate significant-looking but utterly false results, and it’s unfortunately not at all unusual.