{"id":824,"date":"2014-02-18T17:53:09","date_gmt":"2014-02-18T22:53:09","guid":{"rendered":"http:\/\/blog.richmond.edu\/physicsbunn\/?p=824"},"modified":"2014-02-18T17:59:36","modified_gmt":"2014-02-18T22:59:36","slug":"nature-on-p-values","status":"publish","type":"post","link":"https:\/\/blog.richmond.edu\/physicsbunn\/2014\/02\/18\/nature-on-p-values\/","title":{"rendered":"Nature on p-values"},"content":{"rendered":"

Nature<\/em> has a depressing\u00a0piece<\/a> about how to interpret p<\/em>-values (i.e., the numbers people generally use to describe “statistical significance”). What’s depressing about it? Sentences like this:<\/p>\n

Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm.<\/p><\/blockquote>\n

If it’s really true that “most scientists” think this, then we’re in deep trouble.<\/p>\n

Anyway, the article goes on to give a good explanation of why this is wrong:<\/p>\n

But they would be wrong. The P<\/em> value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour \u2014 possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis \u2014 telepathy, aliens, homeopathy \u2014 the greater the chance that an exciting finding is a false alarm, no matter what the P<\/em> value is.<\/p><\/blockquote>\n

The main point here is the standard workhorse idea of Bayesian statistics: the experimental evidence gives you a recipe for\u00a0updating<\/em> your\u00a0prior beliefs<\/em> about the probability that any given statement about the world is true. The evidence alone does not tell you the probability that a hypothesis is true. It cannot do so without folding in a prior.<\/p>\n

To rehash the old, standard example, suppose that you take a test to see if you have kuru<\/a>. The test gives the right answer 99% of the time. You test positive. That test “rules out” the null hypothesis that you’re disease-free with a p<\/em>-value of 1%. But that doesn’t mean there’s a 99% chance you have the disease. The reason is that the prior probability that you have kuru is very low. Say one person in 100,000 has the disease. When you test 100,000 people, you’ll get \u00a0roughly one true positive and 1000 false positives. Your positive test is overwhelmingly likely to be one of the false ones, low p<\/em>-value notwithstanding.<\/p>\n

For some reason, people regard “Bayesian statistics” as something controversial and heterodox. Maybe they wouldn’t think so if it were simply called “correct reasoning,” which is all it is.<\/p>\n

You don’t have to think of yourself as “a Bayesian” to interpret p<\/em>-values in the correct way. Standard statistics textbooks all state clearly that a p<\/em>-value is not the probability that a hypothesis is true, but rather the probability that, if the null hypothesis is true, a result as extreme as the one actually found would occur.<\/p>\n

Here’s a convenient Venn diagram to help you remember this:<\/p>\n

\"\"<\/a><\/p>\n

(Confession: this picture is a rerun<\/a>.)<\/p>\n

If Nature<\/em>‘s readers really don’t know this, then something’s seriously wrong with the way we train scientists.<\/p>\n

Anyway, there’s a bunch of good stuff in this article:<\/p>\n

The irony is that when UK statistician Ronald Fisher introduced the P value in the 1920s, he did not mean it to be a definitive test. He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: worthy of a second look.<\/p><\/blockquote>\n

Fisher’s got this exactly right. The standard in many fields for “statistical significance” is a p<\/em>-value of 0.05. Unless you set the value far, far lower than that, a very large number of “significant” results are going to be false. That doesn’t necessarily mean that you shouldn’t use p<\/em>-values. It just means that you should regard them (particularly with this easy-to-cross 0.05 threshold) as ways to decide which hypotheses to investigate further.<\/p>\n

Another really important point:<\/p>\n

Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P<\/em>-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. \u201cP<\/em>-hacking,\u201d says Simonsohn, \u201cis trying multiple things until you get the desired result\u201d \u2014 even unconsciously.<\/p><\/blockquote>\n

I didn’t know the term P<\/em>-hacking, although I’d heard some of the others. Anyway, it’s a sure-fire way to generate significant-looking but utterly false results, and it’s unfortunately not at all unusual.<\/p>\n","protected":false},"excerpt":{"rendered":"

Nature has a depressing\u00a0piece about how to interpret p-values (i.e., the numbers people generally use to describe “statistical significance”). What’s depressing about it? Sentences like this: Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. If it’s … Continue reading Nature on p-values<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-824","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/824","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/comments?post=824"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/824\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/media?parent=824"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/categories?post=824"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/tags?post=824"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}