{"id":869,"date":"2014-05-21T16:19:46","date_gmt":"2014-05-21T21:19:46","guid":{"rendered":"http:\/\/blog.richmond.edu\/physicsbunn\/?p=869"},"modified":"2014-05-21T16:19:46","modified_gmt":"2014-05-21T21:19:46","slug":"replication","status":"publish","type":"post","link":"https:\/\/blog.richmond.edu\/physicsbunn\/2014\/05\/21\/replication\/","title":{"rendered":"Replication"},"content":{"rendered":"

I heard (via Sean Carroll) about\u00a0this piece<\/a> in Science\u00a0<\/em>headlined “Replication Effort Provokes Praise\u2014And ‘Bullying’ Charges.” It’s about efforts to replicate published results in certain areas of psychology.<\/p>\n

In general, I think that publication bias and dodgy statistics are real problems in science, so I’d bet that lots of results, particularly those that are called “significant” because they clear the ridiculously weak threshold of 5%, are wrong. Apparently lots of people, particularly in certain parts of psychology<\/a>, are worried about this.\u00a0I think it’s great for people to try to replicate past results and find out. (Medical researchers are on the case too, particularly John Ioannidis, who claims<\/a> that “most published research findings are false.”)<\/p>\n

The most striking part of the Science<\/em> piece is the “bullying” claim. It seems ridiculous on its face for a scientist to complain about other people trying to replicate their results. Isn’t that what science is all about? But I can understand in part what they’re worrying about. You can easily imagine someone trying to replicate your work, doing something wrong (or perhaps just different from what you did), and then publicly shaming you because your results couldn’t be replicated. For instance,<\/p>\n

Schnall [the original researcher] contends that Donnellan\u2019s effort [to replicate Schnall’s results] was flawed by a \u201cceiling effect\u201d that, essentially, discounted subjects\u2019 most severe moral sentiments. \u201cWe tried a number of strategies to deal with her ceiling effect concern,\u201d Donnellan counters, \u201cbut it did not change the conclusions.\u201d Donnellan and his supporters say that Schnall simply tested too few people to avoid a false positive result. (A colleague of Schnall\u2019s, Oliver Genschow, a psychologist at Ghent University in Belgium, told Science in an e-mail that he has successfully replicated Schnall\u2019s study and plans to publish it.)<\/p><\/blockquote>\n

The solution, of course, is for Donnellan to describe clearly what he did and how it differs from Schnall’s work. The readers can then decide (using Bayesian reasoning, or as I like to call it<\/a>, “reasoning”) whether those differences matter and hence how much to discount the original work.<\/p>\n

The piece quotes Daniel Kahneman giving an utterly sane point of view:<\/p>\n

To reduce professional damage, Kahneman calls for a \u201creplication etiquette,\u201d which he describes in a commentary published with the replications in Social Psychology. For example, he says, \u201cthe original authors of papers should be actively involved in replication efforts\u201d and \u201ca demonstrable good-faith effort to achieve the collaboration of the original authors should be a requirement for publishing replications.\u201d<\/p><\/blockquote>\n

If the two groups work in good faith to do a good replication, it’ll make the final results easier to interpret. If the original group refuses to work with people who are trying to replicate their results, well, everyone is entitled to take that into account when performing (Bayesian) reasoning about whether to believe the original results.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

I heard (via Sean Carroll) about\u00a0this piece in Science\u00a0headlined “Replication Effort Provokes Praise\u2014And ‘Bullying’ Charges.” It’s about efforts to replicate published results in certain areas of psychology. In general, I think that publication bias and dodgy statistics are real problems in science, so I’d bet that lots of results, particularly those that are called “significant” … Continue reading Replication<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-869","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/869","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/comments?post=869"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/869\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/media?parent=869"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/categories?post=869"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/tags?post=869"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}