{"id":432,"date":"2012-05-16T16:26:25","date_gmt":"2012-05-16T21:26:25","guid":{"rendered":"http:\/\/blog.richmond.edu\/physicsbunn\/?p=432"},"modified":"2012-05-17T10:03:09","modified_gmt":"2012-05-17T15:03:09","slug":"nature-really-hates-psychology","status":"publish","type":"post","link":"https:\/\/blog.richmond.edu\/physicsbunn\/2012\/05\/16\/nature-really-hates-psychology\/","title":{"rendered":"Nature really hates psychology"},"content":{"rendered":"

That’s Nature<\/em>\u00a0the premier science journal, not the nature that abhors a vacuum. Check out this news feature<\/a> on problems in the field.<\/p>\n

These problems occur throughout the sciences, but psychology has a number of deeply entrenched cultural norms that exacerbate them. It has become common practice, for example, to tweak experimental designs in ways that practically guarantee positive results. And once positive results are published, few researchers replicate the experiment exactly, instead carrying out ‘conceptual replications’ that test similar hypotheses using different methods. This practice, say critics, builds a house of cards on potentially shaky foundations.<\/p><\/blockquote>\n

I’m not a psychologist, so I can’t be sure how much merit there is in the article’s indictments. But don’t worry — I won’t let that stop me from commenting!<\/p>\n

The article seems to me to make three main claims:<\/p>\n

    \n
  1. “Publication bias” is a serious problem — results that show something positive and surprising are much more likely to be published.<\/li>\n
  2. Psychologists often use dishonest statistical methods (perhaps unintentionally).<\/li>\n
  3. People don’t replicate previous results exactly.<\/li>\n<\/ol>\n

    Let’s take each in turn.<\/p>\n

    1. Publication bias.<\/strong><\/p>\n

    There’s no doubt that experiments that show a positive result are more highly valued than experiments that don’t show one. It’s much better for your career to find something than to find nothing. That’s bound to lead to a certain amount of “publication bias”: the published literature will contain many more false positives than you’d naively expect, because the false positives are more likely to get published.<\/p>\n

    This problem is exacerbated by the fact that people often use the ridiculously low threshold of 95% confidence to decide whether a result is statistically significant. This means that you expect one out of every 20 tests you do to yield a false positive result. There are lots of people doing lots of tests, so of course there are lots of false positives. That’s why, according to John Ioannidis, “most published [medical] research findings are false<\/a>.”<\/p>\n

    The Nature<\/em>\u00a0piece claims that these problems are worse in psychology than in other fields. I don’t know if that’s true or not. Some evidence from the article:<\/p>\n

    Psychology and psychiatry, according to other work by Fanelli, are the worst offenders: they are five times more likely to report a positive result than are the space sciences, which are at the other end of the spectrum. The situation is not improving. In 1959, statistician Theodore Sterling found that 97% of the studies in four major psychology journals had reported statistically significant positive results5. When he repeated the analysis in 1995, nothing had changed.<\/p>\n

    One reason for the excess in positive results for psychology is an emphasis on \u201cslightly freak-show-ish\u201d results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. \u201cHigh-impact journals often regard psychology as a sort of parlour-trick area,\u201d he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. \u201cWhen we review papers, we’re often making authors prove that their findings are novel or interesting,\u201d he says. \u201cWe’re not often making them prove that their findings are true.\u201d<\/p><\/blockquote>\n

    Incidentally, the article has a graphic illustrating the fraction of times that papers in different disciplines quote positive results, but it doesn’t seem to support the claim that psychology is five times worse than space science:<\/p>\n

     <\/p>\n

    \"\"<\/a><\/p>\n

    I estimate psychology is at about 93% and space science is at about 70%. That’s not a factor of 5.<\/p>\n

    2. Dishonest statistics.<\/strong><\/p>\n

    Nature <\/em>doesn’t use a loaded word like “dishonest,” but here’s what they claim:<\/p>\n

    Many psychologists make on-the-fly decisions about key aspects of their studies, including how many volunteers to recruit, which variables to measure and how to analyse the results. These choices could be innocently made, but they give researchers the freedom to torture experiments and data until they produce positive results.<\/p>\n

    In a survey of more than 2,000 psychologists, Leslie John, a consumer psychologist from Harvard Business School in Boston, Massachusetts, showed that more than 50% had waited to decide whether to collect more data until they had checked the significance of their results, thereby allowing them to hold out until positive results materialize. More than 40% had selectively reported studies that \u201cworked\u201d. On average, most respondents felt that these practices were defensible. \u201cMany people continue to use these approaches because that is how they were taught,\u201d says Brent Roberts, a psychologist at the University of Illinois at Urbana\u2013Champaign.<\/p><\/blockquote>\n

    If you deliberately choose to report data that lead to positive results and not data that lead to negative results, then you’re just plain lying about your data. The same goes <\/a>for continuing to gather data, only stopping when “positive results materialize.”<\/p>\n

    Once again, I am not claiming that psychologists do this. I am claiming that if<\/em> they do this, as\u00a0Nature<\/em>\u00a0claims, it’s a very serious problem.<\/p>\n

    3. People don’t replicate previous studies.<\/strong><\/p>\n

    The Nature<\/em>\u00a0article tells the story of the people who tried to replicate a previous study claiming to show precognition. Journals wouldn’t accept the paper detailing the failed replication. I agree<\/a> with the author that this is a shame.<\/p>\n

    But the article’s concerns on this seem quite overwrought to me. They point out that most of the time people don’t try to replicate studies exactly, instead performing “conceptual replications,” in which they do something similar but not identical to what’s been done before. The author seems to think that this is a problem, but I don’t really see why. Here’s his argument:<\/p>\n

    But to other psychologists, reliance on conceptual replication is problematic. \u201cYou can’t replicate a concept,\u201d says Chambers. \u201cIt’s so subjective. It’s anybody’s guess as to how similar something needs to be to count as a conceptual replication.\u201d The practice also produces a \u201clogical double-standard\u201d, he says. For example, if a heavy clipboard unconsciously influences people’s judgements, that could be taken to conceptually replicate the slow-walking effect. But if the weight of the clipboard had no influence, no one would argue that priming had been conceptually falsified. With its ability to verify but not falsify, conceptual replication allows weak results to support one another. \u201cIt is the scientific embodiment of confirmation bias,\u201d says Brian Nosek, a social psychologist from the University of Virginia in Charlottesville. \u201cPsychology would suffer if it wasn’t practised but it doesn’t replace direct replication. To show that ‘A’ is true, you don’t do ‘B’. You do ‘A’ again.\u201d<\/p><\/blockquote>\n

    It’s true that a “conceptual replication” doesn’t directly falsify a particular result in the same way that a (failed) exact replication would. But I can’t bring myself to care that much. If a result is incorrect, it will gradually become clear that it doesn’t fit in with the pattern built up by many subsequent similar-but-not-identical experiments. The incorrect result will gradually atrophy from lack of attention, even if there’s not a single definitive refutation.<\/p>\n

    At least, that’s the way I see things working<\/a> in physics, where direct replications of previous experiments are extremely uncommon.<\/p>\n","protected":false},"excerpt":{"rendered":"

    That’s Nature\u00a0the premier science journal, not the nature that abhors a vacuum. Check out this news feature on problems in the field. These problems occur throughout the sciences, but psychology has a number of deeply entrenched cultural norms that exacerbate them. It has become common practice, for example, to tweak experimental designs in ways that … Continue reading Nature really hates psychology<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-432","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/432","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/comments?post=432"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/432\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/media?parent=432"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/categories?post=432"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/tags?post=432"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}