Why I am not a Popperian

In my experience, most scientists don’t know or care much about the philosophy of science, but if they do know one thing, it’s Karl Popper’s idea that the hallmark of a scientific hypothesis is falsifiability.  In general, scientists seem to have taken this idea to heart.  For instance, when a scientist wants to explain why astrology, or Creationism, or any number of other things aren’t science, the accusation of unfalsifiability invariably comes up.  Although I’ll admit to using this rhetorical trick myself from time to time, to me the idea of falsifiability fails in a big way to capture the scientific way of thinking.  I hinted about this in an earlier post, but now I’d like to go into it in a bit more detail.

Let me begin with a confession: I have never read Popper.  For all I know, the position I’m about to argue against is not what he really thought at all.  It is of course a cardinal academic sin to argue against someone’s position without  actually knowing what that person wrote.  My excuse is that I’m going to argue against Popper as he is generally understood by scientists, which may be different from the real Popper.  As a constant reminder that I may be arguing against a cartoon version of Popper, I’ll refer to cartoon-Popper as “Popper” from now on.  (If that’s not showing up in a different font on your browser, you’ll just have to imagine it.  Maybe you’re better off.)

I bet that the vast majority of scientists, like me, know only Popper‘s views, not Popper’s views, so I don’t feel too bad about addressing the former, not the latter.

Popper‘s main idea is that a scientific hypothesis must be falsifiable, meaning that it must be possible for some sort of experimental evidence to prove the hypothesis wrong.  For instance, consider the “Matrix-like” hypothesis that you’re just a brain in a vat, with inputs artificially pumped in to make it seem like you’re sensing and experiencing all of the things you think you are.  Every experiment you can imagine doing could be explained under this hypothesis, so it’s not falsifiable, and hence not scientific.

When Popper says that a scientific hypothesis must be falsifiable, it’s not clear whether this is supposed to be a descriptive statement (“this is how scientists actually think”) or a normative one (“this is how scientists should think”).  Either way, though, I think it misses the boat, in two different but related ways.

1. Negativity.  The most obvious thing about Popper‘s falsifiability criterion is that it privileges falsifying over verifying. When scientists talk about Popper, they often regard this as a feature, not a bug.  They say that scientific theories can never be proved right, but they can be proved wrong.

At the level of individual hypotheses, this is self-evidently silly.  Does anyone really believe that “there is life on other planets” is an unscientific hypothesis, but “there is no life on other planets” is scientific?   When I write a grant proposal to NSF, should I carefully insert “not”s in appropriate places to make sure that the questions I’m proposing to address are phrased in a suitably falsifiable way?  It’d be like submitting a proposal to Alex Trebek.

From what little I’ve read on the subject, I think that this objection is about Popper, not Popper, in at least one way.  The real Popper apparently applied the falsifiability criterion to entire scientific theories, not to individual hypotheses.  But it’s not obvious to me that that helps, and anyway Popper as understood by most scientists is definitely about falsifiability of individual hypotheses. For example, I was recently on a committee to establish learning outcomes for our general-education science courses as part of our accreditation process.  One of the outcomes had to do with formulating scientific hypotheses, and we discussed whether to include Popperian falsifiability as a criterion for these hypotheses.  (Fortunately, we decided not to.)

2. “All-or-nothing-ness.” The other thing I don’t like about Popperian falsifiability is the way it thinks of hypotheses as either definitely true or definitely false. (Once again, the real Popper’s view is apparently more sophisticated than Popper‘s on this point.) This problem is actually much more important to me than the first one.  The way I reason as a scientist places much more emphasis on the uncertain, tentative nature of scientific knowledge: it’s crucial to remember that beliefs about scientific hypotheses are always probabilistic.

Bayesian inference provides a much better model for understanding both how scientists do think and how they should think.  At any given time, you have a set of beliefs about the probabilities of various statements about the world being true.  When you acquire some new information (say by doing an experiment), the additional information causes you to update those sets of probabilities.  Over time, that accumulation of evidence drives some of those probabilities very close to one and others very close to zero.  As I noted in my earlier post,Bayes’s theorem provides a precise description of this process.

(By the way, scientists sometimes divide themselves into “frequentist” and “Bayesian” camps, with different interpretations of what probabilities are all about.  Some frequentists will reject what I’m saying here, but I claim that they’re just in denial: Bayesian inference still describes how they reason, even if they won’t admit it.)

For rhetorical purposes if nothing else, it’s nice to have a clean way of describing what makes a hypothesis scientific, so that we can state succinctly why, say, astrology doesn’t count.  Popperian falsifiability nicely meets that need, which is probably part of the reason scientists like it.  Since I’m asking you to reject it, I should offer up a replacement.  The Bayesian way of looking at things does supply a natural replacement for falsifiability, although I don’t know of a catchy one-word name for it.  To me, what makes a hypothesis scientific is that it is amenable to evidence.  That just means that we can imagine experiments whose results would drive the probability of the hypothesis arbitrarily close to one, and (possibly different) experiments that would drive the probability arbitrarily close to zero.

If you write down Bayes’s theorem, you can convince yourself that this is equivalent to the following:  a hypothesis H is amenable to evidence as long as there are some possible experimental results E with the property that P(E | H) is significantly different from P(E | not-H).  That is, there have to be experimental outcomes that are much more (or less) likely if the hypothesis is true than if it’s not true.

Most examples of unscientific hypotheses (e.g., astrology) fail this test on the ground that they’re too vague to allow decent estimates of these probabilities.

The idea of evidence, and the amenability-to-evidence criterion, are pretty intuitive and not too hard to explain:  “Evidence” for a hypothesis just means an observation that is more consistent with the hypothesis being true than with its being false.  A hypothesis is scientific if you can imagine ways of gathering evidence.  Isn’t that nicer than Popperian falsifiability?

8 Responses to “Why I am not a Popperian”

  1. robert61 says:

    Damn, Ted, you keep getting better and better. I subscribed to your feed because you and I have a family history together (think about the guys named Robert whom you know, then narrow them down to the ones born in 1961. Cmon, dude, think! That’s right: me). I loved your earlier Bayes post; this one makes it even better.

    Congrats on getting a nod from Sean Carroll.

    You’re like Eliezer Yudkowsky without the crazy live forever stuff.

  2. Ted Bunn says:

    You’re not Bob Dunlap, are you?

  3. Malcolm Hill says:

    Hi Ted,
    This is great and I enjoyed the diversion into more philosophical territory. Here are some of my comments…
    First, I think it is more correct for you to say that you’re a Bayesian! And I don’t think the two camps (frequentist and Bayesian) are mutually exclusive.
    Second, I think we have to distinguish between experimental and theoretical science. Theoreticians propose mechanistic explanations for how things work. These then serve as hypotheses from which testable predictions flow. Or at least the good theories do. Experimental science, on the other hand, doesn’t work in the same manner, and falsifiability IS really important for us experimentalists employing the frequentist approach.
    Second, I think the distinction between ‘hypotheses’ and null hypotheses has to be made clear. Falsifiability as most scientists practice it involves frequentist type para- and non-parametric statistically thinking. There is no way to avoid the tortured language of “failing to reject” with this approach. I still argue that this type of thinking is extraordinarily important in the sciences, for without it, we have limited the number of tools to discern patterns from noise (Bayesian approaches aside).
    Third, and along those lines, I don’t think anyone would claim that “there is life on other planets" is unscientific. It is a perfect hypothesis. And it is falsifiable in a Popperian sense, though examining the planets that might float around the 10^21 stars might be a challenge. "There is no life on other planets," however, is a great null hypothesis that has value in a statistical sense. Finally, I agree that a Bayesian perspective might be more intuitive and better reflect the way our minds work, but not that it is the way we SHOULD think. Those normative statements are always thorny knots for me. That is ultimately what Popper is trying to avoid – the oughts/shoulds lead to some dangerous intellectual territory when you’re dealing with a universe that is knowable within the context of our evolved brains (which have many known and unknown limitations). I get nervous with verifiability for the following reason – facts change and a clean negation is very nice indeed. But my emotional response doesn’t mean that it is that way, which is why I do like the ‘amenable to evidence perspective’ and a Bayesian perspective.
    We should talk about ways to incorporate more Bayesian perspectives in Biology.
    Great thoughts!
    Malcolm

  4. Ted Bunn says:

    Thanks for the detailed comments, Malcolm! Most of them require a longer response than I can give now; I’ll try to get to them later. For the moment, let me just come out of the closet and say that yes, I absolutely am a Bayesian, and I never intended to imply otherwise.

    As to whether the two camps are mutually exclusive, I think we have to be careful about what we mean by the terms. There are Bayesian and frequentist approaches to the philosophy of probability. That is, there’s a Bayesian idea and a frequentist idea of what the word “probability” means. Those are mutually exclusive: you can’t simultaneously believe both, although you can certainly be agnostic and get on with your life. There are also Bayesian and frequentist methods for dealing with data, and those aren’t mutually exclusive. They’re just sets of tools, and you can (and I for one do) choose different tools on different days.

  5. Sam says:

    Hi Ted,
    I tend to agree with Malcolm that falsifying established as well as tentative ideas is really important. It’s surely necessary as part of the process of discovering new facts about the world (rather than merely verifying existing facts). I’m sure the inferential framework can accommodate that process in terms of a “Crucial Test” of a theory, as described by Jeffreys in his treatise on Scientific Inference:

    http://tinyurl.com/bus8bw

  6. Ted Bunn says:

    Well, I certainly agree that falsifying is important. (I’m also generally pro-motherhood and apple pie!) But

    1. Verifying is just as important!
    2. Both “falsify” and “verify” are just shorthand for probabilistic statements. To me, “gathering evidence for / against” is a much more accurate description of what scientists really do.

    Finally, if I were omnipotent, I would excise all discussion of “crucial tests” from discussions of science. Few ideas do more to misrepresent what science is all about in the popular imagination.

  7. robert61 says:

    Hi Ted. You got it in one.

    I shouldn’t have dissed Yudkowsky. Mere layman’s blather. I find him very provocative. He has done more than anybody to sell me on Bayesianism, which I otherwise encountered first in the form of a 100% effective spam-killing algorithm I implemented in the early 00s and which seems to correspond better than any other philosophical formalism I’ve encountered to my own seat of the pants understanding of how to evaluate uncertain information.

  8. Ted Bunn says:

    I have to confess complete ignorance about Yudkowsky. I have a little rant about Bayesian and frequentist ideas about probability building in my mind. I’ll look him up in preparation.

    Bayesian spam filters are amazing, aren’t they? When I first installed one, I was surprised not only at how well it worked but at how quickly it got itself trained. It really doesn’t take much training data to get good results.

Leave a Reply