{"id":211,"date":"2010-04-03T19:44:23","date_gmt":"2010-04-04T00:44:23","guid":{"rendered":"http:\/\/blog.richmond.edu\/physicsbunn\/2010\/04\/03\/good-or-bad-bayes\/"},"modified":"2010-04-03T19:44:23","modified_gmt":"2010-04-04T00:44:23","slug":"good-or-bad-bayes","status":"publish","type":"post","link":"https:\/\/blog.richmond.edu\/physicsbunn\/2010\/04\/03\/good-or-bad-bayes\/","title":{"rendered":"Good or bad Bayes?"},"content":{"rendered":"

My brother Andy pointed me to this discussion<\/a> on Tamino’s Open Mind blog of Bayesian vs. frequentist statistical methods.\u00a0 It’s focused on a nice, clear-cut statistics problem from a textbook by David MacKay<\/a>, which can be viewed in either a frequentist or Bayesian way:<\/p>\n

We are trying to reduce the incidence of an unpleasant disease called microsoftus. Two vaccinations, A and B, are tested on a group of volunteers. Vaccination B is a control treatment, a placebo treatment with no active ingredients. Of the 40 subjects, 30 are randomly assigned to have treatment A and the other 10 are given the control treatment B. We observe the subjects for one year after their vaccinations. Of the 30 in group A, one contracts microsoftus. Of the 10 in group B, three contract microsoftus. Is treatment A better than treatment B?<\/em><\/p><\/blockquote>\n

Tamino reproduces MacKay’s analysis and then proceeds to criticize it in strong terms.\u00a0 Tamino’s summary:<\/p>\n

Let \"\\theta_A\" be the probability of getting "microsoftus" with treatment A, while \"\\theta_B\" is the probability with treatment B. He adopts a uniform prior, that all possible values of \"\\theta_A\" and \"\\theta_B\" are equally likely (a standard choice and a good one). "Possible" means between 0 and 1, as all probabilities must be.<\/p>\n

He then uses the observed data to compute posterior<\/em> probability distributions for \"\\theta_A,~\\theta_B\". This makes it possible to computes the probability that \"\\theta_A (i.e., that you’re less likely to get the disease with treatment A than with B). He concludes that the probability is 0.990, so there’s a 99% chance that treatment A is superior to treatment B (the placebo).<\/p><\/blockquote>\n

Tamino has a number of objections to this analysis, which I think I agree with, although I’d express things a bit differently.\u00a0 To me, the problem with the above analysis is precisely the part that Tamino says is “a standard choice and a good one”: the choice of prior.<\/p>\n

MacKay’s choice of prior expresses the idea that, before looking at the data, we thought that all possible pairs of probabilities (\"\\theta_A,~\\theta_B\") were equally likely.\u00a0\u00a0 That prior is very unlikely to be an accurate reflection of our actual prior state of belief regarding the drug.\u00a0 Before you looked at the data, you probably thought there was a non-negligible chance that the drug had no significant effect at all — that is, that the two probabilities were exactly (or almost exactly) equal. So in fact your prior probability was surely not a constant function on the\u00a0 (\"\\theta_A,~\\theta_B\") plane — it had a big ridge running down the line \"\\theta_A\" = \"\\theta_B\".\u00a0 An analysis that assumes a prior without such a ridge is an analysis that assumes from the beginning that the drug has a significant effect with overwhelming probability.\u00a0 So the fact that he concludes the drug has an effect with high probability is not at all surprising — it was encoded in his prior from the beginning!<\/p>\n

The nicest way to analyze a situation like this from a Bayesian point of view is to compare two different models: one where the drug has no effect and one where it has some effect. \u00a0\u00a0 MacKay analyzes the second one.\u00a0 Tamino goes on to analyze both cases and compare them. \u00a0 He concludes that the probability of getting the observed data is 0.00096 under the null model (drug has no effect) and 0.00293 under the alternative model (drug has an effect).<\/p>\n

How do you interpret these results?\u00a0 The ratio of these two probabilities is about 3.\u00a0 This ratio is sometimes called the Bayesian evidence ratio,\u00a0 <\/em>and it tells you how to modify your prior probability for the two models.\u00a0 To be specific,<\/p>\n

Posterior probability ratio = Prior probability ratio x evidence ratio.<\/p>\n

For instance, suppose that before looking at the data you thought that there was a 1 in 10 chance that the drug would have an effect.\u00a0 Then the prior probability ratio was (1\/10) \/ (9\/10), or 1\/9.\u00a0 After you look at the data, you “update” your prior probability ratio to get a posterior probability ratio of 1\/9 x 3, or 1\/3.\u00a0 So after looking at the data, you now think there’s a 1\/4 chance that the drug has an effect and a 3\/4 chance that it doesn’t.<\/p>\n

Of course, if you had a different prior probability, then you’d have a different posterior probability.\u00a0 The data can’t tell you what to believe; it can just tell you how to update your previous beliefs.<\/p>\n

As Tamino says,<\/p>\n

Perhaps the best we can say is that the data enhance the likelihood that the treatment is effective, increasing the odds ratio by about a factor of 3. But, the odds ratio after this increase depends on the odds ratio before the increase \u20ac\u201d which is exactly the prior we don't really have much information on!<\/p><\/blockquote>\n

People often make statement like this as if they’re pointing out a flaw in the Bayesian analysis, but this isn’t a bug in the Bayesian analysis — it’s a feature!\u00a0 You shouldn’t expect the data to tell you the posterior probabilities in a way that’s independent of the prior probabilities.\u00a0 That’s too much to ask.\u00a0 Your final state of belief will be determined by both the data and your prior belief, and that’s the way it should be.<\/p>\n

Incidentally, my research group’s most recent paper<\/a> has to do with a problem very much like this situation:\u00a0 we’re considering whether a particular data set favors a simple model, with no free parameters, or a more complicated one.\u00a0 We compute Bayesian evidence ratios just like this, in order to tell you how you should update your probabilities for the two hypotheses as a result of the data.\u00a0 But we can’t tell you which theory to believe — just how much your belief in one should go up or down as a result of the data.<\/p>\n","protected":false},"excerpt":{"rendered":"

My brother Andy pointed me to this discussion on Tamino’s Open Mind blog of Bayesian vs. frequentist statistical methods.\u00a0 It’s focused on a nice, clear-cut statistics problem from a textbook by David MacKay, which can be viewed in either a frequentist or Bayesian way: We are trying to reduce the incidence of an unpleasant disease … Continue reading Good or bad Bayes?<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-211","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/211","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/comments?post=211"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/211\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/media?parent=211"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/categories?post=211"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/tags?post=211"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}