The Null Hypothesis

Facebook Religious ViewsOne of the most accurate ways to describe my religious beliefs (or lack thereof) is by way of a concept known as the “null hypothesis”. Like most atheists, I do not claim that I know God does not exist. I merely claim that there is not enough evidence to justify belief in God. And the best way to illustrate this claim is through the null hypothesis. This is a statistical concept that is used for hypothesis testing in science. Because statistics is not a strong point for many people, I will try to explain it using a minimum of stats jargon; however, some will be required, and I will try to explain what each term means the best that I can. I really feel that this is an important concept to understand when one is trying to assess evidence claims (which happens to us all the time). So hang on for the ride!

Basic Statistics

Normal DistributionWhen I measure a certain phenomenon, such as people’s height, there will always be some variability in the results. And for many phenomena, this variability will result in a bell-shaped curve known as a “normal distribution”. There are some very interesting properties of the normal distribution, but what is important here is that the majority of the cases will be clustered around the mean (the statistical word for “average”), and then fewer cases will occur further away from the mean. So for instance, the mean height for a Canadian adult is about 5’8″ for men and 5’3″ for women.1 So if I randomly selected a Canadian man, he would be most likely to have a height somewhere around 5’8″. Or if I wanted to be more precise, I could say that he is likely to fall between about 4’11” and 6’5″. I’d be very surprised if this randomly selected person turned out to be 8 feet tall. There are very few people who are that tall, so it would be very unusual to have chosen them. So the normal distribution tells us that the most frequent cases occur around the mean, and that cases occurring further away are less frequent.

Hypothesis Testing

So what would I do if someone told me of a town in rural Alberta where all the adult townspeople were over 7 feet tall? I would likely be very skeptical. Such a thing is very unlikely, isn’t it? It’s unusual to find even one person over 7 feet tall, let alone an entire town of both men and women over 7 feet tall! But to be a good scientist, I should hold onto my skepticism but remain open to the possibility that such a strange case is indeed true. I would want to head over to this town and start measuring. But let’s say I don’t have the resources to measure every single individual in the town. I might measure 40 or 50 of them. And perhaps the person who told me this story was exaggerating a little bit, and some were below 7 feet. But I might still ask, “Are the people in this town significantly taller than the general population?” And that is a great question with which to use hypothesis testing.

Null HypothesisWhen scientists want to test a hypothesis, they must come up with two different hypotheses to compare. One of these (known as the alternate hypothesis) is the one they would ideally like to be confirmed; in my case, it is this: “The people in this rural town are statistically taller than the general population.” The other hypothesis is known as the null hypothesis, and in my case it would be the following: “There is no difference in the average height of the people in this rural town and the general population.” But why do we need to compare these two things? Why not just see if the alternate hypothesis is confirmed? The reason is because, like I mentioned earlier, phenomena always have some variability. If I measured the townspeople and found out that they averaged 6’8″, that doesn’t mean that every single person was 6’8″ tall. Some were taller, and some were shorter. So how would I know whether I was finding some actual difference between the townspeople and the general population, or if I just happened to select the 50 tallest people in the town? Hypothesis testing helps us to determine whether we just have a bad sample, or if the normal distribution for the town is actually different from the normal distribution for the general population. Or to put it another way, it helps us distinguish between real differences and random fluctuation. The null hypothesis says, “This is just random fluctuation,” and the alternate hypothesis says, “No, this is a real difference.” Scientists support the alternate hypothesis indirectly by disconfirming the null hypothesis. If the data that scientists collect don’t fit with the null hypothesis, then they have better evidence to support the idea that the alternate hypothesis is true. On the other hand, if the data do fit with the null hypothesis, this doesn’t necessarily mean that the null hypothesis is true; it just means that the data do not contradict it.

The Role of Evidence

Another way to look at the concept of the null hypothesis is to use the analogy of a court case. In the modern justice system, the defendant is considered innocent until proven guilty. The prosecution must provide sufficient evidence to prove his or her guilt. In some ways, this works similar to the null hypothesis. The null hypothesis is assumed to be true until sufficient evidence is provided to demonstrate otherwise. The evidence in favour of the alternate hypothesis must outweigh the evidence in favour of the null hypothesis.2 If I want to prove the statement that the Albertan townspeople are significantly taller than the general population, I have to have sufficient evidence that my measurements are not just a reflection of random variability and then add more evidence suggesting that they are actually due to some real difference in height (whatever the cause of this unusually tall town might be).

Black SwanThe null hypothesis works very similarly when it comes to other types of claims. The statement, “God exists”, is a positive claim about the existence of an entity. So in this case, the null hypothesis would be, “God does not exist.” This works the same as any other positive claim: “Unicorns exist” vs. “Unicorns do not exist”, “It rained yesterday” vs. “It did not rain yesterday”, “Black swans exist” vs. “Black swans do not exist”, and so on. If one wants to prove the truth of these statements, they must provide evidence which is sufficiently inconsistent with the null hypothesis. If I try to prove that it rained yesterday by saying that I held my hand out the window and felt splashes of water, this might be good evidence, unless someone points out that I have a leaky eaves-trough directly above my window. In other words, if my evidence can be shown to be perfectly consistent with the idea that it did not, in fact, rain yesterday, then it should not be used to support the idea that it did rain yesterday. I should provide other evidence to support my claim. Or if I mention that I saw a black swan, but upon further examination it turns out to be a black duck, this is no longer evidence that black swans exist. The fact that the evidence must be thrown out does not prove that black swans don’t exist, but it no longer proves that they do. And if there were absolutely no good evidence to prove that black swans exist, why would anyone believe in them?

The claim that God exists is exactly the same as all these other claims. The burden of proof is always on the person making the positive claim. That person must provide good evidence to demonstrate that the claim is true. And if the evidence they provide is entirely consistent with the null hypothesis (that God does not exist), then the evidence is no good. Again, this does not prove that God does not exist, but if there is absolutely no good evidence to prove that God exists, why would anyone believe in him? This is why the null hypothesis is a crucial concept to grasp. Saying “The evidence does not prove that God exists” is entirely different than saying “The evidence proves that God does not exist”. The null hypothesis is a safe bet that requires other hypotheses to prove themselves to be true before one believes in them. If I went to a random rural town in Alberta, my first assumption would be that their average height is about the same as the average height of the general population. This doesn’t mean that it actually is the case that their average heights are the same, but I have no prior reason to think so until I have evidence to prove otherwise. It would be silly to point to an entirely random town for which one has no prior knowledge and say, “That town is filled with giants.” And it would be even sillier to take measurements of the townspeople, see that they have an average height similar to the general population, and still say, “That town is filled with giants.” As far as I am concerned, however, that is what most believers in God do.

Conclusion

I don’t intend to get into an examination of the evidence for and against the existence of God. I have written an entire ten-part series about just that. The intent of this article was to develop a process for assessing whether God exists. The general process is to assume that God does not exist until one finds sufficient evidence to support the claim that he does. To do otherwise is to choose a position that one likes before even taking a look at the evidence. (It’s like assuming the town is full of giants before even seeing a townsperson.) In many cases where this happens, people do this for emotional reasons. They have some desire to believe a certain thing, and so they make up their mind before they even take a look at the evidence. Such a process is irrational, especially since humans have “tendencies”. We often see patterns that are not there (like people who develop superstitions, for example); we seek out evidence to support the beliefs that we already hold and ignore disconfirming evidence, instead of trying to objectively assess all the evidence; and we often reach conclusions that we like and then make up justifications for them afterwards. The only way to avoid these tendencies is to acknowledge that we suffer from them and then try to minimize them through the use of rigorous processes such as hypothesis testing.

Hypothesis

So when I say that I am a “null hypothesis atheist”, I mean that the evidence for God’s existence is either a) faulty, b) illusory, or c) consistent with the null hypothesis that God does not exist. I do not make the statement, “Therefore, God does not exist.” Instead, I simply say, “There is no good evidence to believe that God exists, and I don’t believe in things with no good evidence.” Sure, God might be out there, hiding behind some distant spiral galaxy or outside of space and time altogether. But I will take the conservative path that tries to minimize irrationality, because such a path is the best process we’ve developed to distinguish fact from fiction.

Notes:

  1. I should note that human height is not technically normally distributed, but rather negatively skewed. I’m using this example for illustrative purposes. []
  2. Any statisticians reading this are probably cringing at this point. I understand that this is a distortion of how hypothesis testing actually works. But as an admittedly imperfect analogy, I still think it works well enough for providing non-statisticians with an understanding of the basic reasoning behind the null/alternate hypotheses. []

12 responses to “The Null Hypothesis”

Nidreya

Hi Jeff,

Is it write to assume that everyone is coming from the same starting point.

Imagine a character who, quite unexpectedly and without any obvious explanation such as drugs or illusions by others, meets ‘God’.

Would that person then not be seeking to explain, or indeed to explain away, an experience for which she has no better name than ‘meeting god’ because that’s what it was. It told me so and it certainly fitted the description.

I am keen to try out ‘the god helmet’ to see if my person al god is repeatable for me at least.

Jeff

Hi Nidreya,

The null hypothesis is not an attempt to “explain away” experiences. It’s a way to ensure that a belief is actually justified with proper evidence. So in the case where someone truly does “meet God”, as you say, then they might be justified in believing in God. If there is no other decent natural explanation for what happened to them, then they probably are justified. But keep in mind that many people in many cultures have visions of “God” that are very different from one another, so even without an explanation such as drugs, the evidence from such experiences is fairly weak. It’s more likely that it is a natural phenomenon that is then being interpreted to fit into their own cultural beliefs about God.

Essentially, what I’m saying is that personal, subjective experiences (as opposed to physical evidence that is accessible to many people) tends to be very weak evidence for the existence of God. Why? Because people of various religions have similar experiences; because we know that our brain can play tricks on us; because there is no way for others to independently test one’s own experience; and because we are able to reproduce these sorts of experiences in lab settings—like the God Helmet you mentioned!

So my final answer: Theoretically, personal experience of God could provide evidence for someone to believe. But practically speaking, it provides very weak evidence at best.

Thanks for the comment!

Jeremy C

Jeff,

Just came across your blog when googling Null Hypothesis and belief in God.

Thanks, very good and clear explanation.

BTW I am a bible believing christian (yeah it must be a bit funny to have a christian give you this response but then I’ve never been big on the subjective experience thingy i.e. I’ve never had an ‘experience’ of God – but thats an aside).

Peter Shaw

Dear All

Just as a postscript to this fascinating blog, I put up a similar line of argument on my web page called “A null hypothesis for religion”, in which I suggest that one can predict all manner of religious behaviours under a Darwinian model of mind-virus evolution. The way that Christ’s message evolved from being anti-family and inclusive to modern evangelicals mandating patriarchal structures and loathing homosexuality, the way that religions are persistently fissiparous and often more intolerant of their close competitors than distant systems, a simple demonstration of churches around the UK lying in public about basic facts about Jesus ….. all can be derived from a Darwinian null hypothesis. I reckon you need just 2 parameters to model the system mathematically. [There is a nice little undergrad project here fitting data on belief systems (in their own words) into evolutionary tree algorithms like PAUP and/or community ordination algorithms like CCA (or 2BPLS, DCA..) and running Monte Carlo tests against the null hypothesis I supply – dunno which department would ever do it though :=) ]

The really interesting question is what the recently evolved change is in our CNS structure that gives Homo sapiens alone the facility for transendence, with which this particular family of replicators seems to have co-evolved. As far as I know chimpanzees don’t get a positive experience off LSD or DMT (though the experimental setups used sound to be worse than useless).

Anyway, see what you think. I’ve been warned that some people (of the evangelical tendency) will get very cross when reading this, so don’t blame me if an acquaintance stops talking to you if you forward the link. I’d say “Duh, that’s how you’d predict a virus to have evolved for its self protection”. When an Islamic student sent me a mass-email about the virtues of the Koran I sent “Reply to All” and sent this link to the lot of them, on the grounds that if you self-censor freedom of speech, they have won without effort.

The URL is
http://www.chezshaw.demon.co.uk/h0.htm
and sorry – I keep seeing typos lurking in it (I create the page in ascii text using notepad, usually after a beer…)

I’m not much of a blogger and dunno if anyone will ever come back about this, but for future reference the university part of my web page tells you how to contact me should anyone want to.
Peter Shaw

Jeff

Hi Peter,

Your site was an interesting read! Thanks for sharing that. I’m not entirely sold on the notion of memetics—I think it’s a useful analogy, but as a theory it seems to be largely unfalsifiable. Most of the analysis (at least that I’ve read) has been retrospective; if a message succeeded, it was a “good” meme, and if a message died out, it was a “bad” meme. Without (a) a better specified model indicating key factors that increase/decrease successful transmission, and (b) an ability to predict in advance the success of a particular message, I don’t think it holds much scientific weight. Those are at least things that can probably be developed. But at this point, you’ve essentially developed a null hypothesis that explains everything, to the point where it can’t, in principle, be rejected. That doesn’t seem to be useful.

At any rate, regardless of the usefulness of memetic theory, I think you’ve set up a double standard with what you write. Your null hypothesis is that organized religions are “mind-viruses”. But when discussing the possibility of refuting the null, you say, “All you have to do is to come up with measurement that can be made that would refute my null hypothesis. That is how science advances – you discard hypotheses that have failed to stand examination.” This is inconsistent. Memes are not (always) transmitted on the basis of their truth-value, or their scientific merit, or their accordance with reality. So saying that such a hypothesis must be refuted by displaying the truth of one’s religion does not actually refute the hypothesis anyway. Even if a religion were true, it would still be passed down through memes, and the message could still die out and be lost given the appropriate factors. And keep in mind that while the word “virus” has a negative connotation, memetics would state that both positive and negative beliefs are passed down through these evolutionary processes. One could make similar observations about scientific advances; certainly science tries to select for models and theories which more closely accord with reality, but because science is a human endeavour, it does not always do this perfectly. The potential for a true theory to be lost due to poor transmission is possible. In short, evaluating an idea or set of ideas on the basis of its truth-value does not indicate whether or not it is a “meme”.

So I think you ultimately have to pick one or the other. (Given my previous statements, I’d suggest the strategy dealing with truth-value.) Either your null hypothesis has to be refuted on the basis of demonstrating that religions are not memes (which seems to me to be impossible, given the all-encompassing scope of memetic theory), or you should revise your null hypothesis to be something more like, “Organized religions are not accurate models of reality,” a statement which is, in principle, falsifiable.

At any rate, sorry to tear apart what you’ve written; I did appreciate it, despite my critique of it. Thanks for stopping by and leaving your comment!

Cheers,
Jeff

Peter Shaw

Dear Jeff

As things stand you are right and I’m not going to contest the point. Unlike a DNA sequence, we cannot yet identify a memory sequence, hence look at a meme. This is only a practical problem, in the same way that Darwin’s inability to measure units of heritance stopped him advancing further into genetics. It did not undermine his basic truism – the item of heritability that gets commoner, gets commoner. Now we can see that in nucleotide sequences, and I can see no a-priori objection to some fancy trans-cranial scanning tool that allows researchers to study the accumulation of memories in humans. Once we can do that then the circularity disappears. One of the grumpier bits of feedback I got from a humanities staff member here was “what next – a null hypotheis for history?” Yes, actually, if you could study the accumulation of ideas as patterns in heads and see each person’s payoff algorithms you could model how society should evolve (or rather model ideas about society get copied down the generations. Note that the availabilty of net/mobile comms to free up flow of memes is transforming the arab world – an evolutionary response to a new selective environment for ideas!)

A second concession; my implicit assumption that humans differ at a neuro-theological level from chimpanzees is probably untenable, just the molecular ptolomeicism of someone who spent too long around old hippies. I knew from Alexander Shulgin’s work that we have no animal models for psychedelics, but that is more of a communication problem than a molecular one. Reports from dosing chimps with LSD in the 1950s include behaviour that sounds much like what Aldous Huxley might have done in “the doors of perception”, had he been a chimpanzee! More seriously, there seem to me few if any substantive differences in the post-synaptic 5HT receptors between our species. When I get around to updating my page further, I’ll concede your points, and raise the (utterly untestable) conjecture that if chimps could communicate like humans they would have enough spontaneous background transcendence in the species to evolve mystical religions just like Homo sapiens.

Best

Peter Shaw

Jay

Hi,

I found this post both informative and helpful in a discussion I am having with a friend. Below is his response and I informed him that I would share this with you for comment.

“Very interesting but he is setting up a universal negative (i.e. atheism) and an intangible concept (i.e. atheism) as the Null Hypothesis, which are both absolute no-no’s in science. In science, the burden of proof is on anyone who makes a claim, whether positive or negative, not just someone who makes a positive claim. As such, you can’t set a negative claim as a default. In addition, in science, you can only set the Null hypothesis for something that can be measured or has been measured. For example you cannot set the “existence” of something as the Null because we have no way to measure the “existence” of something. As an example, you can’t set up “Black swans do not exist as a Null”. Since it is a universal negative based on the intangible concept of “existence”. However, we do know that there are swans with white plumage, based on other measurements, so a better Null would be “The only swans we can measure are those with white plumage”. The alternate hypothesis would be “We can measure swans with black plumage”. As such, you cannot set up atheism (i.e. God does not exist) as the null hypothesis because you cannot set up a universal negative as a Null Hypothesis and because both “existence” and the concept of God are intangible. We have no measurable data to comment on God so you cannot use any position concerning God (atheistic or theistic) as a Null. The best we can hope to do is test measurable differences between concepts (i.e. atheism and theism). Since there is no previous measurements supporting these concepts the best Null would be “There is no measurable difference between the different forms of atheism and theism”. Learning to use the Null Hypothesis is a critical part of gaining a PhD. Do you have posts from any fellow atheists who have actually have a PhD?”

I look forward to your comment.

Jeff

Hi Jay,

Glad to hear that this post has been helpful for you.

In reply to your friend, I would take issue with a few points. For one, he is stating that atheism is a negative claim (i.e., God does not exist), whereas in the post I am specifically arguing for a type of atheism that is a skepticism of a positive claim (i.e., there is insufficient evidence to believe that God exists). It’s a crucial distinction, and while I agree that the burden of proof is upon anyone making either positive or negative claims, part of my argument is that so-called “null hypothesis atheism” is not a negative claim. Many people might call that “agnoticism”, but I generally find that the distinction between agnoticism and atheism withers away when examined. Most atheists do not make a strong negative claim about God.

Aside from that, I take issue with two statements your friend made: “You can’t set a negative claim as a default”, and “you cannot measure the existence of something.” With regard to the first, scientists do that all the time. “These two samples were drawn from the same population” (a positive claim) is equivalent to “there is no statistical difference between these two samples” (a negative claim); the distinction is merely semantic. Perhaps your friend is trying to make the point that we cannot “prove” the null hypothesis, only fail to reject it. If so, that’s fine, but it’s not the type of claim (positive/negative) that is the issue, only the constraints regarding how such claims are tested statistically.

With regard to the latter point, “you cannot measure the existence of something,” that seems patently false to the point of absurdity. At a very basic level, we can measure existence in terms of a count variable; to measure the existence of a black swan, count = 0 would indicate no black swans (in the sample), while count >= 1 would indicate the existence of black swans. Depending on the complexity of the analysis, one could alternatively test for certain properties of the object in question. That would be, for example, how scientists at the Large Hadron Collider have been searching for the existence of the Higgs boson. They test for properties of a particle to determine whether it can be classified as the Higgs. Scientists in virtually every field often present evidence for the existence of a phenomenon.

Now, granted, I take the point that God’s existence may not be directly measurable. In this sense, you should take the concepts of the null hypothesis and evidence as an analogy more than a practical process. But some claims regarding God’s existence are directly measurable, such as miracle claims, responses to prayer, etc.

As a final note, I would like to mention that this post is several years old at this point. Were I to write it again, I would probably instead talk about Bayesian statistics and hypothesis testing, and how evidence could serve to shift prior probabilities in favour of the existence of God compared to competing hypotheses. I think that would serve as a better analogy than null hypothesis significance testing. But the basic principles regarding a) the burden of proof and b) aligning beliefs to the evidence still apply under either framework.

Shane Turner

I find this to be interesting I took statistics years ago and I loved it. In the hypothesis testing you either reject the null hypothesis or do not reject the null hypothesis. It started to come back to me when I started questioning the existence of a god or deity. To me there is not sufficient evidence to conclude that a god or deity exists. I do not reject the null hypothesis in the case of a god. Rejecting the null would be sufficient evidence to conclude a god or deity exists. This is why I consider myself to be agnostic or atheist. I claim to not know if a god or deity exists and have no sufficient evidence to conclude that a god or deity exists. Just because you do not reject the null necessarily means you whole heartedly accept the null hypothesis. It just says there is no sufficient evidence. It is like finding a person not guilty in a court of law. There still may be a small probability that they are guilty but there is no sufficient evidence to prove it. Same with the idea of a god or deity.

CAt

This is a beautifully laid out argument. However, I do not see the applicability of applying science in the form of the null hypothesis or the scientific method – things which were formulated to measure physical concepts/beliefs – to test a metaphysical concept/belief. How does one find evidence for a non-physical entity by looking through evidence in the physical world? Physical investigation and metaphysical investigation (mysticism) are two ways of viewing different forms of reality and are equally valid in acquiring two very different forms of knowledge that, nevertheless, can compliment and inform one another, as they have done down the centuries.

Leave a Reply