Introduction

(The lady tasting tea)} Dr. Muriel Bristol, a colleague of Fisher's, claimed that when drinking tea she could distinguish whether milk or tea was added to the cup first (she preferred milk first). To test her claim, Fisher asked her to taste eight cups of tea, four of which had milk added first and four of which had tea added first.

For each cup, we record the order of actual pouring and what the lady says the order is. We can summarize the result by a table like this:

	Guessed milk	Guessed tea	Total
Really milk	\[n_{11}\]	\[n_{12}\]	\[n_{1+} = 4\]
Really tea	\[n_{21}\]	\[n_{22}\]	\[n_{2+} = 4\]
Total	\[n_{+1} = 4\]	\[n_{+2} = 4\]	\[n = 8\]

Here \(n\) is the total number of cups of tea made. The number of cups where milk is poured first is \(n_{1+}\) and the lady classifies \(n_{11}\) of them as tea first. Ideally, if she can taste the difference, the counts \(n_{12}\) and \(n_{21}\) should be small. On the other hand, if she can't really tell, we would expect \(n_{11}\) and \(n_{22}\) to be about the same. If the lady had no idea and guessed at random \((H_0)\), then \(n_{11}\) follows a hypergeometric distribution:

\[ P(n_{11} = t) = \frac{{n_{1+} \choose t} {n_{2+} \choose n_{+1}-t}}{{n \choose n_{+1}}} \]

Under \(H_0\), the lady would be correct for all cups with probability

\[ P\left(n_{11}=4\right)=\frac{{4\choose 4}{4\choose 0}}{{8\choose 4}}=\frac{\frac{4 !}{4 ! 0 !} \frac{4 !}{0 ! 4 !}}{\frac{8 !}{4 ! 4 !}}=\frac{4 ! 4 !}{8 !}=\frac{1}{70}=0.014\]

Fisher concluded that since this probability was less that 0.05 that it was “statistically significant”. Notice the asymmetry in this description: only a null hypothesis is actually specified (i.e. there is no alternative hypothesis – it is in some sense implicit), i.e. the null hypothesis is often special. Furthermore, there is an arbitrary choice of a cut-off 0.05 below which we declare something is significant.