If the samples are reasonably large we can use the normal approximation to the binomial to develop a test similar to testing whether two normal means are equal.

Let sample 1 have x₁ defects out of n₁ and sample 2 have x₂ defects out of n₂. Calculate the proportion of defects for each sample and the z statistic below:

Compare z to the normal z_a/2 table value for a 2-sided test. For a one sided test, assuming the alternative is p₁>p₂, compare z to the normal z_a table value.

Case 2: An Exact Test for Small Samples

The Fisher Exact Probability Test is an excellent nonparametric technique for analyzing discrete data (either nominal or ordinal), when the two independent samples are small in size. It is used when the results from two independent random samples fall all into one or the other of two mutually exclusive classes (i.e. defects vs good, or successes vs failures).

In other words, every subject in both group obtains one of two possible scores. These scores are represented by frequencies in a 2x2 contingency table. The following discussion, using a 2x2 contingency table illustrates how the test operates.

We are working with two independent groups,  such as experiments and controls, males and females, the Chicago Bulls and the New York Knicks, etc.

	-	+	Total
Group I	A	B	A+B
Group II	C	D	C+D
Total	A+C	B+D	N

The column headings, here arbitrarily indicated as plus and minus, may be of any two classifications, such as: above and below the median, passed and failed, Democrat and Republican, agree and disagree, etc.

Fisher's test determines whether the two groups differ in the proportion with which they fall into the two classifications. For the table above, the test would determine whether Group I and Group II differ significantly in the proportion of plusses and minuses attributed to them.

The method proceeds as follows:

The exact probability of observing a particular set of frequencies in a 2 × 2 table, when the marginal totals are regarded as fixed, is given by the hypergeometric distribution

But the test does not just look at the observed case. If needed, it also computes the probability of more extreme outcomes, with the same marginal totals.

This will become clear in the next illustrative example. Consider the following set of 2 x 2 contingency tables:

Observed Data	More extreme outcomes with same marginals
(a)	(b)	(c)
2 5 7 3 2 5 5 7 12	1 6 7 4 1 5 5 7 12	0 7 7 5 0 5 5 7 12

Table (a) shows some observed frequencies and tables; (b) and (c) show the two more extreme distributions of frequencies, which could occur with the same marginal totals 7, 5. Given the observed data in table (a) , we wish to test the null hypothesis at, say, a = .05.

Applying the previous formula to tables (a), (b) and (c) we obtain

The probability associated with  the occurrence of values as extreme as the observed results under H_o. is given by adding these three p's:

So p = .31040 is the probability that we get from Fisher's test. Since .31040 is larger than a we cannot reject the null hypothesis.

Tocher (1950) showed that a slight modification of the Fisher test makes it a more useful test. He starts by isolating the probability of all cases more extreme than the observed one. In this example that is

Now, if this probability is larger than a, we cannot reject H_o. But if this probability is less thana, while the probability that we got from Fisher's test is greater than a (as is the case in our example) then Tocher advises to compute the following ratio:

The test is a one-tailed test. For a two-tailed test, the value of p obtained from the formula must be doubled.

A difficulty with the Tocher procedure is that the someone else analyzing the same data would draw a different randon number and possibly make a different decision about the validity of H_o.