7.4.7. How can we make multiple comparisons?

7. Product and Process Comparisons
7.4. Comparisons based on data from more than two processes

7.4.7. How can we make multiple comparisons?

What to do after equality of means is rejected

Multiple Comparison test procedures are needed

When processes are compared and the null hypothesis of equality (or homogeneity) is rejected, all we know at that point is that there is no equality amongst them. But we do not know why this is so.

Questions concerning the reason for the rejection of the null hypothesis arise in the form of:

"Which mean(s) or proportion (s) differ from a standard or from each other?"
"Does the mean of treatment 1 differ from that of treatment 2?"
"Does the average of treatments 1 and 2 differ from the average of treatments 3 and 4?"

One popular way to investigate the cause of rejection of the null hypothesis is a Multiple Comparison Procedure. These are methods which examine or compare more than one pair of means or proportions at the same time.

Note: Doing pairwise comparison procedures over and over again for all possible pairs will not, in general, work. This is because the overall significance levels are no longer as specified for a single pair comparison.

The ANOVA uses the F test to determine whether there exists a significant difference among treatment means or interactions. In this sense it is a preliminary test that informs us if we should continue the investigation of the data at hand.

If the null hypothesis (no difference among treatments or interactions) is accepted, there is an implication that no relation exists between the factor levels and the response. There is not much we can learn, and we are finished with the analysis.

When the F test rejects the null hypothesis, we usually want to undertake a thorough analysis of the nature of the factor level effects.

Previously, we discussed several procedures for examining particular factor level effects. These were

These types of investigations should be done on combinations of factors that were decided on in advance of observing the experimental results, or else the confidence levels are not as specified by the procedure. Also, doing several comparisons might change the overall confidence level (see note above). This can be avoided by carefully selecting contrasts to investigate in advance and making sure that:

the number of such contrasts does not exceed the number of degrees of freedom between the treatments

only orthogonal contrasts are chosen.

However, there are also several powerful multiple comparison procedures we can use after observing the experimental results.

Tests on Means after Experimentation

If the decision on what comparisons to make is withheld until after the data are examined, the following procedures can be used:

Tukey's Method to test all possible pairwise differences of means to determine if at least one difference is significantly different from 0.
Scheffé's Method to test all possible contrasts at the same time, to see if at least one is significantly different from 0.
Bonferroni Method to test, or put simultaneous confidence intervals around, a pre-selected group of contrasts

Multiple Comparisons Between Proportions

When we are dealing with population proportion defective data, the Marascuilo procedure can be used to simultaneously examine comparisons between all groups after the data have been collected.