Jim Longshore and Dick Scott collaborated on this blog to provide a sample of techniques taught during the MBB curriculum. Additional information on our upcoming Master Black Belt wave can be found here.
Scenario:
You just finished your Design of Experiments and reduced the terms to an acceptable model. The problem is that you ran only one replicate and your nervous about the results. The reason for concern is there was one setting that provided such an outlier that it doesn’t feel right. Short of rerunning the experiment there is no way to validate the results, or is there?
As the title suggests there are times when experimentation provides data, but there is frustration in believing the interpretation. As the analysis is underway the data just feels messy and short of rerunning the experiment, we’ve had little recourse until now.
SBTI offers:
The Rank Transform Rule: Always rank transform the response values and analyze both the transformed and untransformed response values. If the conclusions from the two analyses differ, use the results for the ranked response values.
At SBTI we’ve taught Master Black Belts for 20+ years on steps to approach experimentation. One common problem we face is that it is very difficult to spot a bad response value in a DOE with only 1 trial per treatment, unless prior to the experiment you had an approximate idea of what each value should be according to historical knowledge. Suspicious results can arise when:
- An outlier is seen in a small experiment (<= 16 runs)
- A residual variance appears to be non-constant
- Or, other possible abnormalities such as odd-looking normal plots.
Rank Transform is a method to validate single replicate DOEs!
The rank transform is a nonparametric method of transforming data by ordering values from low to high in ranked order. Each original response value is replaced by its assigned rank. (Ties are assigned the average of the rank values for the values in that tie).
Example:
| Response | 5 | 2 | 25 | 55 | 55 | 150 | 2000 |
| Ranked | 2 | 1 | 3 | 4.5 | 4.5 | 6 | 7 |
This transform has several useful properties, including constant variance. It is also robust to outliers.
The rank transform works because it deals with a ‘bad’ observation. Evaluation of an 8-run half-fraction factorial has the following consequences:
- Each observation contributes 1/8 of the data used to calculate each effect
- Variance of the residuals can be distorted, creating potentially false conclusions
- Significance of marginal effects can change
- Higher order terms may become more significant than main effects
Recognize that some factors may drive both the mean and the variance of the response, which can lead to false conclusions. To check this possibility, one should plot residuals versus each factor.
An example of applying Rank Transform:
- A half-fraction of a Factorial Design with a simulated response (random Normal, mean = 0, SD = 1)
- Perform initial Normal plot of Effects from NID(0,1) Data
Remember these effects come from random data!
- Copy the response column into a second column, and label it “ModY”, so that we can distinguish modified values from original values.
- Replace the response value in row 7 of the ModY column by the value 8
- We are deliberately distorting a value that was originally close to 0 to a value that is 8 SD’s from 0!
- Rerun the Normal Plot of Calculated effects with ModY using the same terms.
Notice we have lost the significant effects because of the “bad” data point and it appears to be split into 2 groups about 0 (evidence of an outlier)
- Perform a Rank transform of ModY and create a Normal Plot of the calculated effects.
- Analyze the design using the Rank Transformed ModY values
- Leave all the predictor terms in the model.
Notice that almost all of the original significant effects are back (only missing one)
This shows that Rank Transform Analysis is robust to outliers.
In designed experiments check the pareto plot or normal plot of calculated effects to see how reliable the estimate of error is. If either of these plots is unusual, consider one point as being potentially in error.
A good final check on the rank order is to store the Cook’s D values. These can evaluate and highlight potentially influential data points using a simple individual chart. Values falling outside of the control limit are typically highly influential on the ranked transform model and should warrant further investigation.
Using the terms from the reduced model that created the above output on our original ModY, provides the following Cooks D Individual Chart.
Remember to also examine residuals
- Try removing any observation having a large residual and treating the corresponding point as a missing observation.
Examine DOE notes carefully and consider repeating any run that produced a suspicious response value.
Finally, review the estimate of error obtained from the designed experiment and compare it to the value from existing process control charts, or from historical variation of the process.
We often are faced with messy data. This is especially critical when single replicated experiments are run. Performing a rank transform of the response and evaluating the Cooks via control charts will help isolate influential outliers and provide a path forward for analysis.
If you would like to request a listening session for your business, you can contact me, Jim Longshore, by email at jlongshore@sbtimail.com.