A step by step analysis of a fractional factorial "catapult" experiment

This experiment was conducted by a team of students on a catapult – a table-top wooden device used to teach design of experiments and statistical process control.  The catapult has several controllable factors, and a response easily measured in a class room setting. It has been used for over 10 years in hundreds of classes Below is a small picture of a catapult that can be opened to view a larger version.

The experiment has five factors which might affect the distance the golf ball travels
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Design matrix and responses (in run order)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

You can download the data in a spreadsheet

Purpose: To determine the significant factors that affect the distance the ball is thrown by the catapult, and to predict the settings required to hit 3 different target values (30, 60 and 90 inches). 

Response Variable: The distance in inches from the front of the catapult to the spot where the ball lands. The ball is a plastic golf ball. 

Number of observations:  20 (a 2^5-1 resolution V design with 4 center points)

Variables:

1) Response Variable Y = distance 

2) Factor 1 = band height (height of the pivot point for the rubber bands – levels were 2.25, 3.5 and 4.75 inches)

3) Factor 2 = start angle (location of the arm when the operator releases – starts the forward motion of the arm – levels were 0, 10 and 20 degrees)

4) Factor 3 = rubber bands (number of rubber bands used on the catapult – levels were 1 and 2 bands)

5) Factor 4 = arm length (distance the arm is extended – levels were 0, 2, and 4 inches)

6) Factor 5 = stop angle (location of the arm where the forward motion of the arm is stopped and the ball starts flying – levels were 45, 62 and 80 degrees)

Readers who want to analyze this experiment themselves may download an Excel spreadsheet catapult.xls or a JMP spreadsheet capapult.jmp.

Note that 4 of the factors are continuous, and one – number of rubber bands –  is discrete.  The four center points are at both levels of the number of bands.  Runs 7 and 19 are with one rubber band, and the center of the other factors, while runs 2 and 13 are with two rubber bands and the center of the other factors. 

After analyzing the 20 runs and determining factor settings needed to achieve predicted distances of 30, 60 and 90 inches, the team was asked to conduct 5 confirmatory runs at each of the these derived settings.

Analysis of the Experiment

The experimental data will be analyzed using SAS JMP 3.2.6 software. 

The resolution V design can estimate main effects and all 2-factor interactions 

We start by plotting the data several ways to see if any trends or anomalies appear that would not be accounted for by the models. 

The distribution of the response is below: 
 

We can see the large spread of the data, and a pattern to the data that should be explained by the analysis. 

Next we look at the responses versus the run order to see if there might be a sequence component. The four highlighted points are the center points in the design. Recall that run 2 and 13 had 2 rubber bands, and run 7 and 19 had 1 rubber band. There may be a slight aging of the rubber bands in that the second center point was a little shorter than the first for each pair. 
 

Next look at the plots of responses sorted by factor columns. 
 

Several factors appear to change the average response level, but most have a large spread at each of the levels which might indicate possible interactions. 

Step 2: Create the theoretical model

With a resolution V design, we are able to estimate all the main effects, and all two factor interactions cleanly – without worrying about confounding.  Therefore the model will have 16 terms – the intercept term, the 5 main effects, and the 10 two factor interactions. 

JMP output after fitting the trial model (all main factors and 2-factor interactions)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Use  p-values to help select significant effects, and also use a normal plot 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

A refit using just the effects that appear to matter
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

R squared is OK and there is no significant model "lack of Fit"
 
 
 
 
 
 
 
 

Note we have used the orthogonally coded columns for the analysis, and have abbreviated the factor names as follows: 

 Bheight = band height
 Start = start angle
 Bands = number of rubber bands
 Stop = stop angle
 Arm = arm length.

The model has a good R squared value, but the fact that R squared adjusted is smaller indicates that we might have some terms in our model that are not significant. Scanning the column of p-values (labeled Prob>|t| in the JMP output) for small values shows 5 significant effects at the 0.05 level, and another 1 at the 0.10 level. 

The normal plot of effects is a useful graphical tool to determine significant effects.  The graph below shows that there are 9 terms in the model that can be assumed to be noise. That would leave 6 terms to be included in the model as significant.  While the parameter estimates table shows a p-value of 0.086 for the interaction of bands and arm, the normal plot suggests we treat this interaction as significant. 
 
 

Remove the insignificant terms from the model and refit to produce the following output: 
 
 

The R squared and R squared adjusted values are acceptable. The ANOVA table shows us that the model is significant, and the Lack of Fit table shows that there is no significant lack of fit.

The Parameter estimates table is below.
 
 

The residual graphs are not ideal, although the model passes "lack of fit" quantitative tests 
 
 
 
 

We should test that the residuals are centered at zero, normally distributed, do not change over time, and have equal variances.  First we create a histogram of the residual values. 
 
 

The residuals do appear to be centered at zero and normally distributed. 

Next we plot the residuals versus the predicted values. 
 
 

There does not appear to be a pattern to the residuals based on changing predictions.  One observation about the graph is that the model does not appear to predict well at shorter distances. In fact, run number 10 had a measured distance of 8 inches, but the model predicts a –11 inches, giving a residual of 19.  The fact that the model predicts an impossible negative distance is an obvious shortcoming of the model. We may not be successful at predicting the catapult settings required to hit a distance less than 25 inches. 

Next we plot the residual values versus the run order of the design. The highlighted points are the center point values, and they do appear to have a changing residual as the experiment was conducted.  Recall that run number 2 and 13 had two rubber bands, while run number 7 and 19 had only one rubber band. 
 
 

Next we look at the residual values versus each of the factors.
 

Most of the residual graphs versus the factors appear to have a slight "frown" on the graph (higher residuals in the center). This may indicate a lack of fit, or sign of curvature at the center point values. The Lack of Fit table, however, indicates that the lack of fit is not significant. 

At this point, since there are several unsatisfactory features about the model we have fit and the resulting residuals, we should consider whether a simple transformation of the response variable (Y = "Distance") might improve the situation. 

There are at least two good reasons to suspect that using the logarithm of distance as the response might lead to a better model.

1. A linear model fit to log Y will always predict a positive distance for any possible combination of X factor values.
2. Physical considerations suggest a realistic model for distance might require quadratic terms, since gravity plays a key role - taking logarithms often reduces the impact of non-linear terms.

A simpler model with just main effects has a satisfactory fit

Examining the p-values of the 16 model coefficients, only the intercept and the 5 main effect terms appear significant. Refitting the model with just these terms yields 
 
 

This is a simpler model than previously obtained in Step 3 (no interaction term), with larger values for R squared and R squared adjusted. All the parameters are highly significant and there is no quantitative indication of "lack of fit". 

We next look at the residuals for this new model fit.

A normal plot and histogram of the residuals shows no problems.
 
 

A plot of the residuals versus the predicted log Y values looks reasonable, although there might be a tendency for the model to overestimate slightly for high predicted values.

Residuals plotted versus run order again show a possible slight decreasing trend (rubber band fatigue?).

Next we look at the residual values versus each of the factors.

These plots still appear to have a slight "frown" on the graph (higher residuals in the center). However, the model is generally an improvement over the previous model and will be accepted as possibly the best that can be done without conducting a new experiment designed to fit a quadratic model.

"Confirmation" runs were successful

The software used for this analysis (JMP 3.2.6) has an option called the "Prediction Profiler" that can be used to derive settings that will yield a desired predicted log distance value. The top graph in the figure below shows the direction and strength of each of the main effects in the model. Using log 30 = 3.401 as the target value, the Profiler allows us to set up a "Desirability" function that gives 3.401 a maximum desirability value of 1 and values above or below 3.401 have desirabilities that rapidly decrease to 0. This is shown by the desirability graph on the right (see the figure below).

The next step is to set "bands" to either -1 or +1 (this is a discrete factor) and move the values of the other factors interactively until a desirability as close as possible to 1 is obtained. In the figure below, a desirability of .989218 was obtained, yielding a predicted log Y of 3.399351 (or a distance of 29.94). The corresponding (coded) factor settings are: bheight = 0.17, start = -1, bands = -1, arm = -1 and stop = 0. 
 

Repeating  the profiler search for a Y value of 60 (or log Y = 4.094) yielded the figure below where a log distance value of 4.094121 is predicted (a distance of 59.99) for coded factor settings of  bheight = 1, start = 0, bands = -1, arm = .5 and stop = .5.
 

Finally, we set log Y = log 90 = 4.4998 and obtain (see the figure below) a predicted log distance of 90.20 when bheight = -0.87, start = -0.52, bands = 1, arm = 1, and stop = 0. 
 

In the confirmatory runs that followed the experiment, the team was successful at hitting all 3 targets, but did not hit them all 5 times.

NOTE: The model discovery and fitting process, as illustrated in this analysis, is often an iterative process.