4.3.4. I've heard some people refer to "optimal" designs, shouldn't I use those?

4. Process Modeling
4.3. Data Collection for Process Modeling

4.3.4. I've heard some people refer to "optimal" designs, shouldn't I use those?

Classical Designs Heavily Used in Industry

The most heavily used designs in industry are the "classical designs" (full factorial designs, fractional factorial designs, latin square designs, Box-Behnken designs, etc.). They are so heavily used because they are optimal in their own right (e.g., orthogonality) and have served superbly well in providing efficient insight into the underlying structure of industrial processes.

Reasons Classical Designs May Not Work

Cases do arise, however, where the tabulated classical designs do not cover the practical situation that the analyst must deal with; that is to say, user constraints preclude the existence of using tabulated classical designs because such classical designs simply do not exist to accomodate such user constraints. Such constraints include:

Limited maximum number of runs:
User constraints in budget and time may dictate a maximum allowable number of runs which are too small or too "irregular" (e.g., "13") to be accomodated by classical designs--even fractional factorial designs.
Impossible factor combinations:
The user may have some factor combinations which are impossible to run. Such combinations may at times be specified (to maintain balance and orthogonality) as part of a recommeded classical design. If the user simply omits this impossible run from the design, the net effect may be a reduction in the quality and optimaltiy of the classical design.
Too many levels:
The user may have too many levels for too many factors, with the net result that the case is not covered under the tabulated classical designs.
4. Complicated underlying model:
The user may be assuming an underlying model that is too complicated (or too non-linear), and so may be beyond the collection which underlie classical designs.

What to Do If Classical Designs Do Not Exist?

The bottom line is that if user constraints are such that classical designs do not exist to accomodates such constraints, then what is the user to do?

The previous section's list of design criteria (capability for the primary model, capability for the alternate model, minimum variation of estimated coefficients, etc.) is a good passive target to aim for in terms of desirable design properties, but provides little in terms of an active formal construction methodlogy to actually generate a design.

Common Optimality Critierieon

To satisfy this need, an "optimal design" methodology has been developed to generate an actual design when user constraints preclude the use of tabulated classical designs. Optimal designs may be optimal in many different ways, and what may be an optimal design according to one criterion may be sub-optimal for other criteria. Competing criteria has led to a literal alphabet-soup collection of optimal design methodologies. The four most popular ingredients in that soup are

D-optimal designs: minimize the generalized variance of the parameter estimates.

A-optimal designs: minimize the average variance of the parameter estimates.

G-optimal designs: minimize the maximum variance of the predicted values.

V-optimal designs: minimize the average variance of the predicted values.

Mathematical Definitions of Optimal Design

Mathematically, if X refers to the design matrix, and if X'X is referred to as the design's information matrix, then formally

D-optimal designs:	maximize *\|X'X\|*, that is, maximize the determinent of the information matrix.
A-optimal designs:	minimize tr((*X'X*)^-1), that is, minimize the trace of the inverse of the information matrix.
G-optimal designs:	minimize {max *X'(X'X)^-1X* ²)}, that is, minimize the maximum of the above over a specified set of design points.
V-optimal designs:	minimize ..., that is, minimize the average prediction variance over a specified set of design points.

Need 1: a Model

The motivation for optimal designs is the practical constraints that the user has. The advantage of optimal designs is that they do provide a reasonable design-generating methodology where no other mechanism exists. The disadvantage of optimal designs is that they require a model from the user. The user may not have this model.

All optimal designs are model-dependent, and so the quality of the final engineering conclusions which result from the ensuing design, data, and analysis is dependent on the correctness of the analyst's assumed model. For example, if nature is generating responses from a cubic model, and the analyst assumes a linear model and uses the corresponding optimal design to generate data and perform the data analysis, then the final engineering conclusions will be flawed and invalid. Hence one price for getting an in-hand generated design is the designation of a model. All optimal designs need a model; without a model, the optimal design-generation methodology cannot be used, and general design principles must be reverted to.

Need 2: a Candidate Set of Points

The other price for using optimal design methodology is a user-specified set of candidate points. Optimal designs will not generate the best design points from some continuous region--that is too much to ask of the mathematics. Optimal designs will generate the best subset of n points from a larger superset of candidate points. The user must specify this candidate set of points. Most commonly, the superset of candidate points is the full factorial design over a fine-enough gridding of the factor space that the analyst is comfortable with. If the gridding is too fine, and the resulting superset overly large, then the optimal design methodology may prove computationally challenging.

Optimal Designs are Computationally Intensive

The optimal design-generation methodology is computationally intensive. Some of the designs (e.g., D-optimal) are better than other designs (such as A-optimal and G-optimal) in regard to efficiency of the underlying search algorithm. Like most mathematical optimization techniques, there is no iron-clad guarantee that the results from the optimal design methodology are in fact the true optimum. However, the results are usually satisfactory from a practical point of view, and are far superior than any ad hoc designs.

For further details about optimal designs, the analyst is referred to Montgomery (199x).