Which meta model to choose?

There is little doubt that the polynomial-based response surfaces are the most robust, especially for sequential optimization methods. A negative aspect is the fact that the user has to choose the order of the polynomial and a greater possibility exists for bias error of a nonlinear response. They are also, in most cases, not suitable for updating in sequential methods. Linear approximations may only be useful within a certain subregion and therefore quadratic polynomials or other higher order approximations such as RBF networks may be required for greater global accuracy. However the linear SRSM method has proved to be excellent for sequential optimization and can be used with confidence [1][2][3].

RBF Networks appear to be generally the best of the neural networks metamodels. They have the following

advantages:

• Higher prediction accuracy due to built-in cross validation. Although FF networks may appear more accurate due to a smaller fitting error (RMSE), their prediction error is generally larger than that of
• RBF networks. An appealing plot of predicted vs. computed responses showing the training points should not be construed as representing a higher accuracy.
• Higher speed due to their linear nature. When sizable FF committees (e.g. with 9 members) are used they may be vastly more expensive to construct than RBF networks. This is true especially for a relatively small number of variables.
• Relative independence of the calculation time with respect to the number of functions (there is a slight overhead). The user does not have to be as careful with limiting the number of responses.

FF Neural Networks function well as global approximations and no serious deficiencies have been observed when used as prescribed in Section 7.3 (LS-OPT Manual). FF networks have been used for sequential optimization [3] and can be updated during the process. Neural Networks are sometimes better than RBF  etworks for smooth problems.

Although the literature seems to indicate that Kriging is one of the more accurate methods [4], there is evidence of Kriging having fitting problems with certain types of experimental designs [5]. Kriging is very sensitive to  noise, since it interpolates the data [6]. The authors of this manual have also experienced fitting problems with non-smooth surfaces (Z(x) observed to peak at data points) in some cases, apparently due to large values of Θ that may be due to local optima of the maximum likelihood function. The model construction can be very time consuming [30] (also experienced with LS-OPT). Furthermore, the slight global altering of the Kriging surface due to local updating has also been observed [3]. Work is under way to improve the Kriging implementation in LS-OPT.

Reference [3] compares the use of three of the metamodeling techniques for crashworthiness optimization. This paper, which incorporates three case studies in crashworthiness optimization, concludes that while RSM, NN and Kriging were similar in performance, RSM and NN were shown to be the most robust for this application. RBF networks were not available at the time of that study.

A recent study [7] which focuses on the accuracy comparison for FF neural networks and RBF networks for different types of optimization strategies concluded that RBF and FF metamodels are largely similar in terms of the accuracy of a large number of checkpoints. A number of analytical as well as crashworthiness design examples were considered. As mentioned earlier, RBF networks have the speed advantage.

1. Stander, N., Craig, K.J. On the robustness of a simple domain reduction scheme for simulation-based optimization, Engineering Computations, 19(4), pp. 431-450, 2002.
2. Stander, N., Reichert, R., Frank, T. 2000: Optimization of nonlinear dynamic problems using successive linear approximations. AIAA Paper 2000-4798.
3. Stander, N., Roux, W.J., Giger, M., Redhe, M., Fedorova, N. and Haarhoff, J. Crashworthiness optimization in LS-OPT: Case studies in metamodeling and random search techniques. Proceedings of the 4th European LS-DYNA Conference, Ulm, Germany, May 22-23, 2003. (Also www.lstc.com).
4. Simpson, T.W. A Concept Exploration Method for Product Family Design. Ph.D. Thesis, Georgia Institute of Technology, 1998.
5. Xu, Q-S., Liang, Y-Z., Fang, K-T., The effects of different experimental designs on parameter
   estimation in the kinetics of a reversible chemical reaction. Chemometrics and Intelligent
   Laboratory Systems, 52, pp. 155-166, 2000.
6. Jin, R., Chen, W. and Simpson, T.W. Comparative studies of metamodeling techniques under
   multiple modeling criteria, AIAA Paper, AIAA-2000-4801.
7. Stander, N. Goel, T. Metamodel sensitivity to sequential sampling strategies in crashworthiness
   design. Proceedings of the 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization
   Conference,Victoria, British Columbia, Canada, Sep 10-12, 2008. Submitted.

Please refer to FAQ What are the fundamental differences between RBF and FFNN?