Metamodeling Techniques for Verification and Validation of Modeling and Simulation Data

September, 2022
IDA document: P-33230
FFRDC: Systems and Analyses Center
Type: Documents
Division: Operational Evaluation Division
John T. Haman, Curtis G. Miller See more authors
Modeling and simulation (M&S) outputs help the Director, Operational Test and Evaluation (DOT&E), assess the effectiveness, survivability, lethality, and suitability of systems. To use M&S outputs, DOT&E needs models and simulators to be sufficiently verified and validated. The purpose of this paper is to improve the state of verification and validation by recommending and demonstrating a set of statistical techniques—metamodels, also called statistical emulators—to the M&S community. The paper expands on DOT&E’s existing guidance about metamodel usage by creating methodological recommendations the M&S community could apply to its activities. For a deterministic, discrete response variable, we recommend using a nearest neighbor or decision tree model. For a deterministic, continuous response variable, we recommend Gaussian process interpolation. For a stochastic response variable, we recommend a generalized additive model. We also present a set of techniques that testers can use to assess the adequacy of their metamodels. We conclude with a notional example (a paper plane simulation) that demonstrates the recommended techniques. Finally, we include supplemental software written in R that readers can use to reproduce the outputs from this paper.