Meta-Analysis of Multiple Simulation-Based Experiments

October, 2013
IDA document: D-5038
FFRDC: Systems and Analyses Center
Type: Documents
Division: Strategy, Forces and Resources Division
David S. Alberts, Fran├žois Bernier, Paul Pearce See more authors
Computer simulations facilitate the study of real world phenomena by providing safe, controlled, flexible, and repeatable experimental environments at costs far lower than other options. Both the validity of the simulation model and the scope of its validity determine the degree to which the findings can be generalized and applied to real world situations and problems. In practice, no single model captures all of the important aspects of a phenomena of interest, nor is applicable over a wide set of missions and circumstances. Thus, effectively utilizing a variety of models in a prospective meta-analysis (a set of common hypotheses and controllable variables and comparable metrics) offers the opportunity to improve validity and extend the findings to a broader range of real world situations. NATO SAS-085, a research group exploring C2 Agility and Requisite Maturity, conceived and developed an international meta-analysis approach for studying various aspects related to C2 Agility from multiple simulation-based experiments. This paper presents the methodology they employed which was inspired by the prospective meta-analysis domain. The challenges that arose from differences among these experiments, differences in the ways C2 Approaches were instantiated, and differences in the measures of success and in the conditions they considered are discussed.