Test Design Challenges in Defense Testing

Before the Department of Defense acquires any major new system capability, that system must undergo realistic testing in its intended environment with military users. These evaluations increasingly rely on computer modeling and simulation (M&S) to supplement live testing. Validating M&S tools requires rigorous data collection and analysis strategies to ensure their output adequately represents reality. Classical experimental design techniques have been widely adopted across the defense community for planning live tests, but gold standard computer experiment techniques, such as space filling designs and Gaussian process emulators, are much less commonly used for M&S studies. There is currently no consensus on how to incorporate the unique demands of defense testing into the existing academic framework for M&S. This briefing provides an overview of operational testing and discusses defense applications of, and key differences between, classical and space-filling designs. It presents several challenges (and possible solutions) associated with implementing space-filling designs and associated analyses in the defense community.