Statistical Design & Analysis Challenges in Defense Testing

Before the DoD acquires any major new capability, that system must undergo realistic testing in its intended environment with military users. The complex, data-limited, highly variable nature of the t est environment presents many unique statistical challenges. The set of conditions in which a system will operate is typically large, and important variables are often uncontrollable during test, making rigorous experimental design a challenge. Data sets obtained from tests are almost always messy. Issues such as lurking variables, sma ll and unbalanced sample sizes, and ordinal responses necessitate creative and sometimes sophisticated data analysis approaches. This document examines some of these defense-testing situations in detail and discusses how statisticians in the test and evaluation community have approached associated design and analysis challenges.