Empirical Signal-to-Noise Ratios from Operational Test Data

August, 2014
IDA document: D-5243
FFRDC: Systems and Analyses Center
Division: Operational Evaluation Division
Matt R. Avery See more authors
Statistical power is a common metric for assessing experimental designs. While this metric depends on many factors, one of the most critical is the expected effect size of relevant factors and the relative noise expected in the data. Together, these values are summarized as the signal-to-noise ratio (SNR). Software packages like JMP 10 and Design Expert use SNR as a critical component in power calculations, and by general “rule of thumb,” values such as 0.5, 1, and 2 are used. However, it is not clear that these values represent the true spectrum of likely outcomes from operational test data. Operational testing is the final phase prior to fielding in the DOD acquisition process for new systems. Because of the operational realism strived for in such testing, there are often many sources of uncontrolled variation, making it difficult to plan an appropriate test based on the SNR. In this briefing, we summarize observed SNRs from a wide spectrum of operational tests and offer suggestions for the use of SNR in operational test design.