Vetting Custom Scales - Understanding Reliability, Validity, and Dimensionality

December, 2018
IDA document: D-9168
FFRDC: Systems and Analyses Center
Type: Documents
Division: Operational Evaluation Division
Dr. Heather Wojton, Dr. Stephanie Lane See more authors
For these situations in which an empirically vetted scale does not exist or is not suitable, a custom scale may be created. This document presents a com prehensive process for establishing the defensible use of a custom scale. At the highest level , this process encompasses (1) establishing validity of the scale, (2) establishing reliability of the scale, and (3) assessing dimensionality, whether intended or unintended, of the scale. First, the concept of validity is described, including how validity may be established using operators and subject matter experts. The concept of scale reliability is described, with guidelines for computing, interpreting, and using results to inform potential modifications to a custom scale. Next, a method for investigating the dimensionality of a scale, exploratory factor analysis, is described, along with a walkthrough of software implementation and results. Finally, confirmatory factor anal ysis, a technique for testing a priori hypotheses about dimensionality, is presented