2017 Larry D. Welch Award for Best External Publication

This year’s winner of the Larry D. Welch Award for Best External Publication is 

"Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review," by Science and Technology Division (STD) researcher Dexter Fletcher and STD consultant James Kulik. Their paper was published in the Review of Educational Research, March 2016.

This highly selective, peer-reviewed journal has ranked for many years among the top journals in the field of education for impact and citations (e.g., Reuters ranked it 1 out of 219 education journals in 2013). The paper has already been cited by one book and 18 articles or conference papers. 

Dr. Fletcher received his PhD in educational psychology from Stanford University. Prior to joining IDA in 1986, Dr. Fletcher taught at the Universities of Illinois and Oregon, and held positions at the Navy Personnel Research and Development Center, DARPA, and the Army Research Institute.

Dr. Kulik received his PhD in psychology from the University of California at Berkeley. He held teaching and research positions at the University of Michigan from 1969 to 2010. He has been an IDA consultant since 2011.  

The authors’ paper describes a meta-analysis of findings from 50 controlled evaluations of computer-based, intelligent tutoring systems (ITSs). The basic finding is that ITSs can be very effective, raising median test scores by 0.66 standard deviations – or from the 50th to the 75th percentile – in comparison with control groups receiving conventional classroom instruction. Moreover, the evaluations show that ITSs typically raise student performance beyond that achieved with other forms of computer tutoring or with human tutors.

Two of the 50 evaluations reviewed in the winning paper were IDA projects conducted by Dr. Fletcher , in combination with STD’s John Morrison in one case. The IDA analyses focused on the DARPA-developed Digital Tutor for training Navy electronics technicians. They found the DARPA system to be far more effective relative to Navy schoolhouse instruction than the median gain from the 50 diverse applications cited in the journal article. 

Dr. Fletcher deserves much credit for taking knowledge gained over years of sponsored research at IDA and using that knowledge to develop an exceptional article published in a highly respected journal. The article also serves as a reminder of the significant potential payoffs from intelligent tutoring systems in both defense and non-defense training and educational applications, a finding that Dr. Fletcher has been communicating tirelessly within the national security community.

Indeed, in a later IDA project for the Department of Veterans Affairs, not yet complete when this article was written, Dr. Fletcher found similar large gains relative to classroom instruction when the Digital Tutor was used by veterans seeking employment as electronic technicians in the private sector.

Congratulations to Dr. Fletcher and Dr. Kulik for their high-quality research and success in publishing it in the open literature, exemplifying General Welch’s high standards of analytic excellence and relevance.

Dr. Fletcher and Dr. Kulik

2017 Larry D. Welch Award for Best External Publication winners, Dexter Fletcher and James Kulik, Science and Technology Division

Previous Award Winners

2016   Arun Maiya, Dale Visser, Andrew Wan,
Information Technology and Systems Division

  Lee Hirsch, Science and Technology Division,
Jeremy Teichman, Science and Technology Division

  Carl Gaither, Chris Jackson, Dawn Loper, Operational Evaluation Division,
Jasmina Pozderac, former OED researcher
2013    Kent Haspert, Mike Tuley, Science and Technology Division
   Kevin Woods, Joint Advanced Warfighting Division