In this first part of a two-part roadmap for the test and evaluation community, we discuss the challenges and possible solutions to the problem of assessing the effectiveness of artificial intelligence and autonomous systems for the Department of Defense.