The main body of this paper presents an approach to test, evaluation, validation, and verification (TEVV) of autonomous systems based on licensure.1 There are a number of areas where existing work must be converted or new techniques developed in order to make this possible. Detailed treatment in the main body would have distracted the reader from the overall approach. Some of these details are presented here, for selected issues:
- Appendix B, Formal Methods—This appendix provides a review of formal methods techniques, their applicability to licensure and autonomy, and extensive references.
- Appendix C, Requirements and Metrics—This appendix describes a process for defining requirements and associated metrics that supports evidence-based licensure (EBL) for autonomous systems.
- Appendix D, Normative Oracle Generation—This appendix describes the attributes of a Normative Oracle that would support EBL for autonomous systems.
- Appendix E, CoActive Design—This appendix describes co-active design, which focuses on the interdependence of the human and the machine performing a joint activity. We might consider this an extension of teaming among humans, but that would imply a high degree of machine sentience.
- Appendix F, Implications of Learning Autonomous Systems for TEVV—This appendix provides a review of formal methods techniques, their applicability to licensure and autonomy, and extensive references.
- Appendix G, Modeling and Simulation Considerations for Licensure of Autonomous Systems—This appendix addresses modeling and simulation’s role in licensure of autonomous systems. To do so, key drivers from EBL and autonomy are first identified to form a foundation for analysis.