Rapid Learning in Machines: Challenges and Responses

This paper reviews the 40-year history of research on machine learning, which developed a variety of techniques for induction from limited amounts of data and draws lessons for the generic task of rapid learning from small and moderately sized sample sets. We compare two paradigms for inductive learning: search through a space of model structures (which characterizes much of the early work) and search through a parameter space (which includes deep learning methods). We also review techniques that have increased the rate of learning from limited data in both paradigms, highlight common themes, and propose adoption of technical and methodological insights obtained from the prior tradition.