Interactive Human-AI Teaming for AI Model Development, Debugging and Repair
PI: Chris Re; Co-PI: Kayvon Fatahalian
Department: of Computer Science
Sponsor: United States (US) Army Research Lab
Artificial Intelligence (AI) models are rapidly expanding the capabilities of human operators to summarize large data corpora, find the “needle in the haystack” and help humans perform complex, goal-oriented tasks. Our research will reduce the time required to train models from person-years/months to person-days impacting defense/intelligence capabilities in infrastructure management, making sense of large volumes of data, and autonomous systems.
The ability to develop, debug and repair AI models requires significant human expertise, time and effort. Training a high-performing AI model requires multiple cycles of training data selection and labeling, model training, model validation and debugging. A major impediment remains: how should we design interactive systems that give human operators more control over these processes, and enhance their ability to rapidly develop models, comprehensively measure their reliability and perform targeted repairs?
Our goal is to develop new, interactive hybrid human-AI systems that will dramatically accelerate the AI system development process. Navigating the complexity of the AI system development process in this research agenda will advance our understanding of how human operators and AI systems can work together productively to make rapid decisions.
For model development, humans often start out with a vague sense of what they want a model to do, since they have an unclear understanding of the kinds of data the model may encounter. Here, we want to build new hybrid human-AI capabilities that give human operators the ability to quickly sift through large volumes of data with modern embedding-based techniques, rapidly label their data to isolate and iterate on groups of interest, and arrive at an updated objective with which they require AI model assistance.
For model debugging, human operators must iteratively discover as many model errors as possible, analyze the errors carefully and find patterns that explain them. Here, we will build new capabilities to help operators write tests that help them express their domain knowledge and expose model errors, and develop new AI techniques to surface and explain model errors automatically by working together with the human operator.
Lastly, for model repair, human operators generally hypothesize about the cause of the errors in training, and lack mechanisms to implement their intuitions to rapidly repair models. We will implement and execute novel mechanisms for model improvement that allow human operators to modify the data and models used for training, and rapidly iterate to a high-quality model that satisfies their requirements.
H4D Focus Areas: AI/ML, Big Data, Technology Transition