Human Factors Considerations for Enhancing Joint Human-AI Performance: From Function Allocation to Human-AI System Test and Evaluation 

There is growing evidence that how functions are distributed across human and intelligent agents can strongly impact the quality of the joint human-AI system performance. For example, including a person in a decision can sometimes lead to poorer performance than the machine working alone. Similarly, adding a (wrong) machine recommendation can sometimes lead to worse performance than a person working on their own. The design challenge is to identify ways to distribute the work among human and intelligent agents (and provide appropriate displays) so that the joint performance is superior to that of either working alone. 

In this keynote, Dr. Emilie M. Roth will explore practical strategies for designing robust human-AI systems through thoughtful function allocation and rigorous test and evaluation. She will share insights from recent work, including the development of an analytic framework for allocating functions in future human-intelligent system operations, and findings from a recent FAA report she co-authored that offers human-factors guidance on integrating AI and machine learning into aviation systems.

Attendees will gain a clearer understanding of how to design and evaluate human-AI collaborations that outperform either humans or AI alone.

Dr. Emilie M. Roth 

Dr. Emilie M. Roth is the owner and principal scientist of Roth Cognitive Engineering, a company she founded in 1997 that conducts research and applied work in human factors and cognitive engineering. She holds a Ph.D. in cognitive psychology and has more than 30 years of experience in cognitive analysis, design, and evaluation across domains including nuclear power plant operations, railroad operations, military command and control, and health care.

Most recently, in collaboration with colleagues from Applied Decision Sciences, she developed an analytic framework for human–intelligent system function allocation for envisioned world applications. She also co-authored a report providing human-factors guidance for integrating artificial intelligence and machine learning into Federal Aviation Administration systems, together with Phil Smith of The Ohio State University and others.

Dr. Roth is a fellow of HFES and a member of the advisory board for the Journal of Cognitive Engineering and Decision Making. She previously served on the National Academies’ Board on Human-Systems Integration (2015–22) and contributed to several National Academies consensus studies, including investigations of lessons learned from the Fukushima nuclear accident (2012–16) and, more recently, human–AI teaming (2021–22).