Volume 57, Number 6
Cogulator, a Tool for Measuring Cognitive Workload: Interview with Steven Estes
By Pam Savage-Knepshield, HFES Bulletin Features Editor
Reducing mental workload has been a fundamental driver of HF/E research for the past 100 years. When attention turned away from "designing the human to fit the machine" to "designing the machine to fit the human," it became increasingly important to understand more about human operators and their performance with systems – to identify mismatches between the demands of the task and human capabilities. Obtaining this information for HF/E practitioners became the impetus for the development of modeling and simulation tools. With this derived information, we can target human-system interactions and reduce the likelihood that mental overload or underload will contribute to errors that cause injurious accidents.
|Cogulator workload output. View a larger version here.
A quick review of the literature identifies a host of cognitive workload software tools, such as the Cognitive Loading Index for Mariners Assessment Technique (CLIMATE), CogTool, GLEAN, SANLab-CM, Apex, Improved Performance Research Integration Tool (IMPRINT), and PHYSIOPRINT, which was inspired by IMPRINT and combines electroencephalography and electrocardiography physiological signals to understand operator workload levels. Cogulator is another promising cognitive workload-modeling tool for measuring operator workload to optimize system design.
In this brief interview, we asked Cogulator's developer, Steve Estes, how the tool came to be.
Q: How did you become interested in the development of cognitive workload software?
A: In this case, it was less of a general interest in developing the software and more a set of needs that required Cogulator's development.
Q: What need(s) prompted the development of this new tool?
A: We use human performance modeling and GOMS (Goals Operators Methods Selection Rules) for a lot of purposes within MITRE, a not-for-profit organization that operates R&D centers for the federal government. One of those purposes is temporal workload estimates for air traffic controllers performing a wide variety of tasks. As we got smarter about how controllers do their jobs, we found ourselves wanting to make modifications to existing tools to reflect that knowledge. For example, our research showed controller rate of speech and audition to be noticeably faster than the default used in many cognitive-modeling tools, but incorporating that knowledge would have required lifting the proverbial hood on the codebase in order to make the changes – something some licenses don't allow for. That gave us the impetus for developing our own software, one of the goals of which was the ability for anyone to change parameters without touching the code. While we were at it, we tried to address a few other issues. For example, for other projects, we were using working memory load to estimate mental workload. Cogulator provided an obvious platform for integrating that work, so we ended up building a working load-and-decay graph into Cogulator.
Q: In what kinds of settings would the tool be useful? Or for which audiences?
A: It's probably most helpful to think in terms of the design goals for the interface. If interface efficiency or mental workload is important, Cogulator can be a useful tool in the design and evaluation process.
Q: What were the most challenging aspects of developing Cogulator, and how did you overcome them?
A: I thought of the design of Cogulator as a recursive problem and wanted to apply good human factors to the development of a good HF/E tool. That translated into spending many orders of magnitude more time thinking about the design than actually developing the software. To put some numbers around that, I worked on Cogulator on and off for well over a year, but I could probably recreate the codebase from scratch in less than two weeks.
I was greatly aided by the fact that I wasn't developing my own cognitive theory and was in fact standing on the shoulders of folks like Stuart Card, Thomas Moran, Allen Newell, David Kieras, Wayne Gray, and Bonnie John, all of whom have done trailblazing work in applied human performance modeling.
There are more challenges to come. I'm working now, for example, on a much more sophisticated representation of A working memory model based on the literature and some original research. The challenge there will be incorporating that sophistication without complicating the interface.
Q: What advice do you have for those who are about to tackle a Cogulator project for the first time?
A: As is often the case within a technical area, the language of cognitive and human performance modeling makes it sound much more complicated than it actually is. My advice for getting started – especially if you're not familiar with cognitive modeling or GOMS – is simply to install Cogulator and take a look at some of the examples. Models are just a listing of steps needed to accomplish a task – a recipe for completing a task, so to speak. I've removed any quasi-coding formalities from the syntax so there are no variables, no if-then statements, and no functions. In most cases, you'll be able to look at the model text and have an immediate sense of what's going on. Beyond that, at cogulator.io there is a primer and a video introduction to Cogulator to help you get started. And if you get stuck, feel free to get in touch with me at email@example.com.
Q: How will you measure the success of the tool?
A: For me, success is less about the adoption of this particular tool and more about encouraging the human factors community to reintegrate some of its engineering roots. So if that means practitioners using and contributing to the tool that we developed, that's fantastic. But I'd be just as happy to see an uptick in the use of any of the cognitive engineering tools we have available to us. That includes other human performance modeling tools like CogTool or just more frequent applied use of cognitive engineering theory. This is, by the way, why Cogulator is (a) free and (b) open source. Making it free removes a potentially significant obstacle to adoption, and open-sourcing it means that folks can evolve this into whatever is most useful for them or for the community.
Steven Estes holds an MA in human factors and applied cognition from George Mason University. He is a principal human factors engineer at the MITRE Corporation's Center for Advanced Aviation System Design in McLean, Virginia. Prior to working for MITRE, Estes was employed as a human factors engineer at Gulfstream Aerospace. Publications include the book chapter "Macrocognition in Systems Engineering: Supporting Changes in the Air Traffic Control Tower," published in Naturalistic Decision Making and Macrocognition (Ashgate, 2008). His 2005 article with coauthors Oscar Olmos, Cheryl Andrews, Anthony D. Andre, Susan Chrysler, and Dan Hannon on the design of airport markings, which have since been implemented throughout the United States, was the 2005 Best Ergonomics in Design Article Award. His work has been referenced in the New York Times and briefed to the White House Communications Agency and FAA administrator. Steven's research interests include cognitive engineering, human computer interface design, working memory, mental workload, and human performance modeling.
Back to the Table of Contents for the June 2014 HFES Bulletin.
Download a PDF version of this issue.
Archive of past HFES Bulletin issues (in PDF format)