News

A Brief CEDM Perspective on Wielding Models to Cope With Complexity in Real-World Settings

By Sudeep Hegde, Martijn IJtsma and Mike Rayo 

 

“Everything simple is false. Everything that is complex is unusable.” – the French philosopher and poet Paul Valéry [1]. This is the paradox of modeling complex systems: Reducing complex phenomena into simple terms distorts reality, yet fully capturing that reality makes the information too difficult or impractical to apply. In today’s world, in which models of systems are developed and applied at a much larger and faster scale than the early 20th century, models can help decision makers and analysts cope with complexity. Models serve to represent real systems, make sense of large-scale data, detect anomalies, make predictions, simulate and test strategies for real world impact, and inform decision-making. However, models can also hinder decision making through oversimplification, overgeneralization, or misalignment with the world (i.e., becoming stale) as the modeled system changes. One area in which cognitive engineering and naturalistic decision making contribute is the study of how models can be wielded effectively to support decision making in complex, dynamic, and high-consequence domains. 

The way CEDM considers models is reflected in the two hidden letters of the TGs name: Cognitive Systems Engineering and Naturalistic Decision Making. First, the “S” for systems highlights both the systems focus when selecting model attributes and the intended role of the models: CEDM considers the development and use of models as part of a larger cognitive system that includes the human problem holder and the work domain (see Joint Cognitive Systems by Woods and Hollnagel, 2006 [2]). In this view, interactions between the model and the human problem holder(s) (for example, the modeler, analyst, decision maker, or operator) play a key role in the successful or unsuccessful use of models. Thus, CEDM views the design, evaluation, and use of models as needing to consider joint system performance rather than just model performance. Second, the “N” for naturalistic highlights that CEDM studies decision making in the wild, with real-world complexity and variability (see Naturalistic Decision Making by Klein, 2008 [3]). This, too, has dual implications. It determines what is in the model–how do we capture the right aspects of that complexity without oversimplification–and the context in which that model is used–where CEDM is interested in how models are used in complex, varying, real-world contexts. 

Thus, with this systems and naturalistic perspective, what lessons from CEDM literature can be applied to modeling? First, any model will be part of a larger cognitive system, where human problem holders will interact with it and depend on it. What do those interactions and dependencies look like? How can those be supported in model development? How can the joint cognitive work of modelers and problem holders/solvers be better supported? CEDM literature has a rich body of methods and concepts (for example, Cognitive Task Analysis) for analyzing and representing the cognitive work of using and interacting with models. Second, in the context of complex, real-world environments, any model has limitations (“everything simple is false”). The model is likely developed for some baseline context. What happens when accuracy or performance of the model is inevitably challenged somehow? How will the larger cognitive system with human problem solvers perform when the model reaches its limits? When the model is false or fails, a well-engineered cognitive system where human roles are able to anticipate and recognize model performance boundaries, and where human expertise can be brought to bear to interpret or redirect the model, is a key part of safely and effectively wielding models. CEDM literature has many concepts and methods for designing for resilient performance, where the human role is a pivotal part of extending performance when models are limited and/or fail in brittle ways. 

Models, while powerful, can become stale or misaligned with real world situations that vary beyond the assumptions, parameters and constraints of the model (see Woods' Model Surprise tutorial). CSE methods for knowledge elicitation, task analysis, and interaction design can be valuable in allowing both, modelers and model-users to examine model fit with real world scenarios in naturalistic settings, and accordingly, reframe, reconceptualize and strategize how to realign models with changing situations. This involves eliciting and bridging between mental models of real-world situations among modelers and model-users to inform necessary adaptations [4]. CSE itself has several modeling frameworks, such as Work Domain Analysis, Data-Frame Theory of Sensemaking and others, that are well-suited to be tailored to domain-specific contexts [5,6]. Such frameworks could be applied to study and design for the cognitive work of those who build and use models. 

We recommend that human factors researchers explore opportunities to collaborate closely with modelers to leverage the power of models to support adaptive capacity and resilient performance. In doing so, it’s important to not settle for being users of models of convenience that others create through their perspectives of the world, but constantly strive to include the important but often missed system aspects that may be more difficult to conceptualize and operationalize, and therefore model. Recent examples of such collaboration serve as groundwork for new directions to scale CSE methods. This includes a framework to make operations research (OR) modeling adaptive and responsive to fundamental surprise situations, like the COVID-19 pandemic, developed based on mental models of both modelers and model-users in the real world [7].  

Last but not least, as a human factors community, we as researchers and practitioners also develop and apply models. For example, representations of the cognitive work or decision-making processes in real-world contexts are also models. In other words, we are what we study. The same challenges and lessons apply to us and we should be cognizant of the limits, possible staleness, and misalignments of our own models.  

References 

  1. Paul Valéry. Bad Thoughts and Not So Bad. In Collected Works of Paul Valery, Volume 14: Analects; Princeton University Press: Princeton, 2015; pp 367–526.  

  1. Woods, D. D., & Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. CRC press. 

  1. Klein, G. (2008). Naturalistic Decision Making. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(3), 456-460. 

  1. Eisenberg, D., Seager, T., & Alderson, D. L. (2019). Rethinking resilience analytics. Risk Analysis, 39(9), 1870-1884.  

  1. Naikar, N. (2016). Work domain analysis: Concepts, guidelines, and cases. CRC press.  

  1. Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. A. (2007). A data–frame theory of sensemaking. In Expertise out of context (pp. 118-160). Psychology Press.  

  1. Sharkey, T. C., Foster, S., Hegde, S., Kurz, M. E., & Tucker, E. L. (2025). A categorization of observed uses of operational research models for fundamental surprise events: Observations from university operations during COVID-19. Journal of the Operational Research Society, 76(2), 254-266.