Common Subjective Mental Workload Assessment Measures: Advantages and Disadvantages

Introduction

Mental workload (MWL) can be defined as “a burden in which a person’s duties, whether in single or multiple instances, impose on his or her limited mental resources” [1]; and “on the relationship between the demands of the task and the performance of the user” [2]. In fact, the MWL refers to that portion of the operator’s data processing capacity that is needed to meet system requirements [3]. Mental workload are broken generally down into several different aspects include cognitive, perceptual, emotional, and temporal aspects, which are important to consider in understanding the cognitive demands placed on an individual [4, 5]. Cognitive aspects refer to the amount of mental effort and resources required to perform a task. The inherent complexity of the task, the way of information presentation, and the cognitive processing are affective in cognitive aspects mental workload [6]. Perceptual aspect involves the demands placed on an individual’s perceptual systems, such as visual or auditory processing, and the ability to attend to relevant stimuli while filtering out irrelevant distractions [6]. Another aspect of mental workload is related to the emotional factors, such as stress, anxiety, or frustration. Temporal aspect of MWL includes task duration, pacing, and timing of demands that can vary over time and affect the required cognitive resources [5, 6].

Stress and strain are two distinct concepts in the context of mental workload assessment that affect human performance in different working conditions [7]. Stress typically refers to the psychological and physiological response to perceived demands or pressures, while strain often refers to the negative outcomes or consequences of stress, such as fatigue, reduced performance, or physical discomfort. Low MWL, i. e. underload, can cause distraction, errors, accidents, decreased alertness and feelings of being frustrated or annoyed within the workplace [7, 8]. In contrast, high MWL, i. e. overload, can lead to confusion, fatigue, human error and ultimately reduced work performance and job dissatisfaction [8, 9]. Assessment of MWL is an important part of system design and analysis [10]. Techniques for evaluating MWL are classified into three categories: 1) self- assessment, 2) functional measures, and 3) physiological measures. The advantage of physiological measures is the continuous collection of data, without the need for an overt response from the user [8]. However, with physiological and functional methods, evaluation generally takes place using interventional methods [11], special equipment, and specifically trained operators [8]. This limits the evaluation of real-world tasks and is often only performed in laboratory settings [8, 9, 10, 11]. Furthermore, due to the diversity in the way the human body reacts to MWL and the individual differences in the process, no physiological criterion definitely evaluates MWL, or can be classified as a ‘Gold Standard’ [12].

Since MWL is a psychological construct, measuring it with subjective scales such as questionnaires, may be more appropriate [3]. These mental directed self-reported questionnaires have low intrusiveness, high subject mental acceptability [8], ease of implementation [13], and are inexpensive to administer and analyse [14, 15, 16]. Due to their acceptable validity and reliability as reported by some studies, they are transferable to a wide range of tasks and working situations [14].

For the assessment of MWL, various questionnaires have been proposed, including the NASA Task Load Index (NASA- TLX), Subjective Workload Assessment Technique (SWAT), and the CarMen-Q workload questionnaire [7].

The NASA-TLX is a widely used subjective assessment tool for measuring MWL which was developed by NASA in the 1980s to evaluate the mental demand of tasks. This was in order to improve the design of complex systems, and to assess the impact of new technologies on human performance. The questionnaire consists of six subscales: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. Respondents rate each subscale from 0-100, with higher scores indicating higher perceived workload [6, 7, 8, 9, 10, 11, 12, 13].

The SWAT measures workload by considering three different areas: Psychological Stress Load, Mental Effort Load, and Time Load. A three-point grading system is used to describe each dimension (low, medium, and high), with each item is scored as 1-3. Ultimately, a three-digit figure is obtained, which is then converted to a percentage by comparing it to the method’s mean scoring (2727 score modes of 1–27) [17, 18] where scores <33 indicate low mental exertion, 34-66 are moderate, and >67 indicates severe MWL [13].

The Carman-Q questionnaire [13] has 29 items divided into four requirement subscales: cognitive (ten questions), temporal (seven questions), emotional (seven questions) and performance (five questions). Each item has four response options (0=never, 1=rarely, 2=often, and 3=always). In all cases, a higher score indicates a higher MWL [11].

Discussion

We aimed to compare these three commonly advocated and used questionnaires where the subject describes perceived mental demand during a specific task or activity, and to relate the advantages and disadvantages of each tool. In this study, only three common methods of MWL evaluation have been compared and these methods have been selected according to their prevalence usage. There may be other methods in this field that are not considered in this study. Although the SWAT was developed and tested within a psychological model framework for demand on human- perceived information processing, some studies have indicated the NASA-TLX as preferable. This is in terms of sensitivity, especially in low MWL conditions [6, 7, 8, 9, 10, 11] and administrative demand, where calculating the final score is an arduous and lengthy procedure for obtaining a workload rating [18].

The NASA TLX scale has been demonstrated as valid, reliable, and sensitive. It also has high acceptance among ergonomists and within ergonomics organizations for measuring MWL due to its low administrative burden which comes from its ease and quickness of use [19]. However, it has practical limitations for real world field applications. Workers are generally more familiar with questionnaires that use a Likert scale to gain their views about the frequency, intensity, or importance of particular work environment tasks or situations. Further, previous NASA-TLX research has indicated the limited practical significance of the ‘performance dimension’, as it can be affected by changes in the task/s physical load [20]. Further, the NASA-TLX has potential response bias, and its psychometric properties have only been evaluated in experimental/laboratory conditions, with real world task assessment remaining [11].

In contrast the CarMen-Q modifies the performance dimension and does not involve items associated with the physical demands of the job in order to obtain an unadulterated assessment of MWL. It can also be used in job design and occupational health practice [11]. The tool was developed to assess MWL in a valid and reliable way which has provided appropriate psychometric properties. Further it has low administrative burden making it useful and brief for diagnosing and preventing MWL. However, no research has been completed on its factorial structure and its validity has not been confirmed in real world task specific workstations.

In a study comparing the NASA-TLX and SWAT, there were no differences determined in their intrusiveness [13]. However, the NASA-TLX is only able to differentiate only between single and dual tasks, while the SWAT can differentiate between tracking and memory tasks. For both tools the ‘Convergent Validity’ indicates positive correlation coefficients approaching 1. 0 or high, which indicates both tools evaluate the same theoretical concept. However, the ‘Concurrent Validity’ for NASA-TLX is also high in relation to performance. Both tools demonstrate similar sensitivity, although the NASA-TLX is slightly more sensitive when the F value is considered [13].

Conclusion

The NASA-TLX, SWAT, and Carmen-Q are subjective measures used to assess mental workload in different contexts. The NASA-TLX is multidimensional, well established, and widely used. The SWAT in contrast is a single-item measure, easy-to-use, but less sensitive or specific than the NASA-TLX. The CarMen-Q questionnaire is a relatively new tool that focuses on cognitive workload and mental fatigue. The choice of which tool to use depends on the specific needs of the research or application that is under consideration by the clinician or researcher.