CC BY
209
EEG BASED COGNITIVE WORKLOAD CLASSIFICATION DURING NASA MATB-II MULTITASKING
M-Tech. Sushil Chandra, Bio Medical Engineering Department, Institute of Nuclear Medicine and Allied Sciences, DRDO, Delhi, India, E-mail: [email protected] M.Sc. Kundan Lal Verma, Department of Electronics, DDU Gorakhpur University, Gorakhpur, Uttar Pradesh,
India, E-mail: [email protected] M-Tech. Greeshma Sharma, Bio Medical Engineering Department, Institute of Nuclear Medicine and Allied Sciences, DRDO, Delhi, India, E-mail: [email protected] Dr. Alok Mittal, Instrumentaion and Control Engineering Department, Netaji Subhas Institute of Technology,
Delhi, E-mail: [email protected] Dr. Devendra Jha, Scientific Analysis Group, DRDO, Delhi, India
E-mail: [email protected]
Received: January, 06.2015. Accepted: February, 09.2015.
Original Article UDK 004.032.26
Abstract. The objective of this experiment was to determine the best possible input EEG feature for classification of the workload while designing load balancing logic for an automated operator. The input features compared in this study consisted of spectral features of Electroencephalography, objective scoring and subjective scoring. Method utilizes to identify best EEG feature as an input in Neural Network Classifiers for workload classification, to identify channels which could provide classification with the highest accuracy and for identification of EEG feature which could give discrimination among workload level without adding any classifiers. The result had shown Engagement Index is the best feature for neural network classification.
Keywords: Cognitive Workload, Discrete wavelet transform, EEG spectral feature, Neural Network.
1. INTRODUCTION
Cognitive Workload is defined as the load generated due to processing of multiple activities (or sub activities of a task) by the human brain in the time domain. In our context, it depends on how efficiently there is an interaction between operator's resource capabilities and computer/machine on which the tasks have to be displayed. When the operator's workload is at the peak, then a properly
Corresponding Author
Greeshma Sharma, Research Fellow
Bio Medical Engineering Department,
Institute of Nuclear Medicine and Allied Sciences,
DRDO, Delhi.
E-mail: [email protected]
designed automation task can reduce workload or maintain its optimal level (Kramer, 1991). Workload assessment of operator during task can offload some of his tasks to the machine with the help of smart system design. For designing of such system, psychological and physiological aspects are carefully assessed for finding out suitable combination that can provide optimum utilization of human resource with the machine. In this research paper, we tried to identify EEG channels with the best feature that could give a maximum classification of workload. It also included feature identification without adding any classifiers. There are three methods to assess the cognitive workload, i.e. task score, subjective score and spectral features of the EEG (Wickens, 1988). Workload assessment through giving a questionnaire about a task makes subjective judgments about the task (Reid and Nygren, 1988). Multi Attribute Task Battery (MATB-II) was used for workload assessment in two modes, LWL and HWL. It was used for multitasking as well as for workload assessment (Singh, A. L., Tiwari, T., Singh, I. L, 2010; Singh, I. L., Sharma, H., Singh, A. L., 2005). Advantage of choosing MATB over others were its qualification for providing objective and subjective scoring along with different level of workload (Hart and Staveland, 1988).
Different Attentional demands like to divide attention which is generated by a task change pattern of subjective workload and its psychological impacts (Mazloumi, Kumashiro, Izumi, and Higuchi, 2010). There
is a nonlinear relation between workload and performance (Gawron, 2000). Therefore, we didn't rely completely on subjective & objective scores; we included electrophysiological (EEG) factors also (Gevins and Smith, 2003). There are a lot of research articles that show significant relationships between EEG parameters like engagement indices and performance both in fields and off fields (Berka et al., 2007; Freeman, Mikulka, Scerbo and Scott, 2004; Mikulka, Scerbo and Freeman, 2002). EEG analysis is performed using classically defined frequency bands alpha (8-12 Hz), beta (13-22 Hz), delta (23-30 Hz), theta (5-7Hz) and gamma (30-70 Hz) (Dickson, 2005; Dorneich, Ververs, Mathan and Whitlow, 2005) or sometimes it can be analyzed using some ratio of frequency bands, Prinzel et al (2000) developed an EEG-engagement index based on beta power divided by alpha power plus theta power. A feed forward neural network based on back propagation algorithm was used to classify the workload into NWL, LWL and HWL among 14 channels in MATLAB 2012.
The objective was (i) Comparison among input features to determine their qualifications as a workload level discriminating factor. Here input features included objective scores, subjective scores and EEG features. (ii) To determine the most suitable input feature that can be used in designing human-machine load balancing logic for an automated operator. It was based on statistical parametric testing that gave output with significance.
2. MATERIALS AND METHODS
A. Procedure
Subjects (n=10, age group 20-30 years, all male and right handed) from the Institute of Nuclear Medicine and Allied Science, New Delhi were invited to perform NASA's MATB-II pilots multitasking tasks, which had no prior experience of MATB like piloting tasks. Consent was taken from all subjects to participate in this project. They were required to fill a questionnaire to build their general profile. Each subject was well trained with the MATB-II software at the beginning of the testing as per given module of MATB-II software (Fairclough and Venables, 2010; Fairclough, Venables, and Tattersall, 2005).Lab ergonomics were followed by the idealarrangement. The MATB-II task stimuli was presented on a computer screen (15 inches) with a dark
background and a viewing distance of 80cm. EEG from each subject was recorded with low-cost EEG device Emotiv EPOC (Knoll, Wang, Chen, aand Xu, 2011) which was a 14-channel, 128 Hz neuro-signal acquisition and processing wireless neuro headset (Figure 1).
The following steps were performed by the subjects for workload assessment through the tasks assigned as per NASA's MATB-II, which was designed according to an eight minute event file.
1. During the experiment, three minute baselines was recorded with eyes open. In this situation, a laptop screen with no luminescence was kept in front of the subject's eyes.
2. 1 minute rest.
3. NASA MATB-II tasks at Low workload level were given to subjects.
4. 1 minute rest.
5. NASA MATB-II tasks at High workload level were given to subjects.
Figure 1. Lab Arrangement
B. Technique
(i) Feature Extraction
Root mean square (RMS) value
The RMS value is mainly used for varying a quantity and useful statistical parameter to see the effect of workload on EEG signals (Basmajian and Luca, 1985). It is useful to measure power in amplitude of EEG signals from the cerebral cortex (Abdul-latif, Cosic, Kumar, Polus, and Da Costa, 2004).
The RMS for a collection of N values {x1, x2, ..., xN} is given by the equation
Sub band Energy
Wavelet packet node energy is more useful in representing a signal. Wavelet packet component energy E and total signal energy can be defined as
- Ef ~ Zf=iEfi
'£01
Multi-channel full-band EEG signals are decomposed into five well-known frequency sub-bands: delta, theta, alpha, beta, and gamma. Different energy bands like gamma, alpha, beta and theta have been calculated with MATLAB programming (Sun, 2008).
Power Spectral Density (PSD)
The power spectral density is mostly directed at the continuous spectrum of the signal (Sun, Chang, and Tang, 2006). In contrast to the mean-squared spectrum, the peaks in this spectrum have no reflection of the power at a given particular frequency (Zarjam, Epps, and Chen, 2011).
Engagement Index (EI)
Task engagement is defined as how much the subject is involved in particular tasks on different levels of workload (Kamzanova, Matthews, Kustubayeva, and Jakupov, 2011). EEG engagement index denoted by formula (P/(a+9)) and has been to classify workload (Berka et al., 2007).
(ii) EEG Signal Analysis
EEG signal processing means for operating in some fashion on a signal to extract some useful information from EEG data, different features such as power spectral densities, energy and root mean square (RMS) were extracted from the signal with the help of MATLAB programming.
(iii) Neural Network classifiers
We used feed forward neural network based on back propagation algorithm. It is assumed superior classifiers for discriminating workload level (Wilson and Russell, 2003, Wilson and Russell, 2007; Wilson, Estepp, and Christensen, 2010; Wilson, Estepp, and Davis, 2009). Twenty-five percentage of the inputs
were taken for validation and 20% for training and rest inputs were test data. Linear transformation matrix was used to map between input and output. It had used features x channels, matrix for training and testing data set.
3. RESULTS
The overall result was calculated on the basis of psychological (MATB-II test score analysis with subjective score) and physiological (EEGSignal) data analysis separately.
The following three findings were extracted from MATB-II data.
1. Subjective score was calculated using NASA-Tlx scoring and total score displayed in Figure2for comparison between LWL and HWL.
Figure 2. Total NASA-Tlx score of participants in the MATB-II
2. Objective score was calculated for system monitoring (sysmon) and tracking task in the form of response time (in second) and Root mean square deviation (RMSD) respectively. In sysmon, percentage of correct responses were more in LWL case for five subjects (Figure 3) and also response time was less for HWL (increase stress sometime improve performance) (Figure 4).
Figure 3. Percentage correct responses for system monitoring task
Figure 4. Response Time (in second) of system monitoring for participants who gave responses
12
Si S5 » S7 ss S9 S10
PtffKipjdti
3. Three subjects were unable to give the responseinsysmon as they were unable to shift their attention. Improve performance was observed in a tracking task for HWL as in figure 5. Applying t- test on objective scores, we found: t= 1.550 at degree of freedom =9 for tracking task at probability = 0.155 and t = 1.030 at degree of freedom = 9 for system monitoring task at probability = 0.330.There were no significant difference between LWL and HWL task of tracking and system monitoring.
Figure 5. Root means square deviation (RMSD)
scores for tracking task
'iiiiihiiji'
Three features are extracted from the EEG signal. To summarize the findings are:
1. Subband energy value was increased as the workload increased for theta [4-8 Hz], alpha [8-13 Hz], beta [13-30 Hz] as showed in figure 6for channel AF3. More energy was displayed from channels when high attention is required to execute the task.
Figure 6. Beta energy for AF3 channel for different condition, energy is summarized among all ten participants. BL=baseline, LWL=Low Workload, HWL=High Workload
II
LWL HWL
Condition in Enperimemt
2. The RMS value of gamma sub-band had the most significant difference between NWL and LWL as displayed in Table 1 for all channels.
Table1. Significant result on 14 channels for RMS value after applying one way ANOVA. Difference between NWL and LWL after applying post hoc test (Tuckey-d)
Serial number Channel Naine F Value P Value
1 ÀF3 3.772 0.036
i jLr AF4 3.243 0.055
3 F3 4.042 0.029
4 F4 5.639 0.009
5 F7 3.689 0.039
6 FS 4.584 0.019
7 FC5 5.872 O.OOS
8 FC6 5.117 0.013
9 01 6.233 0.006
10 02 5.451 0.011
11 P7 5.919 0.007
12 PS 6.041 0.007
13 T7 6.972 0.004
14 T8 3.664 0.039
3. The most important channels in Engagement Index (EI) were AF3, AF4, F7 and F8 as we could see an increase in EI with the increase in workload clearly from figure 7.
Figure 7. Engagement index for each channel on Baseline (BL), Low workload level (LWL) and high workload Level (HWL)
4. The above mentioned three EEG features were taken as an input for NN classifiers and EI was selected as most appropriate input for workload classification after doing parametric statistical testing.
5. Confusion matrix for all 14 channels had been displayed in Table 2, showed an idea about channel identification for automated operator designing. As per the result accuracy was more on anterior frontal channels while less in temporal channels.
Table 2. Confusion Matrix result for classification of NWL, LWL and HWL, after applying NN classifiers with 1000 iteration for channels
Serial nutn- Channel Percentage
ber Name Accuracy
1 AF3 90
2 AF4 66.7
3 F3 63.3
4 F4 50
5 F7 70
6 FS <53.3
7 FC5 60.9
S FC6 63.3
9 01 50
10 02 56.7
11 P7 60.9
12 PS 60.9
13 T7 60.9
14 TS 46.7
4. DISCUSSION
Workload offloading is often required to reduce error and improve performance in the smart system. It can be done when we acquire
full knowledge regarding operator cognition and system interface. It is evident that workload is task dependent and can vary from individual to individual (Gawron, 2000; Mazloumi, Kumashiro, Izumi, and Higuchi, 2010). The objective of this experiment was to identify the measurement technique that can discriminate the workload and to determine the best possible input EEG feature for classification of workload while designing load balancing logic (i.e. between human and machine) for an automated operating system. We found some answers which could be supportive for objective completion. We discussed three methods to assess cognitive workload, i.e. task score, subjective score and spectral features of the EEG as per Wickens, 1988. Objective scores collected from sysmon and tracking tasks using MATB-II, failed to produce any significant difference. Difference in result failed to correlate the workload level with the performance of individuals. However, we found a subjective score making workload discrimination, but we could not rely on subjective feeling because it was invasive which could be inappropriate for our objective. EEG was reliable and modest measured for workload discrimination (Gevins and Smith, 2003; Gevins and Smith, 2005; Wang, Hope, Wang, Ji, and Gray, 2011). The primary goal was to identify the best EEG feature which could be taken as an input for NN classifiers. Out of three EEG features, i.e. energy, RMS and EI, EI was best identified feature for NN classifiers. In previous studies EI was also identified best correlated feature with workload (Kamzanova et al., 2011). Second goal was to identify the best channels which provided the classification with the highest accuracy. Anterior frontal channels were the most appropriate channels. The third goal was to identify EEG feature which could give discrimination among workload level without adding any classifiers. Gamma sub-band RMS value was qualified to discriminate (except AF4) with significant difference at P < 0.05.
5. CONCLUSIONS
To identify measurement technique for workload assessment supplemented designing of automated operator. Objective score failed to discriminate and subjective score succeeds, but they were invasive. We left with only electrophysiological method of workload discrimination that was necessary for designing of automated operator. Therefore, to rely
on electrophysiological measures did become one of the major key in workload classification (Lysaght, 1989; Noyes, J.M. and Bru-neau, 2007; Rubio, Diaz, Martin, and Puente, 2004; Verma, 2012). Gamma RMS was identified best feature without adding classifiers and EI was the best feature to be used as an input for NN classifiers. Best selected channels were anterior frontal for workload classification. The limitation of our study could be less number of subjects and channels in EEG. Result based on the onetime assessment of the workload, could have changed in the next assessment. Future direction of work will be based on the overcoming limitation with many time assessments with sufficient number of subjects and channels.
ACKNOWLEDGEMENTS
The authors would like to thank the volunteers for participating in the experiment, and are grateful for the assistance of Mr. James R. Comstock and Miss Yamira Santiago, NASA Langley Research Centre in providing the software NASA MATB-II. The authors are also thankful to Director INMAS for their support and help during the course of the experiment. This research was granted by INMAS, DRDO, Delhi.
Conflict of interests
Authors declare no conflict of interest.
REFERENCES
Abdul-latif, A. A., Cosic, I., Kumar, D. K., Polus, B., & Da Costa, C. (2004). Power Changes of EEG Signals Associated with Muscle Fatigue: The Root Mean Square Analysis of EEG Bands. Intelligent Sensors Sensor Networks and Information Processing Conference (pp. 531-534). IEEE. doi:10.1109/ISSNIP.2004.1417517 Basmajian, J. V, & De Luca, C. J. (1985). Muscles alive. Muscles alive: their functions revealed by electromyography, 278, 126. Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., ... & Craven, P. L. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine, 78 (Supplement 1), B231-B244. Dickson, B. T. (2005). The Cognitive Cockpit-a test-bed for Augmented Cognition. In 1st International Conference on Augmented Cognition, Las Vegas, NV
Dorneich, M. C., Ververs, P. M., Mathan, S., & Whitlow, S. D. (2005, October). A joint human-automation cognitive system to support rapid decision-making in hostile environments. In Systems, Man and Cybernetics, 2005 IEEE International Conference on (Vol. 3, pp. 2390-2395). IEEE.
Fairclough, S.H., Venables, L., & Tattersall, A. (2005). The influence of task demand on learning on the psychophysiological response. International Journal of Psychophysiology, 56(2), 171-184.
Fairclough, S.H., & Venables, L. (2010). Prediction of subjective states from psychophysiology: a multivariate approach. Biological psychology, 71(1), 100-110.
Freeman, F. G., Mikulka, P. J., Sceibo, M. W., & Scott, L. (2004). An evaluation of an adaptive automation system using a cognitive vigilance task. Biological psychology, 67(3), 283-297.
Gawron, V.J. (2000). Human performance measures handbook. Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers.
Gevins, A., & Smith, M. E. (2003). Neurophysiologi-cal measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomics Science, 4,113-131.
Gevins, A., & Smith, M. E. (2005). Assessing fitness-for-duty and predicting performance with cognitive neurophysiological measures. In J. A. Caldwell & N. J. Wesensten (Eds.), Proceedings of SPIE Defense and Security Symposium, Biomonitoring for Physiological and Cognitive Performance during Military Operations (pp.127-38). Orlando: The International Society for Optical Engineering.
Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology, 52, 139-183.
Kamzanova, A. T., Matthews, G., Kustubayeva, A. M., & Jakupov, S. M. (2011). EEG Indicesto Time-On-Task Effectsand to a Workload Manipulation (Cueing). World Academy of Science, Engineering and Technology, 80.
Knoll, A., Wang, Y., Chen, F., Xi, J., Ruiz, N., Epps, J., & Zarjam, P. (2011). Measuring Cognitive-Workloadwith Low Cost Electroencephalogram. Human Computer Interaction-Interact, lecture Notes in Computer Sciences, 6949, 568-571.
Kramer, A. F. (1991). Physiological metrics of mental workload. In D. L. Damos (Eds.). Multiple task performance (pp.279-328). London: Taylor & Francis.
Lysaght, R. J., Hill, S. G., Dick, A. O., Plamondon, B. D., & Linton, P. M. (1989). Operator workload: Comprehensive review and evaluation of operator workload methodologies (No. TR-2075-3). ANALYTICS INC WILLOW GROVE PA.
Mazloumi, A., Kumashiro, M., Izumi, H., & Higuchi,
Y. (2010). Examining the Influence of Different Attentional Demands and Individuals' Cognitive Failure on Workload Assessment and Psychological Functioning. International Journal Of Occupational Hygiene, 2(1), 17-24.
Mikulka, P. J., Scerbo, M. W., & Freeman, F. G. (2002). Effects of a biocybernetic system on vigilance performance. Human Factors, 44, 654-64.
Noyes, J. M. & Bruneau, D. P. J. (2007). A self-analysis of the NASA-TLX workload measure. Ergonomics, 50(4), 514 - 519.
Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J., & Pope, A. T. (2000). A closed-loop system for examining psychophysiological measures for adaptive task allocation. The International Journal Of Aviation Psychology, 10(4), 393-410.
Reid, G. B., & Nygren, T. E. (1988). The subjective workload assessment technique: A scaling procedure for measuring mental workload. Advances in psychology, 52, 185-218.
Rubio, S., Diaz, E., Martin, J., Puente, J. M. (2004). Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods. Applied Psychology: An International Review, 53(1), 61-86.
Singh, I. L., Sharma, H., Singh, A. L. (2005). Effect of Training on Workload in Flight Simulation Task Performance. Journal of the Indian Academy of applied Psychology, 31, 83-91.
Singh, A. L., Tiwari, T., Singh, I. L. (2010). Performance Feedback, mental Workload and Monitoring Efficiency. Journal of the Indian Academy of applied Psychology, 36(1), 151-158.
Sun, S. (2008). The extreme energy ratio criterion for EEG feature extraction. In Artificial Neural Networks-ICANN 2008 (pp. 919-928). Springer Berlin Heidelberg.
Sun, L., Chang, G., & Tang, H. (2006). Wavelet Packet. Entropy in the Analysis of EEG Signals. 8th International Conference on Signal Processing (pp.16-20). Beijing: IEEE.doi:10.1109/ IC0SP.2006.346140.
Verma, K. L. (2012). EEG correlates of Cognitive Workload during Multitasking Work. Dissertation, University of Gorakhpur. ISBN: 9783-659-30224-4. Germany, Lambert Academic Publishing.
Wang, Z., Hope, R. M., Wang, Z., Ji, Q., & Gray, W. D. (2011). An EEG Workload Classifier for Multiple Subjects. 33rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Washington, DC: IEEE Press.
Wickens, C. D. (1988). Processing resources in attention, dual task performance and workload assessment. University of Illinois: Engineering-psychology research laboratory.
Wilson, G. F., & Russell, C. A. (2003). Operator functional state classification using multiple
psychophysiological features in an air traffic control task. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(3), 381-389.
Wilson, G. F., Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Factors, 49, 1005-1018.
Wilson, G. F., Estepp, J., & Christensen, J. C. (2010). How does day-to-day variability in psychophysiological data affect classifier accuracy. Human Factors and Ergonomics Society Annual Meeting Proceedings, 54, 264-268.
Wilson, G. F., Estepp, J., & Davis, I. (2009). A comparison of performance and psychophysiologi-cal classification of complex task performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53, 141-145. doi: 10.1177/154193120905300301.
Zarjam, P., Epps, J., & Chen, F. (2011). Spectral EEG Features for Evaluating Cognitive Load. 33rd Annual International Conference of the IEEE EMBS (pp. 3841-3844). Boston, Massachusetts USA. doi: 10.1109/IEMBS.2011.6090954
