CC BY
14
DOI: 10.15514/ISPRAS-2022-34(3)-10
Usability Evaluation of Brain Computer Interfaces: Analysis of State of Art
Y.N. Ortega-Gijón, ORCID: 0000-0002-5396-0676 <zS18016065@estudiantes.uv.mx> C. Mezura-Godoy, ORCID: 0000-0002-5386-107X <cmezura@uv.mx> Facultad de Estadística e Informática, Universidad Veracruzana, Xalapa-Enríquez, Veracruz, México 91020
Abstract. Brain Computer Interfaces - BCI allow users to communicate with the software system through cognitive functions measurable by brain signals, identified as Electroencephalography - EEG. User tests have been the most used method for usability evaluation of BCI software applications. In user tests, the data collected comes from the opinions of users through questionnaires, these tests require a lot of time, since they include not only performing interaction task and the application of the questionnaires, but also include placing and calibrating the EEG device. All this makes the evaluation process a very heavy task for the participants of the test and can mean that the data collected is not entirely reliable. That is why we are interested in including EEG signals in the usability evaluation process of applications with BCI software applications. Therefore, we present in this paper the result of the analysis of state of art in order to identify the relevant works in the area and future lines of research.
Keywords: usability evaluation; brain-computer interfaces; EEG signals
For citation: Ortega Y.N., Mezura-Godoy C. Usability Evaluation of Brain Computer Interfaces: Analysis of State of Art. Trudy ISP RAN/Proc. ISP RAS, vol. 34, issue 3, 2022, pp. 145-158. DOI: 10.15514/ISPRAS-2022-34(3)-10
Оценка пригодности к использованию нейрокомпьютерных интерфейсов: анализ состояния дел
Й.Н. Ортега-Хихон, ORCID: 0000-0002-5396-0676 <zS18016065@estudiantes.uv.mx> К. Мезура-Годой, ORCID: 0000-0002-5386-107X <cmezura@uv.mx> Факультет статистики и информатики, Университет Веракрусана, 91020 Мексика, Веракрус, Халапа-Энрикес
Abstract. Нейрокомпьютерные интерфейсы (Brain Computer Interfaces, BCI) позволяют пользователям общаться с программной системой посредством когнитивных функций, измеряемых сигналами мозга, которые опознаются с помощью. электроэнцефалографии — ЭЭГ. Наиболее часто используемым методом оценки удобства использования программных приложений BCI являются пользовательские тесты. В пользовательских тестах данные собираются на основе мнений пользователей, получаемых путем анкетирования. Такая оценка требуют много времени, поскольку требуются не только выполнение задания на взаимодействие и заполнение анкет, но также и размещение и калибровку устройства ЭЭГ. Все это делает процесс оценки очень тяжелой задачей для участников теста и может означать, что собранные данные не совсем надежны. Вот почему нас интересует включении сигналов ЭЭГ в процесс оценки удобства пригодности к использованию приложений BCI. Поэтому мы представляем в этой статье результат анализа состояния дел, чтобы определить значимые работы в этой области и будущие направления исследований.
Ключевые слова: пригодность к использованию; нейрокомпьютерные интерфейсы; сигналы ЭЭГ
Для цитирования: Ортега-Хихон Й.Н., Мезура-Годой К. Оценка пригодности к использованию нейрокомпьютерных интерфейсов: анализ состояния дел. Труды ИСП РАН, том 34, вып. 3, 2022 г., стр. 145-158.. DOI: 10.15514/ISPRAS-2022-34(3)-10
1. Introduction
Usability is defined as: «The range in which a product can be used by specific users to achieve certain specified goals with effectiveness, efficiency and satisfaction in a specified context of use» (ISO 9241-11). The usability evaluation can be carried out following different paradigms: Quik and dirty, usability tests, field and predictive studies, which through different techniques such as: user tests, thinking aloud, interviews, questionnaires, heuristics, etc. collect data for analysis. The data collected can be quantitative and qualitative, which can also be recovered from physiological measures such as: cardiac rhythm, blood pressure, temperature, etc.
Our interest is the evaluation of the usability of BCI software applications. In these applications, the interaction between users and the computer system takes place through electrical activity of the human brain and the device to be controlled Gentiletti [17]. In BCI software applications, usability is traditionally evaluated by user testing. User tests can be Qualitative usability testing and Quantitative usability testing.
These tests comprise 3 phases: 1) Interaction with the User, 2) Application of the questionnaires (post-task or post-test, and 3) Collection and analysis of data. The interaction is the time allocated to the task, the application of questionnaires, the opinion about the application is obtained, then the data of the questionnaires are collected and analyzed by statistical means (mean, average, mode). Through the questionnaires, subjective measures are retrieved that depend on the opinion or state of mind of the user, which may affect the results of the evaluation. On the other hand, the application of the questionnaire can be given only at specific times, it is usually done after completing the test, however, the questionnaires can also be applied after completing specific tasks, which can increase the time of the test. evaluation, this can cause fatigue in the participant when performing the usability test, for example at work [35], users express that the questionnaires are confusing, long, tedious and presented a high degree of fatigue and workload throughout the evaluation process. Particularly for the evaluation of BCI software applications through usability testing, the calibration phase is added to the process at the beginning of the evaluation process. We call it Phase «0» calibration refers to ensuring that all channels respond equally, looking at the quality of the signal. This phase can make the test longer and can contribute to participant fatigue. On the other hand, it is important to note that EEG signals have been used in «Neuromarketing»[52] and «Clinical studies» [16] where are they used to assess cognitive states of patients. Therefore, the EEG signals have been linked to workload, concentration, emotions, etc. [2,3,14]. The paper by Rhiu [44] shows a review of the BCI evaluation works, classifying by dimension of usability and showing the measures, but obtained only through questionnaires and describing the most used. The objective of this paper is to present the results of the analysis of the literature on the usability evaluation of applications with ICC, that is, to know what are the methods and techniques used in the evaluation, highlighting the analysis of the use of EEG signals in the obtaining usability measures, the data that ensured the EEG signals and exploring the possibility of being used in the evaluation of usability, classifying the EEG signals by measure and dimension. The paper is organized as follows: In section 2 it presents the description of the BCI software applications, in section 3 the techniques, dimensions, measures, and the usability evaluation process are shown, in section 4 the process that was carried out for the search, from the selection of the database to the analysis of the results, the classification and the analysis carried out of the information (papers) obtained, and in section 5 the discussion of the results (evidence) and finally the conclusions.
2. Brain-Computer Interface - BCI
The term interface is used to name the functional connection that exists between two software systems, devices, which provides communication at various levels, making an exchange of information possible. In Brain-Computer Interfaces-BCI, this exchange of information takes place between the electrical activity of the brain and the device to be controlled. BCIs provide their users with communication and control channels that are not dependent on normal output channels [17]. BCI software applications can be developed using a variety of different types of neurological signals, such as functional near-infrared spectroscopy (fNIRS), magnetic encephalography (MEG), or functional magnetic resonance imaging (fMRI). However, one of the most widely used methods to measure neurological activity used in BCI is the electroencephalogram (EEG). The architecture of a BCI can be divided into 3 important components. The first component is the human through cognitive functions that are executed when the human being receives, interprets, and stores information, at the moment of sensory perception, then an action is executed according to the previous perception. The second component is the interaction, when the action performed by the user is sent to an input interface, the recognition of the action is performed, the representation and ending with the sending to the output interface, to start the cycle again with the perception. The third component is the recognition of the actions by the computer and the subsequent representation, the computer is in charge of interpreting the cognitive functions of the user and executing the actions thus giving feedback to the user to continue with the cycle.
There are different types of devices to measure brain activity, from the complete medical EEG with 32 channels, to headbands or caps that contain 32 to 2 channels. One of the most widely used headbands is the Emotiv Epoc [47,49]. Other hardware used are NIRSport 2 [28], the IMEC EEG [39], or they decide to create their own device [39].
Neural activity during user interaction is recovered with EEG signals. In the signal analysis, three stages are defined: 1) the acquisition of the signal, 2) the processing of the signal, and finally, 3) the interaction with the control interface and the device driver. The signal processing stage can be divided into 2 actions: characteristic extraction and classification. In this phase, specialized techniques and algorithms are used. For the extraction of characteristics, the most used algorithms are: ICA [49], LDA [51], PCA, etc., dedicated to obtaining the characteristics, important or predominant patterns in the EEG signal. Subsequently, for the classification of characteristics are: the linear discriminant analysis that uses Bayes' theorem, the vector support machine [10], the artificial neural networks (multilayer perceptron) [53], the model classifiers of hidden Markov, the fast Fourier transform (FFT) [7,16], among others.
3. Usability in Bci Software Applications
Usability is part of the broader term «User Experience-Ux» and refers to the ease of access and/or use of a product or website [33]. A design is not usable or unusable; it depends on its characteristics, the user context (what the user wants to do with it and the user's environment), all this determines its level of usability.
There are 4 paradigms for usability evaluation: Quick and dirty, usability testing, field studies and predictive or heuristics. The first 3 paradigms require user participation and in the last paradigm, the evaluation is done by usability experts, using heuristics or interaction models [18,57]. These techniques require the participation of a representative sample of end users. These evaluations are usually carried out during the later stages of development. Representative techniques are: 1) Thought aloud protocol: When users are tested, while they are in the interaction phase, users are asked to verbally express what they are thinking and what they do not understand, to express their opinions about the system, product, software, etc. [34,38]. 2) Eye-tracking: Allows documenting the system points that the user has been always looking [29]. 3) Card Sorting: This technique helps to discover or validate how users understand the relationship between different elements. It consists of giving the participants a series of «cards» to organize items under predetermined categories [46].
4) Test A/B. It consists of comparing two versions of the same system, interface, or application to check which of the two versions is more efficient. These variations, called A and B, are randomly shown to different users [13]. 5) Questionnaires. These instruments allow the software evaluator to retrieve data during the task or after the test. Some of these questionnaires are described below. NASA Task Load Index (NASA TLX) [25] is a technique for assessing mental workload. Derive a general workload based on six subscales. mental demand, physical demand, time demand, performance, effort, and frustration. Visual Analogue Scale (VAS) [11] is a questionnaire to evaluate a "feeling", generally it is carried out to evaluate the satisfaction of a system in the BCI usability studies. Questionnaire for User Interface Satisfaction (QUIS) [9] is a Satisfaction Questionnaire that elicits user feedback and assesses user acceptance of a computer interface. System Usability Scale (SUS) [25] and Utility, Satisfaction and Ease of Use (USE) questionnaire are simple but effective tools to evaluate the usability of various products. Also, the IBM IT [26,44] usability satisfaction questionnaires also measure user satisfaction with usability in a computer system. In another hand, there are three principles or dimensions have been defined for the measurement of usability, they are efficiency, effectiveness, and satisfaction [ISO/IEC TR 9126-4]. Effectiveness refers to the precision and completeness with which certain users achieved specific objectives in a particular environment. Efficiency corresponds to the fact that the system must be efficient to use so that once the user has learned the system, and satisfaction refers to how pleasant it is to use the product. It has also been considered that usability can also be measurable in terms of. «Ease of use», «Learning ability», «Consistency», «Frustration», «Task speed», «Accuracy», etc. [19,22,37,44]. To obtain the dimension of effectiveness, it is obtained through objective measures, which correspond to «How effective and efficient is a system / product», the questions raised are perfectly delimited, the results are quantitative and admit a single solution, taking as an example of measurements the accuracy of the classification, the error rate, the task completion rate, etc. In the case of efficiency, it is achieved through objective and subjective measures, the latter refer to the personal opinions of the user, as an example of the measures on the part of efficiency is mental demand, frustration, effort, and on the other hand satisfaction, measures of ease of use, learn-ability, usefulness, reliability, consistency, etc.
4. Method of Search Process
There are different types of research like quick review, scope review, etc. However, it was decided to conduct a systematic review. Systematic review is a research method and process for identifying and evaluating relevant research, as well as collecting and analyzing data from such research. The goal of a systematic review is to identify all the empirical evidence that meets the inclusion criteria to answer a given research question [45]. In the process of searching and selecting the papers, the Kitchman proposal for systematic reviews was followed [58]. Kitchman's method includes the following phases. 1) Selected database, 2) keywords for the search, 3) Inclusion and exclusion criteria, 4) Selected papers and finally 5) Quality assessment, then the analysis task was carried out (See Fig.1).
For the search and analysis of the related works, the following research questions were defined. 1) How is the evaluation of the BCI software applications carried out?, Which would allow to know the methods and techniques used for the evaluation in the BCI, to understand the phases, stages and instruments used, 2)What have the EEG signals been used for? and what information can be obtained from the signals?, with the objective of know what data or information the EEG signals can provide, in what area and for which the EEG signals have been used, 3) Do the EEG signals provide enough elements (data) to assess usability?; with the purpose of analyze whether the information provided by the signals is sufficient and influence the usability evaluation. The research method for the search for related papers began with the selection of the most important and well-known databases, subsequently, the keywords related to the subject of usability evaluation in the BCI software
applications were defined. Once the search was carried out, those works that were not directly related to the topic of interest were excluded, and the resulting papers were classified by the evaluation.
Fig. 1. Process for search and selection of related papers
In the Selected Database for the search, the information sources IEEE, Springer, Elsevier, ACM, Taylor and Francis, among others, were defined. These bases are the best known, important, and complete, mainly considering the research area of this work.
For the Keywords in the search the papers were searched in the electronic databases, with the following search string: («EEG» and «HCI») OR ((«Brain Computer Interface» OR «BCI») AND («Usability» OR «User Experience» OR» UX «)), using the boolean operators «AND» and «OR» that are used in the formation of the search string in the database, this string was used in the search of each of the databases selected. Those works that were not indexed and published, also those prior to 2005, were discarded.
In order to focus only on papers relevant to the research, it was necessary to define inclusion and exclusion criteria. Which are described below: Inclusion criteria: 1) The paper is related to some dimension or measure of usability or UX and EEG. 2) The paper presents an experimental study on usability or the use of EEGs or on obtaining usability measures. 3)Papers published since 2005-Present. Exclusion criteria: 1) The paper includes the BCI software applications but has no relation to usability. 2) The paper is related to EEG but does not perform the analysis or evaluation of any measure of usability. 3) If you do not present significant evidence or a conclusion. From the search in the different databases, 139 papers were found. Subsequently, those papers that were found to be duplicates were excluded. In case of doubt, the full text versions of the citations were consulted. Resulting in 96 full-text papers evaluated for eligibility.
Continuing with the filtering by each inclusion and exclusion criteria, 18 papers that were not related to usability or UX were excluded, 21 papers that were related to the topic of electroencephalography but that did not carry out the analysis or evaluation of any usability measure, and finally 18 papers that do not present significant evidence for the study. Culminating with 41 primary papers. The Selected papers in Table 1 shows the total number of selected jobs. Column 1 shows the database consulted, column 2 the total number of papers excluded per database, column 3 the number of primary papers and finally column 4 the percentage of primary papers per database. A total of 41 papers were identified. The databases with the highest results were IEEE, Springer and ACM. Numerous context-aware papers have grown considerably since 2010. The number of papers in 2018 has become 7 times the number of papers in 2007.
Table 1. Classification of papers based on the database
Editor #Disc. #Exc #Prim. %
Elsevier 9 6 3 7.32%
IEEE 26 9 17 41.5%
Inderscience Enterprises 6 5 1 2.44%
Springer 17 10 7 17%
Public Library of Science 4 3 1 2.44%
M D P I 5 3 2 4.88%
ACM 8 3 5 12.2%
Others 64 59 5 12.2%
Total Items 139 98 41 100%
In the Quality assessment, each SLR was evaluated using the York University, Centre for Reviews and Dissemination (CDR) A quality assessment tool for diagnostic accuracy studies (QUADAS)[3], using the following criteria: 1) Are the user's representative of the users who will receive the test in practice?, 2) Is the reference standard likely to correctly classify the target condition?, 3) Was the execution of the experimental study described in sufficient detail to allow replication?, 4) Were un-interpretable/ intermediate test results reported?, and 5) Are the data with which the test results were interpreted available?. these criteria consist of 4 key domains that cover the selection of patients, the reference standard, the execution of the test (description and replication), and the interpretation of results.
5. Analysis of Results
In the works found, it was identified that the usability of the ICC was obtained through usability tests, applying techniques such as questionnaires or EEG signals for data collection. In the Data collection by Questionnaires, some questionnaires that were used in BCI usability studies are as follows: NASA-TLX [25], VAS [11], Assistive Technology Device (ATD-PA) Readiness Assessment Device form, SUS survey, QUEST 2.0 Questionnaire, IBM Computing Usability, USE Questionnaire, and QCM Questionnaire. These questionnaires were described in section 3.1. The works of Garcia Ramirez et al [16], and Pasqualotto. et al. [38], used the SUS questionnaire to obtain ease of use. On the other hand, the works of Chowdhury [10] and Laar et al. [24] used the VAS questionnaire to measure the mood of users. And finally, Pasqualotto et al. [38] and Laar et al. [24] measured workload using the NASA tlx questionnaire. Finally, some studies proposed and carried out their own evaluation tools [44], to obtain measures such as comfort and mood, in a precise way, which may not be possible when using existing questionnaires. Table 2 shows works that have used questionnaires to collect data that measure different elements of efficiency, effectiveness, and satisfaction. Table 2. Subjective measures obtained by questionnaires in BCI
Dimension Measure Reference
Efficiency Workload [24,38]
Comfort [55]
Frustration [24]
Fatigue [10]
Satisfaction Mood [10]
Learning ability [38]
Easy to use [16, 38]
Motivation [10]
Presence [24]
Fun [24]
On the other hand, data collection by EEG is presented, with respect to EEG signals and frequency bands Delta (0.1 Hz to 3.9 Hz), Theta (4.0 Hz to 7.9 Hz), Alpha (8.0 Hz to 12.9 Hz), Beta (13 Hz to 29.9 Hz) and Gamma (from 30 Hz to 100 Hz) in BCI that have been used to obtain measures such as «Concentration», «Emotions» and «Fatigue». In several works they measure the factors by obtaining the frequency bands of the signals, in most they perform the combination of the different bands. For the level of concentration, in the work of Wang et al. [54], they get the concentration level in the entertainment area, reporting the experiments they carried out and identifying that the «01» channel and the combination of all bands help in the detection. In relation to «emotions», the following works were found. In Garcia's work [16] performs emotion detection through EEG signals of a BCI software application for people with cerebral palsy is described, the main contribution of this work is the method used to obtain the emotions of the users during their interaction with BCI,
and it's identified the channels F3 and F4 with the band Alpha are the data that provides or influences the most for the detection of emotions. Another work is of the Sourina et al. [36] where emotion detection is performed and describes some examples where it can be used such as «Emotional Avatar», «website», «music reproductions», etc., using 14 channels and the combination of all bands. And finally, regarding ' fatigue", the work of Arai et al. [4], performs the load measurement with the Alpha band.
Regarding the use of signals, the brain processes that reflect cognitive and attention states during human-machine interaction are studied extensively with EEG. Therefore, the signals have been occupied in obtaining measures such as: "Workload", "Comfort", "Attention", "Stress", "Mistakes" and "Emotions" mainly in the areas of Neuromarketing and clinical studies. In the field of neuromarketing, economists use EEG research to detect brain processes that drive consumer decisions, brain areas that are active, and mental states [50]. In clinical and psychiatric studies, EEG is used to assess the cognitive states of patients, determine sites of lesions, and symptoms [16]. In the workload, some examples of these works are. 1) The work of Kumar et al. [23] carried out the measurement of the workload of cognitive tasks, obtaining with this study which are the channels that most influence to make a correct classification, being AF3, AF4, T7, and T8. 2) The works of Appriou et al. [3] occupying 28 active electrodes in the 10/20 system. And 3) Frey et al. [14] use 32 channels, and Antonenko obtains the workload with the channels «F7 and P3». For "comfort", in the work by Frey et al. [15] they use the 32 active channels. Regarding "attention", in the work of Putze et al [40], they detected that the channels «P8, CP6, and 02» are the ones that contribute the most or influence. In the "stress", the work of Hosseini et al. [19], occupying the channels FP1, FP2, T3, T4, and Pz provides enough information for stress measurement. And finally, regarding "emotions", the work of Ansari et al. [1] given that there are already several studies for the detection of emotions, its main attribution is the selection of the channels that most influence their classification/EEG detection, making the processing of count is lower, selecting the channels F3, F4, CP5, CP6 those that influence the most. The works that carry out the obtaining of measurements by means of EEG are presented in table 3, classifying the works by signal, in this the usability module that is being evaluated is presented, what measure are they obtaining, what are the signals that they are occupying. Table 3. Measures obtained by EEG signals
Measures Signals Reference
Workload AF3, AF4, T7 and T8 [23]
28 channels [3]
32 channels [14]
F7, P3 [2]
Comfort 32 C- Pz [15]
Attention 01 [54]
P8, CP6 y 02 [40]
32 channels [14]
Stress FP1, FP2, T3, T4, Pz [19]
Mistakes 64 channels [37]
Emotions 14 channels [8, 22, 36, 41]
F3, F4 [16]
AF3, F4 and FC6 [27]
32 channels [20, 56]
AFz, F3, F4, CP5, CP6 [1]
63 channels [32]
AF3, AF4, F3, F4 [42]
AF3, AF4, F3, F4, T7, T8 [21]
FP1, F3, P3, O1 [6]
P3, P6, P7 and PO8 [30]
On the other hand, the frequency bands have been used to obtain the measurements: «workload, fatigue, comfort, attention, stress, and emotions». To obtain the workload, Antonenko et al. [2] occupied the Theta and Alpha bands, on the other hand, Appriou et al. [3] only occupied the alpha band, and Frey et al. [14] when performing a combination of the 5 bands. Regarding fatigue, Arai et al. [4] and Mardiyanto [5] determined that the alpha band is decisive. About comfort, for obtaining Frey et al. [15] occupies the Theta, Alpha and Beta bands. In attention, the Alpha [40], Delta [23] have been dealt with, apart from the works of Frey [14] and Wang [54]. And finally for emotions, theta, alpha, beta, and gamma bands have been used, there are also works such as the one by Kortelainen et al.[20] and Liu et al. [27] that use all 5 frequency bands. The table 4 shows the classification of the works by frequency bands. This table presents the usability module that you are evaluating, what measures are you obtaining, what are the occupied bands and the job (s).
Table 4. Measures obtained by bands
Measures Bands Reference
Theta [2, 23]
Workload Alpha [2, 3, 23]
Beta [22, 23]
Gamma [22]
Comfort All [14]
Fatigue Comfort Alpha [4, 5]
Theta [15]
Alpha [15]
Stress Beta [15]
Attention All [14, 54]
Alpha [40]
Delta [23]
All [19]
Emotions Stress Emotions Measures Workload Theta [6, 22]
Alpha [6, 16, 21, 22, 31, 32, 42]
Beta [6, 21, 22, 31, 42]
Gamma [31]
All [20, 27, 30, 36, 41, 56]
Bands Reference
Theta [2, 23]
6. Discussion
According to the analysis of the literature, it is observed that evaluation of BCI software applications, are carried out through usability tests and mainly are used questionnaires for data collection. Usability tests are carried out in three stages: 1) carrying out the task, 2) data collection through questionnaires and 3) data analysis. This evaluation method allows the collection of subjective data that comes from the opinions of users, this can cause a certain bias. It is also observed that some works incorporate EEG signals to complement the evaluation by measuring emotions.
Studies in other areas such as marketing and medicine use EEG signals and frequency bands to measure workload, fatigue, and emotions. Delta bands allow to detect retention and concentration. Several studies analyze Alpha, Beta, Theta bands in combination for the detection of emotions, comfort, and workload level. These works have allowed us to identify how EEG signals and frequency bands can be linked to usability measures in the field of efficiency and satisfaction, since they have been used in other fields with favorable results. Table 5 shows a summary of the signals and bands, linking them to the measurements and therefore to the respective usability dimension. classifying by dimension, measure, signals, and bands.
Table 5. EEG signals and frequency bands linked to usability measures
Dimension Measures Signals Bands
F7, P3 Theta
28 channels Alpha
Workload 32 channels Beta
AF3, AF4, T7 and T8 Gamma
Efficiency All
Fatigue 14 channels Alpha
Theta
Comfort 32 C- Pz Alpha
Beta
01 All
Attention P8, CP6 y 02 Alpha
32 channels Delta
Stress FP1, FP2, T3, T4, Pz All
14 channels Theta
F3, F4
AF3, F4 and FC6 Alpha
32 channels
AFz, F3, F4, CP5 and CP6 Beta
Satisfaction Emotions 63 channels
AF3, AF4, F3, F4 Gamma
AF3, AF4, F3, F4, T7, T8
FP1, F3, P3, 01 All
P3, P6, P7 and P08
7. Conclusions
A systematic and exhaustive review was carried out [58], defining the research questions, keywords, inclusion, and exclusion criteria, and subsequently the analysis of the results. This research allowed to know how the evaluation process is carried out in the BCI software applications, to identify the use of EEG signals, what information can be obtained from them and if they provide enough elements (data) to be able to occupy them in the evaluation process. Usability evaluations of BCI software applications are carried out through usability test, using questionnaires mainly for data collection. However, the main problem we observe is that the questionnaires give subjective answers, without the certainty of precision and based on the user's opinion. When comparing both techniques for data collection: questionnaires and EEG Signals, it is appreciated that the main advantage of using EEG signals objective data are collected, to measure workload, emotions, and concentration. On the other hand, both in the BCI software applications and in other domains like
Neuromarketing and clinical studies, the EEG signals have been used mainly, but EEG signals have not been applied for usability evaluation.
Our analysis reveals that the EEG signals have been used to measure usability, because studies have used the EGG signals, in order to measure workload, fatigue, attention, comfort and emotions, all these human factors that have been considered elements of efficiency and effectiveness. Given the results of this work, the research lines are the following: 1) to explore the possibility to measures other human factors through EEG signals like Learning, Utility, Predictability, Consistency, Reliability, Adaptability, Effort, etc., 2) to know what algorithms are used in EEG analysis to obtain measures such as SVM (Vector Support Machine), NN (Neural Network), ICA (Independent Component Analysis), FFT (Fast Fourier Transform), and 3) to apply in BCI software applications EEG signals and bands used in other domains.
References / Список литературы
[1]. Ansari-Asl K., Chanel G., & Pun T. A channel selection method for EEG classification in emotion assessment based on synchronization likelihood. In Proc. of the 2007 15th European Signal Processing Conference, 2007, pp. 1241-1245.
[2]. Antonenko P., Paas F. et al. Using electroencephalography to measure cognitive load. Educational psychology review, vol. 22, no. 4, 2010, 425-438.
[3]. Appriou A., Cichocki A., & Lotte F. Towards robust neuroadaptive HCI: exploring modern machine learning methods to estimate mental workload from EEG signals. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1-6.
[4]. Arai K. Evaluation of users' impact for using the proposed eye based HCI with moving and fixed keyboard by using eeg signals. International Journal of Research and Reviews in Computer Science, vol. 2, no. 6, 2015, 1-7.
[5]. Arai K., & Mardiyanto R. Eye based HCI with moving keyboard for reducing fatigue effects. In Proc. of the 2011 Eighth International Conference on Information Technology: New Generations, 2011, pp. 417422.
[6]. Bhardwaj A., Gupta A. et al. Classification of human emotions from EEG signals using SVM and LDA Classifiers. In Proc. of the 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), 2015, pp. 180-185.
[7]. Bos D.P.O., Reuderink B. et al. Human-computer interaction for BCI games: Usability and user experience. In Proc. of the 2010 International Conference on Cyberworlds, 2015, pp. 277-281.
[8]. Charisis V., Hadjidimitriou S. et al. EmoActivity-An EEG-based gamified emotion HCI for augmented artistic expression: The i-Treasures paradigm. Lecture Notes in Computer Science, vol. 9178, 2015, pp. 29-40.
[9]. Chin J.P., Diehl V.A., & Norman K. L. Development of an instrument measuring user satisfaction of the human-computer interface. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, 1988, pp. 213-218.
[10]. Chowdhury A., Meena Y.K. et al. Active physical practice followed by mental practice using BCI-driven hand exoskeleton: a pilot trial for clinical effectiveness and usability. IEEE Journal of Biomedical and Health Informatics, vol. 22, no. 6, 2018, 1786-1795.
[11]. Crichton N. Visual analogue scale (VAS). Journal of Clinical Nursing, vol. 10, no. 5, 2001, pp. 697-706.
[12]. Erkan E., & Akbaba M. A study on performance increasing in SSVEP based BCI application. Engineering Science and Technology, an International Journal, vol. 21, issue 3, 2018, 421-427.
[13]. Freidman V., & Mielke C. A field guide to Usability Testing. Smashing Magazine, 2013, 85 p.
[14]. Frey J., Daniel M. et al. Framework for electroencephalography-based evaluation of user experience. In Proc. of the 2016 CHI Conference on Human Factors in Computing Systems, 2016, pp. 2283-2294.
[15]. Frey J., Pommereau L. et al. Assessing the zone of comfort in stereoscopic displays using EEG. In CHI'14 Extended Abstracts on Human Factors in Computing Systems, 2014, pp. 2041-2046.
[16]. Garcia Ramirez A.R., Da Silva J.F. et al. User's emotions and usability study of a brain-computer interface applied to people with cerebral palsy. Technologies, vol. 6, no. 1, 2018, article no. 28, 13 p.
[17]. Gentiletti G., Tabernig C., & Acevedo R. Interfaz cerebro-computadora: Estado del arte y desarrollo en Argentina. Revista Argentina de Bioingenieria, Revista SABI, vol. 13, no. 1, 2007, pp. 22-29 (in Spanish).
[18]. Hartson H.R., Andre T.S., & Williges R.C. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, vol. 15, no. 1, 2013, 145-181.
[19]. Hosseini S.A., & Khalilzadeh M.A. Emotional stress recognition system using EEG and psychophysiological signals: Using new labelling process of EEG signals in emotional stress state. In Proc. of the 2010 International Conference on Biomedical Engineering and Computer Science, 2010, pp. 1-6.
[20]. Kortelainen J., & Seppanen T. EEG-based recognition of video-induced emotions: selecting subject-independent feature set. In Proc. of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2013, pp. 4287-4290.
[21]. Kosinski J., Szklanny K. et al. An analysis of game-related emotions using Emotiv EPOC. In Proc. of the 2018 Federated Conference on Computer Science and Information Systems (FedCSIS), 2018, pp. 913917.
[22]. Kumar, J. Affective modelling of users in HCI using EEG. Procedia Computer Science, vol. 84, 2016, pp. 107-114.
[23]. Kumar N., & Kumar J. Measurement of cognitive load in HCI systems using EEG power spectrum: an experimental study. Procedia Computer Science, vol. 84, 2016, pp. 70-78.
[24]. Laar B.V.D., Gürkok H. et al. Brain-computer interfaces and user experience evaluation. In Towards Practical Brain-Computer Interfaces, Springer, Berlin, Heidelberg, 2012, pp. 223-237.
[25]. Laubheimer P. Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests. Nielsen Norman Group, 2018. URL: https://www.nngroup. com/articles/measuring-perceived-usability/.
[26]. Lewis J.R. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, vol. 7, no. 1, 1995, pp. 57-78.
[27]. Liu Y., Sourina O., & Nguyen M.K. Real-time EEG-based human emotion recognition and visualization. In Proc. of the 2010 International Conference on Cyberworlds, 2010, pp. 262-269.
[28]. Lührs M., & Goebel R. Turbo-Satori: a neurofeedback and brain-computer interface toolbox for real-time functional near-infrared spectroscopy. Neurophotonics, vol. 4, issue 4, 2017, article no. 041504, 11 p.
[29]. Massa S.M., De Giusti A.E., & Pesado P.M. Métodos de evaluación de usabilidad: una propuesta de aplicación en Objetos de Aprendizaje. In Proc. of the Workshop de Investigadores en Ciencias de la Computación, vol. 14, 2012, pp. 922-926 (in Spanish).
[30]. Murugappan M., Juhari M.R. et al. An Investigation on visual and audiovisual stimulus based emotion recognition using EEG. International Journal of Medical Engineering and Informatics, vol. 1, no. 3, 2009, pp. 342-356.
[31]. Murugappan M., & Murugappan S. Human emotion recognition through short time Electroencephalogram (EEG) signals using Fast Fourier Transform (FFT). In Proc. of the 2013 IEEE 9th International Colloquium on Signal Processing and its Applications, 2013, pp. 289-294.
[32]. Murugappan M., Rizon M. et al. Lifting scheme for human emotion recognition using EEG. In Proc. of the 2008 International Symposium on Information Technology, 2008, vol. 2, pp. 1-7.
[33]. Nielsen J. Usability inspection methods. In Proc. of the Conference Companion on Human Factors in Computing Systems, 1994, pp. 413-414.
[34]. Nielsen J. Usability Engineering. Morgan Kaufmann, 1993, 376 p.
[35]. Ortega-Gijón Y.N., & Mezura-Godoy C. Usability evaluation process of brain computer interfaces: an experimental study. In Proc. of the IX Latin American Conference on Human Computer Interaction, 2019, pp. 1-8.
[36]. Sourina O., Liu Y. et al. EEG-based personalized digital experience. Lecture Notes in Computer Science, vol. 6766, 2011, pp. 591-599.
[37]. Parra L.C., Spence C.D. et al. Response error correction-a demonstration of improved human-machine performance using real-time EEG monitoring. IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 11, no. 2, 2003, 173-177.
[38]. Pasqualotto E., Federici S. et al. Usability of brain computer interfaces. In Everyday Technology for Independence and Care. IOS Press, 2011, pp. 481-488.
[39]. Pradhapan P., GroBekathofer U. et al. Toward practical BCI solutions for entertainment and art performance. In Brain-Computer Interfaces Handbook: Technological and Theoretical Advances, CRC Press, 2018, pp. 65-115.
[40]. Putze F., Scherer M., & Schultz T. Starring into the void? Classifying Internal vs. External Attention from EEG. In Proc. of the 9th Nordic Conference on Human-Computer Interaction, 2016, pp. 1-4.
[41]. Puwakpitiyage C.A., Rao V.R. et al. A Proposed Web Based Real Time Brain Computer Interface (BCI) System for Usability Testing. International Journal of Online & Biomedical Engineering, vol. 15, no. 8, 2019, pp. 111-123.
[42]. Ramirez R., & Vamvakousis Z. Detecting emotion from EEG signals using the emotive epoc device. Lecture Notes in Computer Science, vol. 7670, 2012, pp. 175-184.
[43]. Nielsen Norman Group. Usability Testing 101. URL: https://www.nngroup.com/articles/usability-testing-101/.
[44]. Rhiu I., Lee Y. et al. Toward usability evaluation for brain-computer interfaces. In Brain-Computer Interfaces Handbook, CRC Press, 2018, pp. 563-584.
[45]. Snyder H. Literature review as a research methodology: An overview and guidelines. Journal of Business Research, vol. 104, 2019, pp. 333-339.
[46]. Spencer D., Warfel T. Card Sorting. Boxes and arrows: A Definitive Guide, 2014. URL: https://boxesandarrows.com/card-sorting-a-definitive-guide/.
[47]. Stein A., Yotam Y. et al. EEG-triggered dynamic difficulty adjustment for multiplayer games. Entertainment computing, vol. 25, 2018, pp. 14-25.
[48]. Spüler M. A high-speed brain-computer interface (BCI) using dry EEG electrodes. PloS one, vol. 12, no. 2, 2017, article no. e0172400, 12 p.
[49]. Taherian S., Selitskiy D. et al. Are we there yet? Evaluating commercial grade brain-computer interface for control of computer applications by individuals with cerebral palsy. Disability and Rehabilitation: Assistive Technology, vol. 12, no. 2, 2017, pp. 165-174.
[50]. Theofanos M., & Quesenbery W. Towards the design of effective formative test reports. Journal of Usability Studies, vol. 1, no. 1, 2005, pp. 27-45.
[51]. Tidoni E., Abu-Alqumsan M. et al. Local and remote cooperation with virtual and robotic agents: a P300 BCI study in healthy and people living with spinal cord injury. IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, no. 9, 2017, pp. 1622-1632
[52]. Valderrama C.E., & Ulloa G. Spectral analysis of physiological parameters for emotion detection. In Proc. of the 2012 XVII Symposium of Image, Signal Processing, and Artificial Vision (STSIVA), 2012, pp. 275-280.
[53]. Villegas A., Salvatierra E. et al. Reconocimiento de patrones de actividad cerebral asociados a tareas mentales mediante RNA para una interfaz cerebro computador. Revista Ingeniería UC, vol. 15, no. 1, 2008, 88-92 (in Spanish).
[54]. Wang Q., Sourina O., & Nguyen M. K. Eeg-based «serious» games design for medical applications. In Proc. of the 2010 International Conference on Cyberworlds, 2010, pp. 270-276.
[55]. Xing X., Wang Y. et al. A high-speed SSVEP-based BCI using dry EEG electrodes. Scientific reports, vol. 8, no. 1, 2018, pp. 1-10.
[56]. Zhang J., Chen M. et al. PNN for EEG-based Emotion Recognition. In Proc. of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016, pp. 002319-002323.
[57]. Kurniawan, S. Interaction design: Beyond human-computer interaction by Preece, Sharp and Rogers. Universal Access in the Information Society, vol. 3, issue 3-4, 2001, p. 289.
[58]. Kitchenham B., Brereton O.P. et al. Systematic literature reviews in software engineering-a systematic literature review. Information and Software Technology, vol. 51, no. 1, 2009, pp. 7-15.
[59]. Tello-Rodríguez M., Ocharán-Hernández J.O. et al. A design guide for usable web APIs. Programming and Computer Software, vol. 46, issue 8, 2020, pp. 584-593.
[60]. Lukin V.N., Dzyubenko A.L., & Chechikov Y.B. Approaches to user interface development. Programming and Computer Software, vol. 46, issue 5, 2020, pp. 316-323 / Лукин В.Н., Дзюбенко А.Л., Чечиков Ю.Б. Подходы к разработке пользовательского интерфейса. Программирование, том 46, no. 5, 2020 г., стр. 16-24.
Information about authors / Информация об авторах
Yoselyn Nohemí ORTEGA-GIJÓN - PhD student in Computer Science from the Universidad
Veracruzana. Master's in Computer Systems from the Instituto Tecnológico Superior de Misantla.
Main research interests: Brain Computer Interaction, Usability evaluation, EEG processing signals.
Йоселин Нохеми ОРТЕГА-ХИХОН - аспирант. Степень магистра в области компьютерных
систем получила в Высшем технологическом институте в Мисантле. Основные научные
интересы: взаимодействие мозга с компьютером, оценка пригодности к использованию, обработка сигналов ЭЭГ.
Carmen MEZURA-GODOY - PhD in Computer Science from the University of Savoie in France. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Main research interests: Human-Computer Interaction, User eXperience-UX, Computer Support Collaborative Work, Visualization and Multiagent Systems.
Кармен МЕЗУРА-ГОДОЙ получила степень доктора компьютерных наук в Университете Савойи во Франции. Профессор факультета статистики и информатики Университета Веракруса в Мексике. Основные исследовательские интересы: взаимодействие человека и компьютера, компьютерная поддержка совместной работы, визуализация и мультиагентные системы.
