CC BY-NC-ND
92
Accessibility of Mathematics MOOCs to Learners with Disabilities
Y. Kosova, M. Izetova
Received in Yekaterina Kosova
June 2019 Candidate of Sciences in Pedagogy, Associate Professor, Department of Applied Mathematics, Faculty of Mathematics and Computer Science, Tavrida Academy, V. I. Vernadsky Crimean Federal University. Email: lynx99@inbox.ru Milera Izetova
Master's Degree Student, Department of Applied Mathematics, Faculty of Mathematics and Computer Science, Tavrida Academy, V. I. Vernadsky Crimean Federal University. Email: milera16@mail.ru
Address: 4 Akademika Vernadskogo Ave, 295007 Simferopol, Republic of Crimea.
Abstract. Fifty-six mathematics Massive Open Online Courses (MOOCs) for Bachelor's degree students were sampled by browsing through the course lists of five leading MOOC platforms offering learning content in Russian. Accessibility of the sampled MOOCs to persons with disabilities was evaluated by experts using 70 predetermined criteria. No accessibility issues related to hyperlink behavior, quality of sound reproduction, visualization in graphical web browsers, mobile user experience, background and text contrast, or keyboard operation were revealed during analysis. However, it transpired that
none of the MOOCs allowed customizing user interface or displaying content in text-based web browsers. In addition, 98% of digital documents, 82% of mathematical notation, and 91% of tests are rendered unfaithfully by screen readers; captions are unavailable in 64%, transcripts in 66%, and lecture notes in 52% of the MOOCs; no MOOC offers sign interpretation for their video; and audio alone is not sufficient for adequate perception of the content in any MOOC. The findings reveal low accessibility of mathematics MOOCs in Russian to people with disabilities, particularly those with severe visual impairments, and illustrate the need to bring the existing MOOCs into compliance with the Web Content Accessibility Guidelines (WCAG 2.1), ensure that platform administrators conform strictly to WCAG 2.1 before posting new MOOCs, teach MOOC authors and developers to create accessible course content, and involve persons with disabilities in beta testing of MOOCs. Keywords: MOOCs, mathematics education, e-learning, distance learning technology, web accessibility, persons with disabilities.
DOI: 10.17323/1814-9545-2020-1-205-229
Massive Open Online Courses (MOOCs) are the most promising tool for teaching people with disabilities, as they imply distance learning without actual learner presence in the classroom [Policar, Crawford, Alligood 2017; Kent 2015; De Waard et al. 2014].
MOOC developers should consider the needs of vulnerable learner groups, people with disabilities in the first place [De Waard et al. 2014]. Providers and developers are required to make MOOC content accessible to student with disabilities under a number of national and international legal instruments: the Universal Declaration of Human Rights (1948)1, the Convention on the Rights of Persons with Disabilities (2006)2, Federal Law No. 181-FZ On Social Protection of People with Disabilities in the Russian Federation of November 24, 19953, Federal Law No. 273-FZ On Education in the Russian Federation of December 29 , 20124, etc. In particular, according to Article 19 (Education of People with Disabilities) of the Federal Law No. 181-FZ, "the State shall promote education for persons with disabilities and guarantee to provide the necessary conditions." Article 79 (Organization of Education for Learners with Disabilities) of Federal Law No. 273-FZ obligates vocational schools and higher education institutions to create "special conditions to facilitate learning for people with disabilities". Article 9.13 of the Code of Administrative Offenses of the Russian Federation imposes administrative liability on officials for not providing access to services, social, engineering and transport infrastructure for people with disabilities. However, accessibility of MOOCs is not prescribed straightforwardly in any Russian law or bylaw.
Adaptation of MOOC content to vulnerable learners' needs may be defined, according to the Convention on the Rights of Persons with Disabilities, as "reasonable accommodation", i. e. "necessary and appropriate modification and adjustments not imposing a disproportionate or undue burden, where needed in a particular case, to ensure to persons with disabilities the enjoyment or exercise on an equal basis with others of all human rights and fundamental freedoms", whereas failure to meet web accessibility requirements may be regarded as disability discrimination.
In some countries (Australia, Great Britain, the European Union, the United States, and others), national web accessibility standards are based on Web Content Accessibility Guidelines (WCAG)— WCAG 2.0 (2008)6 or WCAG 2.1 (2018)7—and have the force of law [AHRC2014; BSI 2010; Eur-Lex 2016; GSA 1998, and others]. The regulations apply to all products and services offered through web browsers, including learning resources such as MOOCs.
Students, regardless of their disability status, should have no problems viewing learning materials, doing assignments, communi-
http://www.cons ultant.ru/document/cons doc LAW 120805/
https://www.un.org/ru/docume its/decl conv/conventions/disability.shtml
http://www.cons ultant.ru/document/cons doc LAW 8559/
http://www.cons ultant.ru/document/cons doc LAW 140174/
http://www.cons ultant.ru/document/cons doc LAW 34661/
https://www.w3.i org/TR/WCAG20/
https://www.w3.i org/TR/2018/REC-WCAG21-20180605/
cating in forums, and obtaining knowledgeable support from tutors. Therefore, MOOCs should comply with WCAG, the principles of universal design [NC State University 1997], and MOOC platform recommendations on creating accessible content8.
Research on the accessibility of MOOCs in mathematics requires special attention, as people with disabilities often tend to gravitate toward science, technology, engineering and mathematics (STEM) fields, which is especially true of individuals with a high-functioning autism spectrum disorder [Wei et al. 2013]. Mathematical and programming jobs provide autonomy and independence, allowing to work remotely and avoid working conditions contra-indicated for people with disabilities. Jobs like mathematician, software engineer, software technician, mathematical researcher, computer information researcher, or university lecturer are recommended for people with impaired vision, hearing, locomotor activity and blood circulation9.
Demand for MOOCs in mathematical fields is one of the highest [Semenova, Vilkova, Shcheglova 2018]. Meanwhile, it is harder to bring mathematical content into compliance with accessibility requirements as compared to content that has no scientific notation or visual elements [Lowe, Mestel, Williams 2016; Ramírez-Vega, Iniesto, Co-vadonga 2017]. Significance of research on web content accessibility in mathematics has been proved by creation of a specialized working group of the World Wide Web Consortium (W3C) that mainly seeks to disseminate the best modern web design practices in the semantic markup of mathematical and scientific content10.
No comprehensive research has been done on web accessibility of mathematics MOOCs in Russia so far. This article aims at evaluating content accessibility of mathematics MOOCs in Russian to persons with disabilities.
1. The Concept Web accessibility is understood as such organization of web content of Web where a website can be accessed by as many user groups as possi-Accessibility ble. The more people have the opportunity to use web content, the
8 Development Requirements and Recommendations for Online Courses Posted on the National Open Education Platform (http://npoed.ru/docs); Accessibility Best Practices Guidance for Content Providers (https://edx. readthedocs.io/projects/edx-partner-course-staff/en/latest/accessibility/ index.html#); Accommodations for learners with disabilities (https://learner. coursera.help/hc/en-us/articles/208280056-Accommodations-for-learn-ers-with-disabilities).
9 Order of the Ministry of Labor of the Russian Federation No. 515 On Approval of Guidelines on the List of Recommended Types of Work and Professional Activity of Persons with Disabilities Taking into Account the Impaired Functions and Limitations of Their Life Activity (https://rosmintrud.ru/docs/ mintrud/orders/268).
10 W3C Math Home (https://www.w3.org/Math).
more accessible content is considered to be [Carter, Markel 2001; W3C2012]. Individual differences, including medical conditions, can affect the possibility of getting access to web content. The following clusters of disabilities that can be barriers to access are identified: blindness, other visual impairments (including limited vision and colorblindness), deafness (including hearing loss), speech impairments, mobility impairments (including hand motor impairments), cognitive impairments (including specific learning disabilities), and neurological disorders (including seizure disorders) [Carter, Markel 2001; W3C2012; Burgstahler 2015]. Web accessibility implies creating a learning environment that ensures compatibility with assistive technologies, such as screen magnifiers, scanners, screen readers, voice-to-text technologies, Braille translators, and other software and hardware [Wentz, Jaeger, Lazar 2011]. Web accessibility refers to web design that will allow people to perceive, understand, navigate and interact with the Web, contributing with content [Luján-Mor, 2013; Acosta et al. 2018].
The international web content accessibility guidelines (WCAG) are developed by W3C within the framework of Web Accessibility Initiative (WAI). Generally accepted on the web as constitutive documents on web accessibility, they represent a set of guidelines on how to make web content perceivable, operable, understandable and robust to all users regardless of their ability or disability. The W3C Working Group elaborated the Website Accessibility Conformance Evaluation Methodology (WCAG-EM)11 that provides guidance on evaluating how well websites conform to the WCAG.
MOOC accessibility is assessed using several methods: automatic accessibility checking tools [Iniesto, Rodrigo 2014; Ferati, Mripa, Bunjaku 2016; Martin, Amado-Salvatierra, Hilera 2016; Ramírez-Vega, Iniesto, Covadonga 2017; Akgül 2018; Kosova, Khalilova 2019], expert evaluation using manual, audio and visual accessibility tests [Al-Mouh, Al-Khalifa, Al-Khalifa 2014; Iniesto, Rodrigo 2014; Ferati, Mripa, Bunjaku 2016; Martin, Amado-Salvatierra, Hilera 2016; Ramírez-Vega, Iniesto, Covadonga 2017; Shutova 2018], disability simulators [Iniesto, Rodrigo 2014], and testing by people with disabilities [Bohnsack, Puhl 2014]. Lately, web accessibility has been used as a criterion in rankings of MOOC quality [Iniesto, Rodrigo 2016].
Initially, MOOC technologies were developed without paying adequate attention to barriers for learners with disabilities [McAndrew, Gruszczynska 2013]. Contemporary findings demonstrate low accessibility of MOOCs and MOOC platforms for such students. Heuristic evaluations and expert reviews show that Coursera MOOCs do not conform to WCAG 2.0 [Al-Mouh, Al-Khalifa, Al-Khalifa 2014]. Anoth-
11 Website Accessibility Conformance Evaluation Methodology 1.0 (https://www. w3.org/TR/WCAG-EM/).
er study assessed the accessibility of five MOOC platforms and found EdX to be the only one entirely accessible to blind people [Bohnsack, Puhl 2014]. In yet another study, automatic testing and expert analysis of eight MOOC platforms revealed that all the providers except EdX and FutureLearn had major accessibility problems [Martin, Amado-Salvatierra, Hilera 2016]. None of the Spanish MOOC platforms were found accessible to users with disabilities, especially people with severe visual impairments [Iniesto, Rodrigo 2014]. Similar findings were obtained by researchers investigating Albania's MOOC platform Almooc [Ferati, Mripa, Bunjaku 2016] and testing the accessibility of three Turkish MOOC providers [Akgul 2018]. Visual evaluation of certain pages of four MOOCs in Russian revealed low accessibility of the courses to users with sensory impairments [Shutova 2018]. Numerous violations of WCAG 2.0 were discovered by automatic tests evaluating web accessibility of Russian mathematics MOOCs [Kosova, Khalilova 2019].
In the recent years, some studies [W3C2014; Iniesto et al. 2017; Gay, Djafarova, Zefi 2017] have focused on auditing the accessibility of MOOCs to people with disabilities. It appears reasonable to profile learner preferences and needs in order to personalize accessible MOOCs for students with disabilities [Iniesto, Rodrigo 2016]. A software design for MOOC platforms (using the case of EdX) has been proposed in order to conceptualize, design, build and test accessible MOOCs [Sanchez-Gordon, Lujan-Mora 2016]. Investigations based on a series of interviews with individuals involved in the MOOC development suggest some progress in producing universally accessible MOOCs and tailoring the existing MOOCs to meet the needs of learners with disabilities [Iniesto et al. 2016].
Therefore, despite being open and cost-free, modern MOOCs and MOOC platforms fail to provide equal access to their content for vulnerable learner groups, particularly people with severe visual impairments.
2. Data and Courses were sampled by browsing through the course lists of five Methods leading MOOC platforms offering content in Russian: Open Education^, Lektorium13, Universarium14, Coursera15, and Stepik16. Sampling was performed based on the criterion of MOOC applicability in the Bachelor's degree majors of Mathematics, Applied Mathematics, and Applied Mathematics & Computer Science. The resulting sample
12 https://Open Education.ru/
13 https://www.lektorium.tv/
14 https://universarium.org/
15 https://www.coursera.org/
16 https://welcome.stepik.org/ru
was comprised of 56 free online courses in Russian that were delivered during the period of research (January-March 2019).
The methodology of expert analysis implied manual evaluation carried out by two experts who tested MOOCs in the Google Chrome, Mozilla Firefox, Microsoft Edge, Opera and Lynx web browsers along 70 criteria developed in accordance with WCAG 2.1", the content accessibility best practices provided by EdX18, the program practices to ensure access to students with disabilities by Sheryl Burgstahler [Burgstahler 2019], and the web accessibility design considerations developed by the Center for Persons with Disabilities at Utah State University [WebAIM 2019a; Ramirez-Vega, Iniesto, Covadonga 2017]. The criteria checklist contained six modules depending on the type of content assessed: general accessibility of MOOCs (15 criteria), accessibility of multimedia material (20 criteria), accessibility of digital documents (9 criteria), accessibility of tests (6 criteria), accessibility of mathematical notation (15 criteria), and accessibility of modelling and simulation (5 criteria). The latter module was not included in analysis as none of the courses contained simulations.
Expert evaluation consisted in assessing each characteristic in the set of data on the Yes / No / Partially / N/A scale. To validate expert evaluation, independent parallel testing of seven random MOOCs from different providers was carried out by two experts at the design stage, with subsequent comparison of the findings. The experts—the authors of this study — used the 70 pre-developed criteria and the same hardware and software. Identical results were obtained, proving the method of manual analysis reliable.
A body of 56 records (equal to the number of MOOCs) obtained as a result of web accessibility tests was managed with IBM SPSS Statistics 23.0 and presented as descriptive statistics (data aggregated into charts and graphs, frequency analysis, contingency table analysis).
The following assumptions and limitations applied:
1. Accuracy of data visualization was tested for the 800x600, 1024x768, and 1280x1024 screen resolutions.
2. Background and text/graphics contrast ratio was tested using the Color Contrast Checker tool [WebAIM 2019b], with the minimum required contrast ratio of 4:1.
3. Mobile website experience was tested using the Web Developer emulator in Google Chrome and by directly viewing web pages from Android-based smartphone and tablet.
4. Media player and screen reader compatibility was tested using the ChromeVox extension in Google Chrome.
17 https://www.w3.org/TR/2018/REC-WCAG21-20180605/
18 https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/ac-cessibility/index.html#
5. Text-to-speech accuracy was tested using the ChromeVox extension in Google Chrome, the in-built Read Aloud feature in Microsoft Edge, and the NVDA screen reader. In addition to Google Chrome as the default web browser, Mozilla Firefox was also used to access mathematical content as it includes native MathML support.
6. No auxiliary aids or accessibility software for persons with disabilities were used in analysis, except built-in screen readers.
7. If at least three hours were allocated for testing, the test was deemed to allow unlimited time for taking.
8. Tests requiring good hand-eye coordination involved object moving, object matching, and other tasks where sharp eyes and confident hands are needed.
3. Findings All the MOOCs have an adequate quality of sound reproduction and al-3.1. General low adjusting audio speed and volume. Sufficient background and text accessibility of ratio is provided in every MOOC. All the pages are displayed properly MOOCs on mobile devices and visualized with no loss in quality in the Chrome, Firefox, Opera and Edge web browsers. All content can be accessed with the keyboard alone. All the MOOCs contain video lectures, their media players being compatible with screen readers and all the controls being keyboard accessible.
Hyperlinks are valid and clear in 54 MOOCs (96.4%).
Meanwhile, none of the courses offer user tools to modify font size or color palette. Besides, none of the websites have content accessible via text-based web browsers (tested using Lynx).
Changes in screen resolution result in loss of content organization and readability in 15 MOOCs (26.8%).
3.2. Accessibility A little over one third of the MOOCs provide closed captioning (Ta-of multimedia ble 1), Coursera and Lektorium accounting jointly for 31.2% of the material sample with all of their courses (15 and 3, respectively) coming with closed captions. On Open Education, only two MOOCs (3.6%) are closed-captioned. Four MOOCs on Universarium (7.1%) and 22 on Stepik (39.3%) have no captions at all. Unlike on Coursera or Open Education, captions on Lektorium are created automatically and cannot be downloaded, which affects their quality and limits their further use. Figure 1 shows the distribution of conformance to web accessibility requirements for closed-captioned MOOCs.
Visual fragments essential for the plot have either incomplete audio description or none at all in 51 MOOCs (91.1%). Meanwhile, the audio alone in every MOOC is not enough to ensure adequate perception of the content.
Lecture transcripts are provided by one third of the MOOCs assessed, including 15 on Coursera (26.8%), three on Open Education (5.4%), and one Ha Lektorium (1.8%). All the transcripts on Coursera and two on Open Education (3.6%) are downloadable. Visual descrip-
Table 1. Accessibility evaluation of multimedia material in mathematics MOOCs (N = 56)
Evaluation Criterion Satisfied, % (n) Not satisfied, % (n) Partially Satisfied, % (n) N/A, % (n)
Captions are available 35.7 (20) 64.3 (36) 0 0
Captions are generated automatically 5.4 (3) 30.4 (17) 0 64.3 (36)
Captions are appropriate and reflect the audio faithfully 30.4 (17) 5.4 (3) 0 64.3 (36)
All captions remain on the screen for at least two seconds 10.7 (6) 25.0 (14) 0 64.3 (36)
There are no more than two lines per caption 33.9 (19) 1.8 (1) 0 64.3 (36)
There are no more than 45 characters in each line 26.8 (15) 8.9 (5) 0 64.3 (36)
All captions are closely synchronized with the audio 33.9 (19) 1.8 (1) 0 64.3 (36)
If there are multiple speakers, captions indicate a speaker change 0 0 0 100 (56)
Background sound effects essential for the plot are designated in square brackets 1.8 (1) 14.3 (8) 10.7 (6) 73.2 (41)
Subtitle files are available for download 30.4 (17) 5.4 (3) 0 64.3 (36)
All visual fragments essential for the plot are described in words by the speaker 8.9 (5) 48.2 (27) 42.9 (24) 0
Audio-only is sufficient for adequate perception of the content 0 71.4 (40) 28.6 (16) 0
Transcripts are available 33.9 (19) 66.1 (37) 0 0
Transcripts are downloadable 30.4 (17) 3.6 (2) 0 66.1 (37)
Transcripts contain visual descriptions essential for understanding video-only content 28.6 (16) 5.4 (3) 0 66.1 (37)
Lecture notes equivalent to the video content are available for download 39.3 (22) 51.8 (29) 8.9 (5) 0
Sign language interpretation for audio and video media is available 0 100 (56) 0 0
tion essential for understanding video-only content is available in most of the transcripts, including 12 MOOCs on Coursera (21.4%), three on Open Education (5.4%), and one on Lektorium (1.8%). No transcripts are offered by Universarium or Stepik.
Lecture notes, provided for nearly half of the MOOCs, can be an alternative or supplement to transcripts. In most of the cases, notes reproduce the video content completely (Table 1).
None of the MOOCs offer sign language interpretation for their lectures.
3.3. Accessibility of
digital documents Three quarters of the MOOCs contain digital documents such as notes, presentations, glossaries, reading recommendations, etc. (Ta-
Figure 1. Conformance of MOOCs to caption accessibility requirements
Captions are not generated 100% automatically
Subtittle files are available for download
Coursera Lectorium Open Education
All captions are closely sychronized with the audio
Captions are appropriate and reflect the audio faithfully
All captions remain on the screen for at least two second
There are no more than 45 characters in each line
There are no more two lines per caption
ble 2). The most common formats are.pdf, followed by.html and, in some cases,.doc(x) and.xls(x). Analysis shows that content is adequately displayed by screen readers only in the Game Theory course (Tomsk State University x Coursera). Meanwhile, most MOOCs contain well-organized documents with clickable structural elements including multilevel headings and tables.
Availability and correct rendering of image descriptions is an important accessibility requirement that is only partially satisfied for the Mathematical Game Theory course (St. Petersburg State University x Coursera), whereas 97% of the MOOCs containing graphics do not conform to this requirement.
Lossless image scaling is supported by nearly half of the documents containing images.
In designing the content of digital documents, MOOC developers do not regard color as a key source of information, which is confirmed for 87.9% of the MOOCs containing graphics.
In most cases, hyperlinks contained in documents are clickable and go to the right place, yet one in every five links has improper formatting.
3.4. Accessibility Testing is used to assess students' knowledge in all the MOOCs ex-of tests cept two on Coursera and one on Universarium. Of all the MOOCs using tests, 90.6% contain no tasks requiring good hand-eye coor-
Table 2. Accessibility evaluation of digital documents in mathematics MOOCs (N = 56)
Evaluation Criterion Satisfied, % (n) Not satisfied, % (n) Partially Satisfied, % (n) N/A, % (n)
The MOOC contains digital documents (notes, presentations, etc.) 76.8 (43) 23.2 (13) 0 0
Text of the document is adequately (accurately and consistently) read by screen readers 1.8 (1) 39.3 (22) 35.7 (20) 23.2 (13)
The document is well-structured, has a hyperlinked table of contents, headings, and bookmarks 51.8 (29) 10.7 (6) 14.3 (8) 23.2 (13)
Tables have clickable column and row headers 23.2 (13) 3.6 (2) 0 73.2 (41)
Images that are essential for the plot have descriptions readable by screen readers 0 57.1 (32) 1.8 (1) 41.1 (23)
Users are able to enlarge images (to at least twice the standard size) without losing quality 28.6 (16) 30.4 (17) 0 41.1 (23)
Color is not used as the only visual means of distinguishing a visual element 51.8 (29) 5.4 (3) 1.8 (1) 41.1 (23)
Hyperlinks are represented by words and redirect users straight to the right place 48.2 (27) 14.3 (8) 5.4 (3) 32.1 (18)
Sufficient background and text contrast ratio is provided 76.8 (43) 0 0 23.2 (13)
dination (Table 3), which people with visual and motor impairments normally find difficult to complete.
Assessing the incidence of tests by their type was beyond the scope of this study. Nevertheless, it appears reasonable to emphasize that multiple choice questions and open-ended questions are the most widespread.
Some MOOCs (four courses on Stepik) offer alternatives to tasks requiring good hand-eye coordination (e. g. instead of using a mouse to match objects, users are asked to select the options from a dropdown list for every object).
In 40% of the MOOCs using tests, the content of assignments is conveyed accurately and consistently by screen readers. Screen reading errors were detected in 20% of the courses. Meanwhile, screen readers are usually not enough to ensure adequate perception of the tests. Tasks that can be understood and done entirely without looking at the screen are rare exceptions. This paradox is explained, in particular, by speakers omitting images contained in the tests due to the lack of image description.
All the tests support keyboard-only feedback, and most of them can be completed in an unlimited time.
Table 3. Accessibility evaluation of tests in mathematics MOOCs (N = 56)
Evaluation Criterion Satisfied, % (n) Not satisfied, % (n) Partially Satisfied, % (n) N/A, % (n)
The tests contain no tasks requiring good hand-eye coordination 85.7 (48) 8.9 (5) 0 5.4 (3)
The tests offer an alternative to tasks requiring good hand-eye coordination 7.1 (4) 1.8 (1) 0 91.1 (51)
The tests are read accurately and consistently by screen readers 37.5 (21) 37.5 (21) 19.6 (11) 5.4 (3)
A screen reader is sufficient for adequate perception of the tests 8.9 (5) 76.8 (43) 8.9 (5) 5.4 (3)
Feedback can be provided using keyboard commands only (without a visual editor) 94.6 (53) 0 0 5.4 (3)
Unlimited time is allotted for taking the tests 87.5 (49) 7.1 (4) 0 5.4 (3)
3.5. Accessibility of Mathematical notations in the MOOCs are mostly displayed using im-mathematical notation ages, video, LaTeX and MathML formats, and other means (Table 4), including linear format, visualization using the MathJax library, Microsoft Equation Editor, and, in isolated cases, scanned copies of lecture notes and HTML math symbols. Figure 2 shows the distribution of the principal ways of displaying mathematical notation among the MOOC platforms.
MathML notation system is considered to be the most accessible format [W3C2014] and is available in the majority of MOOCs.
Image description is only available in 11.8% of the MOOCs that use images to render mathematical content (drawings, graphs, formulae).
Detailed visual description for videos containing mathematical notation is available in Linear Algebra and Analytic Geometry (Peter the Great St. Petersburg Polytechnic University x Lektorium) and Advanced High School Mathematics (Moscow Pedagogical State University x Universarium).
The majority of MOOCs allow searching for and rescaling mathematical content.
All the MOOCs, to varying degrees, scale the size of their mathematical notations to fit various screen resolutions.
Proper formula transcripts are available for 25% of the video lectures containing mathematical notation. However, it is only in Probability Theory (Tomsk State University x Open Education) that the transcripts are sufficient for adequate perception of the video content.
Most MOOCs support reading of mathematical notation by screen readers, but only one in five MOOCs has its math content read cor-
Table 4. Accessibility evaluation of mathematical notation in mathematics MOOCs (N = 56)
Evaluation Criterion Satisfied, % (n) Not satisfied, % (n) Partially Satisfied, % (n) N/A, % (n)
Mathematical notation is rendered by means of images 60.7 (34) 39.3 (22) 0 0
Mathematical notation is rendered by means of video 98.2 (55) 1.8 (1) 0 0
Mathematical notation is rendered by means of LaTeX 75.0 (42) 25.0 (14) 0 0
Mathematical notation is rendered by means of MathML 57.1 (32) 42.9 (24) 0 0
Mathematical notation is rendered by other means 96.4 (54) 3.6 (2) 0 0
Where mathematical notation is rendered by means of images, image description is available 7.1 (4) 51.8 (29) 1.8 (1) 39.3 (22)
Where mathematical notation is rendered by means of video, visual description is available 3.6 (2) 83.9 (47) 10.7 (6) 1.8 (1)
Search for mathematical notation is available and accessible 55.4 (31) 44.6 (25) 0 0
Mathematical notation can be rescaled 55.4 (31) 44.6 (25) 0 0
Mathematical notation is adaptable and displayed adequately at different screen resolutions 60.7 (34) 0 39.3 (22) 0
Video containing mathematical notation feature proper transcription of math formulae 25.0 (14) 67.9 (38) 5.4 (3) 1.8 (1)
Mathematical notation can be read by screen readers 83.9 (47) 10.7 (6) 5.4 (3) 0
Mathematical notation is read properly by screen readers 17.9 (10) 55.4 (31) 16.1 (9) 10.7 (6)
Captions and/or transcript are sufficient for adequate perception of the video containing mathematical notation 1.8 (1) 10.7 (6) 23.2 (13) 64.3 (36)
Additional material is available for interpreting the video containing mathematical notation 44.6 (25) 53.6 (30) 0 1.8 (1)
rectly. The highest quality of reading was observed for the linear math notations in Coursera transcripts.
Supplementary material to facilitate the interpretation of video lectures containing mathematical notation is provided by 45.5% of the video-based MOOCs and normally includes lecture notes, glossaries, chapters from textbooks, and bibliography.
4. Discussion A review of literature shows that this study is so far the third publication assessing the accessibility of mathematics MOOCs after the one produced by Alexa Ramirez-Vega et al. [Ramirez-Vega, Iniesto, Cova-donga 2017] and our previous article [Kosova, Khalilova 2019]. It is the first time that expert evaluation of mathematics MOOCs for students in mathematics and IT fields is performed in Russia.
Figure 2. Distribution of ways of rendering mathematical notation across the MOOC platforms (%)
I llll
Lectorium Universarium Coursera Open Education Stepik
Images 66.7 100.0 66.7 41.7 59.1
Video 100.0 100.0 100.0 100.0 95.5
LaTeX 66.7 0.0 66.7 50.0 95.5
MathML 66.7 0.0 53.3 75.0 95.5
MathJax 66.7 0.0 53.3 75.0 95.5
I | Linear 0.0 25.0 100.0 8.3 4.5
H MsEquation 0.0 100.0 13.3 8.3 0.0
None of the MOOCs examined allows customizing user interface or displaying content in text-based web browsers. Low accessibility of multimedia content is associated with poor design or lack of captions, transcripts, and lecture notes. Moreover, none of the MOOCs offers sign language interpretation for their audio content. Similar findings were obtained in a study of mathematics MOOCs in English [Ramirez-Vega, Iniesto, Covadonga 2017].
General accessibility of media-based lessons for visually impaired learners is low, which may be related to specific video lecture design principles and the complexity of mathematical content that is rarely voiced completely. Ramirez-Vega et al. [Ibid.] also observed low accessibility of multimedia content and recommended considering the use of templates for such content to allow the minimum accessibility set out to be guaranteed.
Half of the digital documents contained in the MOOCs are not reproduced faithfully by screen readers, first of all due to the lack of description for images which often contain mathematical concepts essential for understanding (formulae, reasoning, constructions, etc.). In the vast majority of the cases, screen readers cannot guarantee
faithful and correct reading of tests because of incorrect mathematical notation and lack of image description. This is in line with the findings of a number of international studies which revealed flaws in the reproduction of various types of MOOC content by screen readers [Akgül 2018; Al-Mouh, Al-Khalifa, Al-Khalifa 2014; Bohnsack, Puhl 2014; Ramírez-Vega, Iniesto, Covadonga 2017].
Mathematical content represented in all the MOOCs in various formats (images, video, markup languages) is the main challenge in providing web accessibility, especially for people with visual impairments. Similar results were obtained by Ramírez-Vega et al. [Ramírez-Vega, Iniesto, Covadonga 2017] who revealed accessibility flaws in all types of mathematical content, despite proper visual interpretation of scientific notation.
One must distinguish between accessibility problems associated with platform limitations and those caused by design failures. For example, Coursera does not support MathML, suggesting linear format and LaTex for mathematical notation instead, which makes content less accessible to screen readers. Open Education, Lektorium and Stepik (the former two use the Open edX open source platform) support MathML and MathJax natively, yet mathematical notation is not always read faithfully as a result of MOOC designers using alternative rendering formats (images without image description, video without visual description, inaccessible PDF files, etc.) as well as platform limitations. In particular, no page on Stepik identifies the default text-processing language, which is violation of WCAG and can lead to incorrect reproduction of the text [Bohnsack, Puhl 2014], including mathematical notation [Ramírez-Vega, Iniesto, Covadonga 2017]. To introduce formulae on platforms with MathJax support, developers use LaTeX, which is less accessible than MathML and may result in defective rendering of content by screen readers. Overall, screen readers reproduce only 20% of all mathematical content accurately and correctly, which is a huge barrier to learning for persons with severe visual impairments. Mathematical notation on Universarium proved to be the least accessible.
All the accessibility problems discovered can be grouped into three categories: i) flaws associated with violation of WCAG 2.1, which are relatively easy to fix by editing the HTML codes of the web pages; ii) flaws associated with the logical structure and content of MOOCs — fixing them will require expertise in creating accessible learning content for persons with various types of disability; and iii) platform design failures that cannot be fixed without changing the MOOC platform source code.
Summarizing the above, it is fairly safe to say that, despite the existing web accessibility guidelines and legal frameworks stipulated in international and federal regulatory documents (Universal Declaration of Human Rights, Convention on the Rights of Persons with Disabilities, Federal Law On Social Protection of People with Disabilities
in the Russian Federation, Federal Law On Education in the Russian Federation, and others) as well as national and international standards (GOST R ISO 9241-151-201419, GOST R52872-201220; WCAG 2.1), creating accessible courses in conformity to the applicable international rules was obviously beyond the mission of MOOC developers from the very beginning. Indeed, no automated accessibility tests were carried out, no users with disabilities participated in beta testing, the web accessibility algorithms built in the MOOC platforms were ignored or used inadequately, and in many cases the platforms were completely inaccessible to learners with disabilities.
The results obtained here are consistent with international findings [Bohnsack, Puhl 2014; Ferati, Mripa, Bunjaku 2016; Iniesto, Rodrigo 2014; Martin, Amado-Salvatierra, Hilera 2016; McAndrew, Gruszczyn-ska 2013; Ramírez-Vega, Iniesto, Covadonga 2017] and illustrate the need to bring the existing MOOCs into compliance with WCAG 2.1, ensure that MOOC platform administrators conform strictly to WCAG 2.1 before creating new courses, involve persons with disabilities in testing new and existing MOOCs, and teach MOOC authors and developers to create accessible course content. Training in accessible mathematics MOOC development should be designed in accordance to web content accessibility rules and guidelines (W3C2014; WCAG 2.12018).
5. Conclusion The findings of this study reveal low accessibility of mathematics MOOCs in Russian to people with disabilities, particularly those with severe visual impairments. Accessibility issues result from MOOC platform limitations and course design failures. A great majority of accessibility violations are associated with improper formalization of mathematical content, and complex mathematical notation structures make web accessibility of MOOCs even a greater challenge. The problem of elaborating unified guidelines on web accessibility of mathematical content remains acute and requires further investigation.
Development of a national accessibility standard for online learning resources (including MOOCs) on the basis of the international web content accessibility guidelines WCAG 2.1, web accessibility standardization best practices, and the rules of online learning content development and design will allow MOOC providers and developers to rely on a single web accessibility regulatory framework and unified accessibility evaluation methodology in creating new MOOCs and MOOC platforms as well as fixing those that already exist.
19 GOST R ISO 9241-151-2014 Ergonomics of Human-System Interaction. Part 151. Guidance on World Wide Web User Interfaces (2015) http://docs.cntd. ru/document/1200113012 2° GOST R52872-2012 Internet Resources. Accessibility Requirements for Visually Impaired People (2014) http://docs.cntd.ru/document/1200103663
Once binding web accessibility requirements are included in federal laws and bylaws related to online learning, a framework will be provided for legitimating the measures to ensure conformance of MOOC platforms and web developers to the web content accessibility guidelines.
References Kosova Ye., Khalilova M. (2019) Analiz web-dostupnosti massovykh otkrytykh onlain-kursov po matematicheskim distsiplinam [Web Accessibility Analysis of Massive Open Online Courses on Mathematical Disciplines]. Vysshee obrazovanie v Rossii/ Higher Education in Russia, vol. 28, no 10, pp. 157-166. Semenova T., Vilkova K., Shcheglova I. (2018) Rynok MOOK: perspektivy dlya Rossii [The MOOC Market: Prospects for Russia]. Voprosy obrazovaniya / Educational Studies Moscow, no 2, pp. 173-197. DOI: 10.17323/1814-95452018-2-173-197
Shutova A. (2018) Otkrytoe obrazovanie dlya lyudey s ogranichennymi vozmozh-nostyami zdorovya: zadachi dizajna [Open Education for People with Disabled Health Opportunities: Design Objectives]. Akademicheskij Vestnik Ural-NIIproekt RAASN, no 1, pp. 85-91. Acosta T., Acosta-Vargas P., Salvador-Ullauri L., Lujan-Mora S. (2018) Method for Accessibility Assessment of Online Content Editors. Proceedings of the International Conference on Information Technology & Systems (ICITS2018) (Libertad City, Ecuador, January 10-12, 2018), (eds A. Rocha, T. Guarda), pp. 538-551.
AHRC (2014) World Wide Web Access: Disability Discrimination Act Advisory Notes. Available at: https://www.humanrights.gov.au/our-work/disa-bility-rights/world-wide-web-access-disability-discrimination-act-adviso-ry-notes-ver (accessed 10 January 2019). Akgul Y. (2018) Accessibility Evaluation of MOOCs Websites of Turkey. Journal
of Life Economics, no 5, pp. 23-36. Al-Mouh N., Al-Khalifa A., Al-Khalifa H. (2014) A First Look into MOOCs Accessibility: The Case of Coursera. Computers Helping People with Special Needs. ICCHP 2014. Lecture Notes in Computer Science (eds K. Miesenberger et al.), vol. 8547, pp. 145-152. Bohnsack M., Puhl S. (2014) Accessibility of MOOCs. Computers Helping People with Special Needs. ICCHP 2014. Lecture Notes in Computer Science (eds K. Miesenberger et al.), vol. 8547, pp. 141-144. BSI (2010) Web Accessibility. Code of Practice. Available at: https://shop.bsi-group.com/ProductDetail/?pid=000000000030180388&rdt=wmt (accessed 10 January 2019).
Burgstahler Sh. (2015) Real Connections: Making Distance Learning Accessible to Everyone. Available at: http://www.washington.edu/doit/Brochures/ Technology/distance.learn.html (accessed 10 January 2019). Burgstahler Sh. (2019) 20 Tips for Teaching an Accessible Online Course. Available at: https://www.washingt on.edu/doit/20-tips-teaching-accessible-on-line-course (accessed 10 January 2019). Carter J., Markel M. (2001) Web Accessibility for People with Disabilities: An Introduction for Web Developers. IEEE Transactions on Professional Communication, vol. 44, no 4, pp. 225-233. De Waard I., Gallagher M. S., Zelezny-Green R., Czerniewicz L., Downes S., Kukulska-Hulme A., Willems J. (2014) Challenges for Conceptualising EU MOOC for Vulnerable Learner Groups. Proceedings of the European MOOC Stakeholder Summit 2014. Europe: Open Education Europa (Laussane, Switzerland) (eds U. Cress, C. D. Kloos), pp. 33-42.
Eur-Lex (2016) Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the Accessibility of the Websites and Mobile Applications of Public Sector Bodies. Available at: https://eur-lex.europa. eu/eli/dir/2016/2102/oj (accessed 10 January 2019).
Ferati M., Mripa N., Bunjaku R. (2016) Accessibility of MOOCs for Blind People in Developing Non-English Speaking Countries. Proceedings of the 8th International Conference Applied Human Factors and Ergonomics (AHFE) (Orlando, July 27-31, 2016), pp. 519-528.
Gay G., Djafarova N., Zefi L. (2017) Teaching Accessibility to the Masses. Paper presented at the 14th Web for All Conference on the Future of Accessible Work. Art. no 15. Available at: https://dl.acm.org/citation.cf-m?id=3058555.3058563 (accessed 10 January 2019).
GSA (1998) Section 508 of the US Rehabilitation Act of1973. Available at: https:// www.section508.gov/section-508-of-the-rehabilitation-act (accessed 10 January 2019).
Iniesto F., McAndrew P., Minocha S., Coughlan T. (2016) Accessibility of MOOCs: Understanding the Provider Perspective. Journal of Interactive Media in Education, vol. 1, art. no 20, pp. 1-10.
Iniesto F., McAndrew P., Minocha S., Coughlan T. (2017) Auditing the Accessibility of Massive Open Online Courses (MOOCs). Paper presented at the 14th Association for the Advancement of Assistive Technology in Europe Congress. Available at: http://www.aaate2017.eu/ (accessed 10 January 2019).
Iniesto F., Rodrigo C. (2014) Accessibility Assessment of MOOC Platforms in Spanish: UNED COMA, COLMENIA and MiriadaX. Proceedings of the IEEE International Symposium on Computers in Education (Logrono, Spain, November, 12-14, 2014), pp. 169-172.
Iniesto F., Rodrigo C. (2016) A Preliminary Study for Developing Accessible MOOC Services. Journal of Accessibility and Design for All, vol. 6, no 2, pp. 102-124.
Kent M. (2015) Disability and eLearning: Opportunities and Barriers. Disability Studies Quarterly, vol. 35, no 1. Available at: http://dsq-sds.org/article/ view/3815/3830 (accessed 10 January 2019).
Lowe T., Mestel B., Williams G. (2016) Perceptions of Online Tutorials for Distance Learning in Mathematics and Computing. Research in Learning Technology, vol. 24, art. no 30630. Available at: http://oro.open.ac.uk/46797/ (accessed 10 January 2019).
Luján Mora S. (2013) Web Accessibility among the Countries of the European Union: A Comparative Study. Actual Problems of Computer Science, no 1(3), pp. 18-27.
Martin J. L., Amado-Salvatierra H.R., Hilera J. R. (2016) MOOCs for All: Evaluating the Accessibility of Top MOOC Platforms. International Journal of Engineering Education, vol. 32, no 5-B, pp. 2374-2383.
McAndrew P., Gruszczynska A. (2013) Accessibility Challenges and Techniques for Open Educational Resources (ACTOER) Project Report. Available at: https://www.researchgate.net/publication/280656260 ACTOER project report (accessed 10 January 2019).
NC State University (1997) The Principles of Universal Design. Available at: https://projects.ncsu.edu/design/cud/pubs p/docs/poster.pdf (accessed 10 January 2019).
Policar L., Crawford T., Alligood V. (2017) Accessibility Benefits of e-Learning for Students with Disabilities. Available at: https://www.disabled-world.com/ disability/education/postsecondary/e-learning.php (accessed 10 January 2019).
Ramírez-Vega A., Iniesto F., Covadonga R. (2017) Raising Awareness of the Accessibility Challenges in Mathematics MOOCs. Paper presented at the
5th International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM), art. no 92. Available at: http://oro.open.ac.uk/id/ eprint/50954 (accessed 10 January 2019).
Sanchez-Gordon S., Lujan-Mora S. (2016) How Could MOOCs Become Accessible? The Case of EdX and the Future of Inclusive Online Learning. Journal of Universal Computer Science, vol. 22, no 1, pp. 55-81.
W3C (2012) Introduction to web accessibility. Available at: http://www.w3.org/ WAI/intro/accessibility.php (accessed 10 January 2019).
W3C (2014) Mathematical Markup Language (MathML). Version 3.0. Available at: https://www.w3.org/TR/MathML3/ (accessed 10 January 2019).
WebAIM (2019a) Articles. Available at: https://webaim.org/articles/ (accessed 10 January 2019).
WebAIM (2019b) Color Contrast Checker. Available at: https://webaim.org/re-sources/contrastchecker/ (accessed 10 January 2019).
Wei X., Yu J. W., Shattuck P., McCracken M., Blackorby J. (2013) Science, Technology, Engineering, and Mathematics (STEM) Participation among College Students with an Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, vol. 43, no 7, pp. 1539-1546. Available at: https://www. ncbi.nlm.nih.gov/pmc/articles/PMC3620841/ (accessed 10 January 2019).
Wentz B., Jaeger P. T., Lazar J. (2011) Retrofitting Accessibility: The Legal Inequality of After-the-Fact Online Access for Persons with Disabilities in the United States. First Monday, vol. 16, no 11. Available at: https://journals.uic. edu/ojs/index.php/fm/article/view/3666/3077 (accessed 10 January 2019).
