CRITERIA OF EVALUATING AUGMENTED REALITY APPLICATIONS Текст научной статьи по специальности «Компьютерные и информационные науки»

ИНФОРМАТИКА, ВЫЧИСЛИТЕЛЬНАЯ

ТЕХНИКА И УПРАВЛЕНИЕ INFORMATION TECHNOLOGY, COMPUTER

SCIENCE, AND MANAGEMENT

УДК 004.93 https://doi.org/10.23947/2687-1653-2020-20-4-414-421

Criteria of evaluating augmented reality applications

M. R. Ablyaev, A. N. Abliakimova, Z. S. Seidametova

Crimean Engineering-Pedagogical University (Republic of Crimea) Q ^

Introduction. The field of augmented reality (AR) is growing rapidly and has great advances in interaction, navigation and tracking. Nowadays there are a lot of trends for AR applications in different areas (education, entertainment, business, medicine, etc.). However, there is a lack of research to provide the evaluating AR apps framework to support developers when creating suitable AR applications for specific needs. We provide a practical approach to quantify some of the AR applications features. We focus on the development of criteria for evaluating augmented reality applications. We discuss the criteria of choosing dimensions for that space such as standards for AR, tools for AR development, navigation and tracking, content management, usability. We provide analysis and evaluation of AR apps through each characteristic using guidelines which we have developed.

Materials and Methods. An AR application is a software application that integrates digital visual, audio and other types of content into a real-world environment. The software quality and performance are the main characteristics of the application, which are key factors for AR applications. The analysis of scientific papers, documents and standards made it possible to determine characteristics that are the most significant quality indicators based on well-grounded users' needs and demands.

Results. The criteria we have developed for evaluating applications with augmented reality enable developers to create their own software products in stages, based on step-by-step requirements for them, evaluating the development process by characteristics. This approach will allow you to create high-quality software products using standardized, modern development tools.

Discussion and Conclusions. In addition, developers will have a detailed understanding of each stage of creating the application and the necessary development tools and technologies to obtain the highest quality result. That will give an opportunity to decide on specific development tools, methods, models and technologies before starting work on a project. As a result, it will provide the final high-quality software product with good extensibility and compliance with the modern requirements of the digital industry market.

Keywords: augmented reality, software development kit, navigation and tracking, content management, usability.

For citation: M. R. Ablyaev, A. N. Abliakimova, Z. S. Seidametova. Criteria of evaluating augmented reality applications. Advanced Engineering Research, 2020, vol. 20, no. 4, pp. 414-421. https://doi.org/10.23947/2687-1653-2020-20-4-414-421

© Ablyaev M. R., Abliakimova A. N., Seidametova Z. S., 2020

Критерии оценивания приложений дополненной реальности

д М. Р. Абляев, А. Н. Аблякимова, З. С. Сейдаметова

ГБОУВО РК «Крымский инженерно-педагогический университет имени Февзи Якубова», (г. Симферополь, Республика Крым)

О

ТЗ

^

•д Введение. Дополненная реальность (AR) является быстроразвивающейся отраслью, которая значительно про-« двинулась в вопросах взаимодействия, навигации и отслеживания объектов. В настоящее время существует множество направлений AR, которые применяются в различных областях (образование, развлечения, бизнес, л медицина и т.д.). Однако отсутствуют чёткие критерии оценки фреймворков AR-приложений для поддержки разработки таких приложений под конкретные задачи. В статье предложен подход, позволяющий количественно оценить некоторые характеристики AR-приложений. В данной статье сделана попытка разработки критериев оценки приложений дополненной реальности. Предложены критерии выбора измерений пространства разра-414 ботки, такие как стандарты для AR, инструменты для разработки AR, навигация и отслеживание объектов,

управление контентом, удобство использования (юзабилити). Предоставлены анализ и оценки AR-приложений по каждой предложенной характеристике.

Материалы и методы. AR-приложение - это программное приложение, которое интегрирует цифровой визуальный, аудио-контент и другие типы контента в реальную среду. Качество и производительность программного обеспечения являются основными характеристиками приложения, которые являются критически важными факторами для приложений AR.

Анализ научных работ, документов и стандартов позволил определить характеристики, которые являются наиболее значимыми качественными показателями, основанными на обоснованных потребностях и требованиях пользователей.

Результаты исследования. Разработанные нами критерии оценки приложений с дополненной реальностью позволяют разработчикам создавать свои собственные программные продукты поэтапно, основываясь на пошаговых требованиях к ним, оценивая процесс разработки по характеристикам. Такой подход позволит создавать высококачественные программные продукты, используя стандартизированные, современные средства разработки.

Обсуждение и заключения. Разработчики будут иметь детальное понимание на каждом этапе создания приложения, что позволит выбирать необходимые инструменты и технологии разработки для получения результата наивысшего качества. Это также даст возможность определиться с конкретными инструментами, методами, моделями и технологиями разработки до начала работы над проектом. В результате будет получен конечный высококачественный программный продукт с хорошей расширяемостью и соответствием современным требованиям рынка цифровой индустрии.

Ключевые слова: дополненная реальность, комплект разработки программного обеспечения, навигация и отслеживание, управление контентом, юзабилити.

Для цитирования: Абляев, М. Р. Критерии оценивания приложений дополненной реальности / М. Р. Абляев, А. Н. Аблякимова, З. С. Сейдаметова // Advanced Engineering Research. — 2020. — Т. 20, № 4. — С. 414-421. https://doi.org/10.23947/2687-1653-2020-20-4-414-421

Introduction. The development of the modern market of augmented reality (AR) technology contributes to the emergence of a larger number of AR-applications widely used in industries including healthcare, public safety, gas and oil, tourism and marketing, entertainment and academia. In connection with the increased interest in this technology, the development of functional capabilities of AR projects is also carried out, which stimulates an increase in the needs of AR-technology using in the most diverse sectors of modern society.

Unfortunately, some significant aspects of the AR applications development and implementation and AR services, often ignored for the design simplicity and implementing speed, are the compliance with such systems to real ^ conditions and the evaluating under real operating conditions. To develop successful and highly efficient AR systems ^ that can be adopted in everyday scenarios, user assessment and feedback are very important [1].

Augmented reality is a very young industry, and there are still no generally accepted standards for developing G AR applications. Although the main platforms for creating augmented reality applications have been defined: ARKit , к ARCore2 and Unity3 (mostly used) - even now AR applications developed using these tools are available for launch only on a limited type of digital and mobile devices. E

The situation is exactly the same with wearable devices, HoloLens and Magic Leap glasses - each manufactur- g er offers its own unique software for creating augmented reality programs. Experts give several years before the indus- к;

ей

try consolidates and common standards allow making the development of augmented reality applications accessible and £

universal for all AR developers. §

Rapid progression of the AR field requires effective and validated methods of design evaluation to be devel- к

oped. Failure to consider the usability of AR applications during the design process will result in an increase in user о

s

errors and accidents, limiting user trust of the technology and undermining user perceptions of the technology, for both E AR and Virtual Reality (VR) technologies [2]. «

The authors of the papers [1], [2], [3] provided an overview of the important designing and implementing features of AR applications and proposed theoretical evaluation of AR systems and frameworks through the standardiza- н

ей

tion aspects. Endsley and others [4] described principles of design heuristics for AR for multi-dimensional augmented environments. Some examples of user experience evaluations were presented in the papers [5], [6], [7], [8], [9], [10]. ^

J. L. Gabbard and J. E. Swan [11] proposed a Usability Engineering (UE) for Augmented Reality approach that к

s

1 Augmented Reality - Apple Developer. https://developer.apple.com/augmented-reality/

2 ARCore - Google Developers. https://developers.google.com/ar 415

3 Unity for all. https://unity.com/

inserts iteratively a series of user-based studies into a traditional usability-engineering life cycle. Several usability testing methods of the AR application (subjective measurement using human perception, objective measure from observation, evaluation by expert through cognitive walkthrough, heuristic evaluation, lab observation, questionnaire) were described by Pranoto and others [12]. Martins and others [13] presented practical use of the usability methods for evaluating an AR children's book with multiple methods. Other aspects of AR technology tools and AR applications evaluating were presented in the papers [14] and [15].

Akgul and others [16] adapted an existing deep learning architecture to solve the detection problem in AR application using camera-based tracking. Other methods that help to improve AR application and to increment the productivity in manufacture were described in [17], [18], [19].

There are several survey papers on AR development, but none is dedicated to Mobile Augmented Reality. Huang and others [20] present the results of the latest technologies and methods survey that improves run-time performance and energy efficiency for the practical implementation of mobile AR applications.

We have presented AR application "Tilsimli arifler" ("Magic letters") and special features of design and d e-veloping mobile AR application for enhancing early literacy skills in our papers [21], [22].

Software quality and performance. An AR application is a software application that integrates digital visual, audio and other types of content into a real-world environment. The software quality and performance are the main characteristics of the application, which are key factors for AR applications. Software development is a complex and multi-faceted process, in which a large number of specialists of various areas of expertise and various skill levels participate. In addition, many technical, technological, and managerial issues intertwine in the application development process. The success of the project and the quality of the developed product depend on their adequate involvement.

The analysis of papers, documents (e.g., [2], [3], [10], [16], [22]) and standards (e.g., ISO-IEC JTC 1 SC 24, 18521-11, ISO 90002) shows that the following characteristics are the key quality indicators based on well-grounded users' needs and demands:

— inadequate functioning of the software product;

— insufficient interaction of the product with other software, hardware, telecommunications;

— failures of the software application during the intended use;

— the slowed down time of the software product and the delay in the presentation of intermediate and output information;

— incomplete display of information;

— inconsistency of stored data and information entered by the operator;

— loss of relevance of the information;

— violation of the confidentiality of information.

In addition to such "primary" quality data coming directly from the consumer, developers use "internal" indicators to evaluate the parameters of the current project:

— the lines of code in the standard mode;

— the number of detected errors per 1000 lines of code;

— program complexity parameters;

— the probability of occurrence of specific errors;

— the project complexity and cost of a code unit;

— price of a "man-month";

— statistical characteristics of processes (expectation, variance, correlation function, etc.) and other estimated parameters.

g Augmented Reality Application Evaluation Criteria. Ten experts with more than 3 years' experience in AR

3 application development help us to create criteria for evaluating AR apps. We divided the main criteria for assessing the g quality of augmented reality applications into 5 groups (Fig 1):

— AR app design and Art design;

M

(L) >

Graphic programming; AR applications programming; Application profiling and optimization; Publishing applications (build).

When developing an AR application design, we recommend the following characteristics that should be con-

1 Kim G. Augmented Reality Continuum Concepts and Reference Model - Part 1: ARC Reference Model (Work Item Proposal), ISO-IEC JTC 1 SC 24, 18521-1. 2012.

2 ISO 9000 Family Quality Management. https://www.iso.org/iso-9001-quality-management.html

sidered:

1. The surface. How the application can adapt to various surfaces. If you use frameworks, you can use the built-in surface detection function. For example, the application can recognize the floor, walls and objects.

2. Shine. Evaluation of illumination is very important for the realism of objects. It is advisable to use dynamic lighting with shadows in real time.

3. Space needed. Users can experience AR in three different sizes: table scale, room scale, and open environment. The user should always have enough space to simply enjoy the experience. Thus, it is necessary to think over this before the user starts using the application. For example, if an application requires an open environment, you need to inform the audience in advance before they start using the product.

4. Single-user or multi-user product. If you are developing a multi-user interface, you should design the product when everyone is involved in the process. It is important to create a sense of audience connection with the product. For example, if you are developing an AR game, you can provide a map that shows the location of users and provides real-time status notifications.

Fig. 1. Evaluation criteria for AR applications The main criteria are highlighted as follows (Table 1).

AR App Design and Art Design evaluation points

Table 1

AR App Design and Art Design 25

Actual requirements (of the market) in terms of application design 2

Interface design requirements 2

UX features for AR applications 1

Target platform specifications 2

3D modeling principles 1

Principles of work with particle systems 1

Types and features of the creation and use of textures 1

Principles of working with shaders and materials, rendering features 2

Features for customizing models and textures, as well as materials for export to the game engine 1

Create high-quality 3D models in modern editors 2

Create UV-scan in modern editors 2

Texturize models in modern editors 2

Rigging and animating models in modern editors 2

Customize animation in game engines 2

Create and customize shaders, materials in simulation programs and game engines 2

The initial version of Table 1 was proposed by the authors, then it was discussed with ten experts in AR devel-

(U К X <u 4 и

eö Л

С

^

IS eö И IS

X

*

(U

н «

eö X Л

ч

(U

н

IS

ч о

IS

Е 3 и

eö И

IS

eö

s

л о

X

К

opment using brainstorming techniques.

To solve the problems of displaying complex graphic objects, it is necessary to develop effective methods and algorithms for processing graphic information at the stages of input, encoding, transformation and image formation. All of this makes up a set of basic computer graphics tasks, so we have identified the main components needed to create a high-quality graphic component (Table 2).

Graphic programming enables to create visual effects through the development of shaders using the functions of graphic libraries, customizing the rendering of the development environment, using effective postprocessing libraries, customizing shadows, and more. All these components make it possible to obtain high-quality images by applying various effects, as well as improve the optimization of application performance.

Table 2

Graphic programming evaluation points

Graphic programming 25

Rendering features on the target platform 2

Using textures and materials in the game engine 2

Features of the work of graphic libraries 2

Principles of rendering geometric objects and images 2

Principles of proper postprocessing 3

Features of implementing lighting and shadows on the used game engine and target platform 3

Configure static and dynamic lighting in the used game engine 2

Optimize rendering processes 3

Customize postprocessing and final image appearance 2

Create procedural geometry using game engine tools 2

Programming the rendering of the frame 2

Evaluation points presented in Table 2 were proposed by the authors and later they were discussed with ten experts in AR development.

During the development of the software part of the mobile application with augmented reality, it is necessary to precisely determine the fundamental development tools, and, accordingly, the programming languages, programming environments, patterns, architecture, etc. This is necessary for a clearer and more coordinated team work, understanding the requirements and tasks, comprehension of the logic and sequence of application development, orientation in the project, reducing risks, and interaction of each team member with each other. To achieve this, we have identified the following points for the programming phase, which should be followed in order to obtain a positive result of the development process (Table 3).

Table 3

AR Applications Programming evaluation points

AR Applications Programming 25

Modern programming languages used in AR development 2

OOP principles 2

Building an application architecture 2

Principles of building AR applications 2

Code design standards 2

Basic math for AR applications 2

Work with a network in the context of AR 2

Work with modern AR application development environments 2

Write high-quality code in modern programming languages used in AR 2

Implement specific application mechanics as soon as possible 2

Use development environments for writing and debugging code 2

Work with necessary SDKs for AR 2

Use collaboration tools 1

The evaluation points presented in Table 3 were proposed by the authors and brainstormed with ten experts in AR development.

When developing applications with augmented reality, it is important to consider the features of application

optimization for PC and mobile devices, as well as the architecture of mobile devices in the context of application optimization. In this regard, we have identified the points that we advise to adhere to during the development in order to achieve maximum optimization of AR applications (Table 4).

The profiler tool provides specific data on game performance and facilitates its optimization process. The profiler provides frame-by-frame metrics with which you can more easily identify problem areas.

Table 4

Application Profiling and Optimization evaluation points

Application Profiling and Optimization 16

Optimization of AR application performance 3

Optimization of 3D objects 2

Geometry optimization 2

Optimization of textures and materials for the target platforms 2

Optimization of the main application processes 2

Optimization of physics in the application 2

Using built-in game engine profilers, as well as external profilers 3

Table 4 was created by the authors and then it was discussed with ten experts in AR development.

After completion of all stages of the application development, it is important to publish the application. The publication represents the release of the application on any platform where the customer (end-user) can easily download the final version of product, get acquainted with it, get all the necessary documentations, technical support and feedback from the developer. Each of the platforms puts forward its specific requirements to the publishing application, which are necessary for correct displaying the application in the platform's market, obtaining all information about the application's operation, ensuring end-user security, promoting the application, and more. In this regard, the publication is one of the most important and crucial stage of the development. We have identified several main platforms, before using which it is necessary to familiarize yourself with all the documentation and assembly features for the appropriate platform (Table 5).

Table 5

Publishing applications (build) evaluation points

Publishing applications (build) 9

Features of building an application for Windows 3

Features of the build application for Android. 3

Features of the build application for OS X/ iOS. 3

Evaluation points presented in Table 5 were proposed by the experts in AR development.

Conclusions. Nowadays, the augmented reality is one of the most innovative and a new digital trend in the developing applications for all type of devices. The AR technology opens a new horizon and is going to get more popular in the foreseeable future.

The criteria we have developed for evaluating applications with augmented reality enable developers to create their own software products in stages, based on step-by-step requirements for them. This will allow you to create high-quality software products using the standardized, modern development tools.

In addition, developers will have a detailed understanding of each stage of creating the application and the necessary development tools and technologies to obtain the highest quality result. That will give an opportunity to decide on specific development tools, methods, models and technologies before starting work on a project; and, in the process of working on the basis of existing criteria, gradually create key application stages, with possible subsequent upgrades and improvements. As a result, it will provide the final high-quality software product with good extensibility and compliance with the modern requirements of the digital industry market.

In the follow-up study, we are going to apply this approach for evaluating several AR applications. To thoroughly verify the proposed criteria, additional testing will be required, where more software field experts should be involved. Engaging third-party experts will assess the suitability of the proposed criteria. We believe that this approach will become part of the AR application development process.

References

1. Ritsos PD, Ritsos DP, Gougoulis AS. Standards in augmented reality: a user experience perspective. In: 2nd International Workshop on AR Standards. 2011; 17:9.

2. Lee J, Lee Y, Lee S, et al. Standardization for augmented reality: introduction of activities at ISO-IEC SC

<u S X <u 4 и

Л

G

^

IS ей И IS

X

*

(U H

Ч ей X л ч

(U H IS

4 о

IS

Е 3 и

ей И

IS

fe

5 л о

X

S

24 WG 9. In: Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, 2013. P. 279-280. DOI: 1145/2534329.2534379

3. Perey C, Engelke T, Reed C. Current Status of Standards for Augmented Reality. Recent Trends of Mobile Collaborative Augmented Reality Systems. Springer, New York, NY, 2011. P. 21-38. DOI: 10.1007/978-1-4419-9845-3_2

4. Endsley TC, Sprehn KA, Brill RM, et al. Augmented Reality Design Heuristics: Designing for Dynamic Interactions. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2017;61(1):2100-2104.

5. Dhir A, Al-kahtani M. A Case Study on User Experience (UX) Evaluation of Mobile Augmented Reality Prototypes. Journal of Universal Computer Science. 2013;19(8):1175-1196.

6. Caudell TP, Mizell DW. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences. IEEE. 1992;2:659-669.

7. Olsson T, Lagerstam E, Kârkkâinen T, et al. Expected user experience of mobile augmented reality services: a user study in the context of shopping centres. Personal and ubiquitous computing. 2013;17(2):287-304.

8. Knoerlein B, Luca MD, Harders M. Influence of visual and haptic delays on stiffness perception in augmented reality. In: 8th IEEE International Symposium on Mixed and Augmented Reality, IEEE. 2009. P. 49-52.

9. Duenser A, Billinghurst M. Evaluating augmented reality systems. In: Handbook of augmented reality. Springer, New York, NY, 2012. P. 289-307.

10. Arifin Y, Sastria TG, Barlian E. User experience metric for augmented reality application: a review. Procedia Computer Science. 2018;135:648-656.

11. Gabbard JL, Swan JE. Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design. In: IEEE Transactions on Visualization and Computer Graphics 14, 2008. P. 513-525.

12. Pranoto H, Tho C, Warnars HL, et al. Usability testing method in augmented reality application. In: 2017 International Conference on Information Management and Technology (ICIMTech), 2017. P. 181-186.

13. Martins VF, Sanches GB, de Almeida NG, et al. Usability Evaluation of an Augmented Reality Children's Book. In: 2019 XIV Latin American Conference on Learning Technologies (LACLO), IEEE. 2019;1:381-386. DOI: 10.1109/LACLO49268.2019.00070

14. Da Silva MMO, Teixeira JMXN, Cavalcante PS, et al. Perspectives on How to Evaluate Augmented Reality Technology Tools for Education: A Systematic Review. Journal of the Brazilian Computer Socciety. 2019;25(1):1—18. DOI: 10.1186/s13173-019-0084-8

15. Guimaraes MDP, Martins VF. A Checklist to Evaluate Augmented Reality Applications. In: 2014 16th Symposium on Virtual and Augmented Reality, IEEE, 2014. P. 45-52. DOI: 10.1109/SVR.2014.17

16. Akgul O, Penekli HI, Genc Y. Applying deep learning in augmented reality tracking. In: 12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), IEEE. 2016;1:47-54.

17. Harders M, Bianchi G, Knoerlein B, et al. Calibration, Registration, and Synchronization for High Precision Augmented Reality Haptics. In: IEEE Transactions on Visualization and Computer Graphics. 2009;15(1):138-149.

18. Ramirez H, Mendoza E, Mendoza M, et al. Application of augmented reality in statistical process control, to increment the productivity in manufacture. Procedia Computer Science. 2015;75:213-220. http://www.sciencedirect.com/science/article/pii/S1877050915037011

19. Chen H, Dai Y, Meng H, et al. Understanding the Characteristics of Mobile Augmented Reality Applications. In: 2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Belfast, 2018. P. 128-138.

20. Huang Z, Hui P, Peylo C, et al. Mobile augmented reality survey: a bottom-up approach. 2013. P. 112-126. arXiv preprint arXiv:1309.4413

g 21. Ablyaev M, Abliakimova A, Seidametova Z. Design of mobile augmented reality system for early literacy.

3 Ermolayev V, Mallet F, Yakovyna V, et al. (eds.) In: Proceedings of the 15th International Conference, ICTERI 2019, a Vol. I: Main Conference, CEUR Workshop Proceedings (CEUR-WS.org) on ICT in Education, Research, and Industrial Applications. Ukraine, Kherson, 12-15 June 2019. 2019;2387:274-285. CEUR-WS.org http://ceur-ws.org/Vol-^ 2387/20190274.pdf (accessed: 22 March 2020).

22. Ablyaev M, Abliakimova A, Seidametova Z. Developing a mobile augmented reality application for en-

<D

.> hancing early literacy skills. Ermolayev V, Mallet F, Yakovyna V, et al. (eds.) In: Information and Communication ¡¿, Technologies in Education, Research, and Industrial Applications. ICTERI 2019. Communications in Computer and ja Information Science Springer, Cham. 2020;1175:163-185. DOI: 10.1007/978-3-030-39459-2

Submitted 29.07.2020 Scheduled in the issue 12.10.2020

About the Authors:

Ablyaev, Marlen R., lecturer of the Applied Informatics Department, Crimean Engineering-Pedagogical University (8, per. Uchebny, Simferopol, Republic of Crimea, 295015), ORCID: http://orcid.org/0000-0003-0571 -2333, [email protected]

Abliakimova, Afife N., lecturer of the Applied Informatics Department, Crimean Engineering-Pedagogical University (8, per. Uchebny, Simferopol, Republic of Crimea, 295015), ORCID: http://orcid.org/0000-0003-3867-5401, [email protected]

Seidametova, Zarema S., Head of the Applied Informatics Department, Crimean Engineering-Pedagogical University (8, per. Uchebny, Simferopol, Republic of Crimea, 295015), Dr.Sci. (Pedagogics), Cand.Sci. (Phys.-Math.), professor, Scopus ID: 57189935002, ResearcherID: AAF-9106-2019, ORCID: http://orcid.org/0000-0001-7643-6386, [email protected]

Claimed contributorship

M. R. Ablyaev: critical revision of the assessment criteria; writing the first version of the paper. A. N. Abliakimova: collection and processing of material; concept and design of the research; writing the first version of the paper. Z. S. Seidametova: collection and processing of material; concept and design of the research; writing the final version of the paper.

All authors have read and approved the final manuscript.

Все авторы прочитали и одобрили окончательный вариант рукописи.

(U

Сдана в редакцию 29.07.2020 Запланирована в номер 12.10.2020

Об авторах:

Абляев Марлен Рефатович, преподаватель кафедры «Прикладная информатика» ГБОУВО РК КИПУ имени Февзи Якубова» (295015, Республика Крым, г. Симферополь, пер. Учебный, 8), ORCID: http://orcid.org/0000-0003-0571-2333, [email protected]

Аблякимова Афифе Наримановна, преподаватель кафедры «Прикладная информатика» ГБОУВО РК КИПУ имени Февзи Якубова» (295015, Республика Крым, г. Симферополь, пер. Учебный, 8), ORCID: http://orcid.org/0000-0003-3867-5401, [email protected]

Сейдаметова Зарема Сейдалиевна, заведующая кафедрой «Прикладная информатика» ГБОУВО РК

КИПУ имени Февзи Якубова» (295015, Республика Крым, г. Симферополь, пер. Учебный, 8), доктор педагоги- к

ческих наук, кандидат физико-математических наук, профессор, Scopus ID: 57189935002, Researcher ID - AAF- g

9106-2019, ORCID: http://orcid.org/0000-0001-7643-6386, [email protected] g

o,

Заявленный вклад соавторов: ^

М. Р. Абляев - критический пересмотр критериев оценивания, написание первой версии статьи. ^

А. Н. Аблякимова - сбор и обработка материала, концепция и дизайн исследования, написание первой версии g

s

статьи. З. С. Сейдаметова - сбор и обработка материала, концепция и дизайн исследования, написание Я окончательной версии статьи ¡^

ч eö К Л

ч

<U

н к

4 о

к

Е 3 и

eö И К

<3

5 а о

к К