CC BY-NC-ND
95
DOI: 10.17323/2587-814X.2022.2.21.35
The development of a model for a personalized learning path using machine learning methods*
Ekaterina S. Morozevich a
E-mail: katyamorozevich@mail.ru
Vladimir S. Korotkikh b ©
E-mail: vskor@bk.ru
Yevgeniya A. Kuznetsova a
E-mail: zhenya.kuz-1997@yandex.ru
a Reshetnev Siberian State University of Science and Technology Address: 31, Krasnoyarsky Rabochy Av., Krasnoyarsk 660037, Russia
b Municipal budget general education institution secondary school No. 4 Address: 9, st. Naberezhnaya, Divnogorsk 663091, Russia
Abstract
Today, the economy is undergoing a digital transformation. Its key barriers are a lack of qualified personnel, competencies and knowledge, as well as internal resistance in organizations. It can be overcome through quality staff development and training. An urgent problem is to build a personalized learning path. Modern research is aimed at the implementation of recommendation systems in order to select relevant material. However, these recommendations are based on digital traces; the student's full personal profile, as well as organizational values are not considered. This study aims is to create an intelligent guide that would accompany an employee throughout his life in the organization, involving him in the learning process according to a personalized path based on a complex personal profile and reactions to educational material, training soft and hard skills in accordance with the values of the organization and the employee. Methods of system analysis, system engineering, psychodiagnostic research (the DISC model, Rowe's "Decision-making Style" methodology, Honey and Mumford's method of determining activity styles, psychotype test), software design and artificial intelligence (matrix factorization and neural networks) were used in
* The article is published with the support of the HSE University Partnership Programme
this study. The study was conducted on a unique database collected as part of its implementation and consisting of educational tasks for soft skills development, plus data on their implementation by users with different soft skills profiles. An intelligent guide model has been developed and implemented as a software component for an enterprise management system. The basis consists of psychodiagnostic modules, organizational management, training and recommendations. The intelligence of the system we developed allows you to qualitatively form a personalized learning path that will involve an employee not only in the learning and development process, but also in achieving organizational goals. The organization receives T-shaped specialists who have a proactive position and are capable of self-organization by investing in the development of employees. The results of this study can be used by enterprises not only at the organizational level, but also through broadcasting in the education system to form an education ecosystem in accordance with the requirements of innovative development of a given region's economy.
Keywords: training and development, supra-professional competencies, adaptability, recommendation system, machine learning, matrix factorization, neural network
Citation: Morozevich E.S., Korotkikh V.S., Kuznetsova Y.A. (2022) The development of a model for a personalized learning path using machine learning methods. Business Informatics, vol. 16, no. 2, pp. 21-35. DOI: 10.17323/2587-814X.2022.2.21.35
Introduction
Today, the economy is undergoing a digital transformation. This is a revolutionary process of transforming an organization's business model, not only by using digital technologies, but also by introducing fundamental organizational changes in technology, culture, operations and the principles of creating new products.
The connecting link between the integration of deep organizational transformations and new technologies are people — employees with their knowledge, skills and values. According to Deloitte Consulting LLC, 53% of organizations already understand that in the current conditions from 50 to 100% of their employees should acquire new skills and abilities [1].
Organizations are created by people with their value system's investment in the structure being created. An employee who comes to an organization at a certain stage of the life cycle
does not realize what is happening, because he or she does not know what happened, and does not guess what will happen. In view of this, the urgent task is the development of personnel, taking into account the points of contact between the values of the employee and the organization.
Activities' digitalization generates an increase in data volume that requires further processing. It is data that underlies digital transformation, being the lifeblood of the transition process to the digital economy.
Already in 2020, according to Data Age Report, about 51 zettabytes of information were generated by mankind, and by 2025 this data volume will increase almost 3.5 times and amount to 175 zettabytes [2]. This shows the trend of digital data exponential growth.
The users' activity recorded by various devices represents their digital traces. Today, digital footprints occupy a significant part of
the big data cloud. Their key direction is to extract information about the preferences of potential customers and offer products which they will be interested in, that is, increasing sales [3].
However, the information extracted from digital traces can be used not only in external interactions, but also in the internal activities of the company. Modern companies are increasingly using data to increase employee engagement in achieving and separating goals, that is, to increase the productivity of both employees and organizational processes, thereby creating new sources for their competitive advantage [4].
Thus, recommendation systems are an integral part of the major market players today. The recommendation system is a complex of algorithms, programs and services designed to form relevant recommendations to users regarding the object of information search [5].
Many studies note [6—8] that it is possible to overcome the key barriers of digital transformation in the form of a shortage of qualified personnel, competencies and knowledge, as well as internal resistance in organizations [9—10] by creating a system of training and staff development. This system is based on the formation of a personalized learning path [11] relying on the values of the organization and the employee and relevant educational material, in other words, on those objects, actions, tasks, which the employee will perform with increased interest and, as a result, with maximum efficiency.
In the education field, the use of information systems based on recommendations is designed to solve one of the student's main problems — choosing educational material. Often such material is training programs and educational courses. When selecting educational material in such systems, the student's personal characteristics are usually not taken into account, but his preferences are used [12]. Recently,
researchers have begun to actively study using personal characteristics in educational recommendations. For example, the use of information about the learning style and emotional reactions [13]. Moreover, the issue of the recommendation systems used in online learning is increasingly rising in the research [14].
Thus, this study's purpose is to create an intelligent guide which would accompany an employee throughout his life in the organization, involving him in the learning process according to a personalized path based on data on the personal profile and reactions to educational material, training soft and hard skills in accordance with the values of the organization and the employee.
1. An intelligent system of personnel training and development concept
The principles of system engineering provided the basis for the intelligent guide development for the employees' growth and training. The requirements' collection of interested parties to the system was being developed as the breakaway point in the creation of an intelligent guide.
As the result of analyzing the requirements, it was revealed that the modular architecture of the system will not only take into account all the stakeholders' requirements, but also differentiate access to various functions depending on the positions held. The modular architecture proposed for implementation in accordance with the identified requirements is shown in Fig. 1.
A database was created using the MySQL database management system to implement the intelligent guide and support the work of the proposed modules.
The diagnostic module's main task is the formation of a digital profile of an employee, which will make it possible to take into account
Fig. 1. Intelligent guide model.
all the characteristics of the individual and will become the basis for effective training and development.
Modern trends are such that organizations are increasingly focusing on supra-professional skills or so-called soft skills, which are much more difficult to reproduce with the help of digital transformation tools and which underlie the ongoing changes [15]. Soft skills are the basis for the effective development of professional skills or so-called hard skills.
Thus, it was proposed to form a digital profile of an employee, consisting not only of personal data (gender, age, education, position, work experience), but also revealing the levels of development of employee soft skills in demand in the labor market in the digital economy.
Comprehensive automated diagnostics were developed to determine the level of proficiency in one or another soft skill (Fig. 2). The DISC model, Rowe's "Decision-making Style" methodology, Honey and Mumford's method of determining activity styles, and a psychotype test were used as diagnostic material. These methods reveal the psychoemotional characteristics of a person in 16 projections with their respective competencies.
The organizational management module is designed to fix organizational values through formation of job profiles. First of all, an organ-
izational structure is created through this module. After that, the head of the structural unit determines the points of the critical and desired profile of the employee for each position under consideration, in accordance with the labor functionality and labor actions, as well as the supra-professional skills designated on the basis of organizational values.
Thus, having compiled a position profile for each element of the organizational structure, a comprehensive profile of the organization is formed which reflects all the features of its activities.
The educational process organization is the basis of the training module. Users in the framework of soft skills development perform unique educational tasks (Fig. 3). For each task, the user evaluates how much he liked the task, its effectiveness and complexity. Thus, a data set is formed which characterizes the educational tasks' performance by users and is used in the future to form relevant recommendations.
The recommendations module is designed to form a personalized learning path in accordance with the values of the employee and the organization and to select the most relevant educational material for the employee.
It is possible to determine the soft skills directions' development of an employee within the interests of the organization if there is an
G
Question: 5/64.
When I face a problem, I
Ö
rely on proven approaches 0 1 13 4
do a thorough analysis 0 1 13 4
look for a creative solution 0 1 13 4
rely on my feelings 0 1 13 4
rate each answer a score from 1 to 4, that describes how statement is close to you, 4 is closest
back to the profile
Fig. 2. Diagnostic module.
employee and position profile. However, as mentioned earlier, it is necessary to synchronize the values of the employee and the organization. So it is also necessary to determine the desired development directions for the employee. Within the framework, an employee development needs questionnaire depending on his value orientations has been developed.
After the diagnostics, employee is asked to answer a list of questions to form the desired employee profile and determine the competencies that he would like to develop
The overlapping of the obtained profiles on each other makes it possible to visualize the level of competencies' expression and to identify the directions of an employee's personal development in accordance with the needs of the organization, all of which leads to the formation of their common development path.
The next step is the direct selection of relevant educational material using a recommendation system. To select an algorithm for its operation, a study was conducted on technologies such as machine learning based on matrix factorization and a neural network.
2. Computational experiment 2.1. Problem Statement
There are many users U = {u1, u2, ... , un} and many educational tasks T = {t1, t2, ... , tm}.
The Vnxm matrix contains the scores assigned by users to educational tasks. There will be a number in place v.(i<e 1, ... n, je 1 ... m) if the ul user has evaluated the task tj and is empty otherwise.
It is required to find a vector v~t containing the user's already known estimates v.. , as well as the
DIAGNOSTICS
EDUCATION
ORGANIZATIONAL MANAGEMENT
PROFILE
Tasks
Active task
[competence: Benevolence]
Read the poem about benevolence "The Way of Benevolence'
complete the task
Available tasks (picked randomly, override by administrator)
[competence: Objectivity]
Read nan-fictian: A Briefer History of Time (Hawking and MLodinow book)
[competence: Energy]
Listen to your favorite music whiLe exercising
[competence: Experimentation]
Put yourself to the challenge more often. Goto a church that is not of your faith.
[competence: Logical thinking]
Try to think and make the choice yourself, otherwise others will do it instead of you
[competence: Diligence]
Find a way to tactfully tell your employee he is fired
[competence: Creative thinking]
Listen to the sounds outside the window, breathe in the smell coming from
the kitchen, look around, feel the touch of a warm blanket on your feet.
Try to maintain this state for at least a few minutes. Increase the time each time.
take this task
take this task
take this task
take this task
take this task
Recent completed tasks (total: 18)
[competence: Impatience]
How do you feel about bribery in the country?
Ö36.0- ft \2 ß
[competence: Abstractness]
Compare moped and motorcycle, find similarities and differences.
G55.0- ft \3 ß
Fig.3. Training module.
expected ones v^. Based on the obtained vector, you need to rank the list of educational tasks T
for the u. user.
i
2.2. Machine learning based on matrix factorization
Matrix factorization implies decomposition of the original matrix into a product of two matrices of small rank. The user's interaction with the object is modeled as a scalar product of the vectors of user's imagine and the object in the factor space. Factorization models work effectively with highly sparse matrices [16—18].
Let us imagine the evaluation matrix Fas a product of two matrices:
♦ W v, matrix, which contains both latent user characteristics and explicit ones (gender, education, work experience and user profile values consisting of 16 parameters);
♦ matrix, which is characterized educa-
fcxm '
tional tasks.
Let us fill in the latent characteristics of the W and H matrices with random variables based on the law of uniform distribution on the interval [o^maxfy}/*]. Next, we will solve the minimization problem:
3X&mrn\\V-V\\+a\\H\\+p\\W\\, (1)
where V is the matrix obtained by approximation from W and H;
а,P — algorithm parameters.
Each algorithm's iteration for finding a solution to minimize the error consists of the following steps:
1. Fix the H matrix.
2. Find the error 6 = |vy— v„|,je 1, ..., n.
3. Find new values W.p= W.p - + X Wip\ where p e 1, ..., k;
X — regularization parameter; € — learning rate.
4. Fix the W matrix.
5. Find the error S= |vy — v„|1,..., m.
б. Find new values HpJ — e(SWj+ XHpJ), where p e 1, ..., k;
X — regularization parameter;
€ — learning rate.
When solving the problem experimentally, the rank of the matrices Wand H k = 40 was found. This value is a boundary value for a given case in which the root-mean-square and absolute error of solving the factorization problem are optimal. With a value of k > 40, the advantage is not traced as a result, and the
calculation time increases. Conversely, with a value of k < 40, too rough an approximation is obtained.
Based on the test results (Table 1), with an increase in the learning rate from 0.005 to 0.03, there is an improvement in the results (a decrease in error, an increase in prediction accuracy). At the value e = 0.04, the results are usually equivalent to the results at e = 0.03 or worse. With the number of iterations 20 (e = 0.03), high accuracy of factorization is already observed on the given samples. Based on these results, we use the following parameters for the algorithm: e = 0.03; the number of iterations is 20.
With an increase in the amount of data received and a decrease in the sparsity of data, it may be necessary to increase the number of iterations. The control of the number of iterations was automated by testing after the end of factorization and comparing the accuracy of the prediction with the previous testing. As the prediction accuracy decreases, the number of iterations increases. Otherwise the number of iterations remains the same. Another alternative is to set the permissible accuracy of factorization, at which the algorithm should be stopped.
Table 1.
Speed and factorization errors (n = 14000)
Iterations' number £ 0.005 0.01 0.02 0.03 0.04
20 absolute average 0.197 0.119 0.064 0.05 0.048
the quadratic mean 0.24 0.09 0.013 0.007 0.0064
time, sec. 6 6 5 5 5
40 absolute average 0.113 0.057 0.028 0.02 0.022
the quadratic mean 0.082 0.013 0.002 0.001 0.001
time, sec. 11 10 10 11 11
60 absolute average 0.075 0.03 0.017 0.014 0.014
the quadratic mean 0.03 0.003 0.0009 0.0006 0.0006
time, sec. 16 15 16 15 15
One of the main tasks in the construction of recommendation systems is to solve the problem of 'cold' start [19]. It occurs when new elements appear in the system: whether it is a user or preference objects.
In the problem under consideration, two variants of cold start of users can be distinguished: the user did not perform the educational tasks; the user performed the educational tasks, but the factorization task was solved before his registration; that is to say, there are no user factors in the system.
Let us consider the first case. Nothing is known about the user's preferences. We know only about his profile. It is some vector which is formed from the values of gender, seniority, position. If the user has passed the test, then the vector will also have a soft skills profile of 16 components.
The problem of cold start in this case is proposed to be solved using cosine similarity. To do this, there is a cosine between the new user and each existing user. It is ranked in descending order, and the user to whom the maximum cosine corresponds is selected. As a result, recommendations are formed for a new user already based on the similarity of the user found through the cosine:
(wm,wt)
cos <p - _
\Wa\\x\\Wb\
(2)
In the second case, the problem is proposed to be decided by solving the factorization problem for a specific user. In this case, the Wand V matrices will take the form of row vectors. The problem factors H matrix will be fixed, and the search for values will be performed only for the W vector-string. The search algorithm is reduced to a special case: i = 1, and the steps of the matrix factorization algorithm from the 4th to the 6th are excluded.
We also need to note the 'cold' start problem for educational tasks. At the current stage of
the study, tasks for which there are no factors are randomly offered to users.
To assess the quality of predictions, the set of V estimates was divided into samples for V ■
r train
training and Vtest testing. The model was tested with the following quantitative characteristics:
♦ Users' number: 303;
♦ Task's number: 11726;
♦ Task's number completed by users: 17733;
♦ Training sample: 75%;
♦ Test sample: 25%;
♦ Matrices' rank: 40;
♦ Factorization steps: 20;
♦ Range of original matrix' values: 0—10 (normalized data).
The V matrix is very sparse; most of the cells are not filled (more than 95% of the cells). When evaluating the performance of the model, three types of approximation were carried out: for efficiency, complexity and preferences. Four cases were considered:
Using only latent factors (marked as FREE on graphs);
Using fixed user factors along with latent (soft skills profile, gender, position, education; designated as UV);
Using fixed task factors along with latent ones (the average values of performance indicators, complexity and preferences set by users, the level of complexity, the orientation of tasks to competencies, are designated as TV);
Using fixed factors, users and tasks (designated as UV+TV).
Fifty factorizations were carried out on random data for each case. Then these data were averaged. The testing also took into account the cold start of users based on cosine similarity. The cold start of tasks was not taken into account in testing.
per cent of sample 100 -,.......................
90 80 70 60 -50 -40 -30 -20 -10 -0
— FREE
- UV
. TV -- UV+TV
0 1 0 2 0 30 40 50 60 70 80 90 1 00 110 1 20 1 30 1 40 1 50
Fig. 4. Error in predicting preference using factorization on percentage intervals.
per cent of sample 100 -
90 -80 -70 -60 -50 -40 -30 -20 -10 -0
FREE
UV
TV
UV+TV
0 10 20 30 40 50 60 70
90 1 00 110 1 20 130 140 150
Fig. 6. Error in predicting preference using cosine similarity on percentage intervals.
The data for the preference prediction error are presented in Fig. 4—7. The data for the efficiency prediction error are presented in Fig. 8-11.
per cent of sample 100
FREE
UV
TV
UV+TV
Fig. 5. Error in predicting preference using factorization on discrete values [0; 1].
per cent of sample 100-1.........................
9080706050403020100
FREE
UV
TV
UV+TV
Fig. 7. Error in predicting preference using cosine similarity on discrete quantities [0; 1].
The fixed factors' use mainly worsens the result, as can be seen from the graphs (Fig. 4-11). This may be due to the small size of the data set.
0
0
per cent of sample 100
90 -
80 -
50 -|
40
30 -20 -10 0
. FREE
— UV
—— TV
. UV+TV ...........
0 10 20 30 40 50
70 80 90 100 110 120 130 140 150
Fig. 8. Error in predicting efficiency using factorization on percentage intervals.
per cent of sample
40 -
35 -
30 -
25 -
29
15
10 -
5 -
0 10 20 30 40 50
70 80 90 100 110 120 130 140 150
Fig. 10. Error in predicting efficiency using cosine similarity on percentage intervals.
2.3. Neural network
A neural network is a computational structure assembled into computational elements' network created in the likeness of a biological neuron [20]. Neural networks are capable of
per cent of sample
70-
60-
50-
40-
30-
20-
10-
1 2 3
Fig. 9. Error in predicting efficiency using factorization on discrete quantities [1; 5].
per cent of sample 70605040-
30-
20-
10-
0 1 2 3 4
Fig. 11. Error in predicting efficiency using cosine similarity on percentage intervals and on discrete quantities [1; 5].
solving a wide range of tasks in various fields of activity [21].
To solve this problem, such neural network architectures as a convolutional neural network [22] and a multilayer perceptron accord-
70
60
0
0
4
0
0
ing to Rumelhart [23] were considered. The latter proved to be the most effective. In addition, a convolutional neural network requires more data and training time.
Two methods were investigated: encoding words with a unique number and embedding words (Word2Vec) to vectorize the educational task [24]. Encoding words with a unique number is faster, but it does not allow you to record any relationships between words, as well as the similarity of words.
Using the Word2Vec model for word embedding allows you to capture subtle relationships between words, but requires a large amount of data for training and is a lengthy process. Due to the absence of the need for constant vec-torization of educational tasks, but only when replenishing the database of educational tasks, the Word2Vec model was used.
The input layer of the neural network has one neuron, which receives a vector containing data about the user and tasks.
The activation function 'ReLU' is activated on the hidden layers, the activation function 'Sigmoid' is activated on the output layer, which allows you to amplify weak signals and is not saturated by strong signals [25].
During training and testing the neural network, a data set consisting of 16 862 vectors containing data about users and tasks was used. The data were divided into training and test samples in the proportion of 80/20.
On the studied data set, the neural network with the following topology showed the great-
est accuracy: two hidden layers (the first layer — 64 neurons, the second layer — 128 neurons), the size of the batch — 32, the number of epochs — 20.
The neural network results and the prediction accuracy are presented in Table 2.
To construct a vector, the network is launched as many times as the values are contained in the database of educational tasks. Thus, after all iterations, a vector v~t is formed on the basis of which it is possible to select the most preferred tasks for the user.
3. A comparison of Artificial Intelligence technologies
Two approaches to the formation of recommendations were considered: neural network and machine learning based on matrix factorization. Both approaches have both advantages and disadvantages.
As opposed to vectors fed to the neural network input, the factor matrix consists of unique user vectors that contain latent features that are not taken into account by the neural network. The neural network, in turn, relies only on a given known user vector consisting of parameters laid down by the researcher. It allows us to talk about a more personalized approach when using matrix factorization.
The neural network also requires the mandatory availability of not only all data about the user, but also their correctness. When using machine learning based on matrix factoriza-
Table 2.
Neural network results
Preference Complexity Effectiveness
Standard deviation 0.056 0.102 0.098
Accuracy, % 93.31 68.79 61.0
tion, you can make recommendations after several evaluations without having any more information.
The matrix factorization's disadvantage is a cold start, when the user has not yet evaluated any tasks, and he needs to give recommendations. However, this problem is solved very quickly using the method described earlier.
Despite the fact that the researchers were faced with the task of ranking, it was necessary to compare the work of these methods according to three criteria: preferences (binary value), complexity (discrete [1; 4]) and efficiency (discrete [1; 5]). The comparison results are presented in Table 3.
As can be seen from Table 3, on discrete values, the accuracy of neural network predictions decreases compared to matrix factorization. Based on the above, the researchers decided to use machine learning based on matrix factorization as the core of the recommendation module.
Conclusion
The problem of low adaptability of management in the conditions of digital transformation was considered in this article. Within the research framework, a model of an employee's intelligent guide has been developed. This model allows us to level this problem due to the qualitative development of the employee's competence profile, taking into account the value orientations of the employee and the organization. The model's implementa-
tion has been carried out in the form of a software component for the enterprise management system.
The new competencies' development will lead to the expansion of the employee's profile and ability to perform new tasks in the area of interest, thereby obtaining a high-quality result. The organization, in turn, will receive new opportunities for development in the market and rapid adaptation to changing conditions. Thus, by investing in employee development, the organization receives T-shaped specialists with a proactive position and capable of self-organization. This directly leads to the realization of the potentials of both the employee and the organization.
The intelligent guide has a modular architecture, which is due to the interdiscipli-narity and novelty of the research, as well as the simplicity of implementing the received developments into real management practice. This approach made it possible to conduct qualitative research in two projections: the development of comprehensive automated soft skills diagnostics of an employee's profile and relevant filling of an individual educational path with educational material using artificial intelligence, and then to combine the results into a single system for training and staff development.
In the proposed approach artificial intelligence is expressed by machine learning using matrix factorization. Such intelligence allows you to qualitatively select educational material which will be interesting to the employee from his personal positions and maximally involve
Table 3.
Accuracy of artificial intelligence technologies' predictions
Preference Complexity Effectiveness
Neural network 93.3% 68.8 61%
Matrix factorization 93.1% 76.4 62.4%
him not only in the learning and development process, but also in achieving organizational goals.
The proposed database structure allows you to collect digital traces of users describing biographical characteristics, skills and level of their proficiency at a certain stage of development, the history of choosing and completing educational tasks, and further use of the collected data as the basis for the work of the recommendations module and the formation of personalized proposals.
The interdisciplinarity and novelty of the research also determine the variability of its development. As part of further research, it is planned to integrate the software component into the enterprise's business process management system. This is supposed to predict the time when the labor resource is not loaded and offer educational tasks to fill it. It will be the basis of a fundamentally new approach to
considering downtime not as a loss, but as an opportunity for the development and training of employees.
The developed software component can be used by enterprises not only at the organizational level, but also through broadcasting in the education system. On the one hand, enterprises will be able to reduce staff downtime, expand the personal competence profile of employees, and increase the growth of the production potential of both employees and enterprises in the market. On the other hand, educational institutions will be able to train highly qualified personnel within the framework of orientation to the values of the market and the values of students with the formation of individual educational trajectories. Such cooperation will make it possible to form an education ecosystem in accordance with the requirements of innovative development of a region's economy. ■
References
1. Deloitte Insights (2020) Results of the study "International trends in human resource management — 2020" in Russia. Available at: https://www2.deloitte.com/content/dam/Deloitte/ru/Documents/ human-capital/russian/hc-trends-2020-Russia.pdf (accessed 22 November 2021) (in Russian).
2. TAdviser (2020) Big Data global market. Available at: https://tadviser.com/a/e.php?id=129607 (accessed 22 November 2021).
3. Fatyanov A.A. (2018) Big Data in the digital economy: its value and legal challenges. Economics. Law. Society, no. 4, pp. 37—40 (in Russian).
4. Reinsel D., Gantz J., Rydning J. (2018) The digitization of the world — from edge to core. IDC White Paper. Available at: https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf (accessed 22 November 2021).
5. Ali N.M., Gadallah A.M., Hefny H.A., Novikov B.A. (2021) Online web navigation assistant. Vestnik Udmurtskogo Universiteta. Matematika. Mekhanika. Komp'uternye Nauki, vol. 31, no. 1, pp. 116—131. https://doi.org/10.35634/vm210109
6. Gokhberg L. (ed.) (2019) What is the digital economy? Trends, competencies, measurement. Moscow: HSE (in Russian).
7. Kulikova M.Kh., Kulikova M.Kh., Magomadov M.A. (2020) IT in education: today, tomorrow and always. Proceedings of the I Student Scientific and Practical Conference "Information technologies in business and education", Grozniy, 20February 2020, pp. 85-90 (in Russian). https://doi.org/10.36684/21-2020-1-32-35
8. Geissbauer R., Lubben E., Schrauf S., Pillsbury S. (2018) Digital Champions. How industry leaders build integrated operations ecosystems to deliver end-to-end customer solutions. PwC Strategy& Global Digital Operations Study. Available at: https://www.strategyand.pwc.com/gx/en/insights/industry4-0/ global-digital-operations-study-digital-champions.pdf (accessed 22 November 2021).
9. Harvard Business Review (2017) High-performance sourcing and procurement driving value through collaboration. Harvard Business School Publishing. Available at: https://hbr.org/resources/pdfs/ comm/scoutrfp/HighPerformanceSourcing.pdf (accessed 22 November 2021).
10. Dolganova O.I., Deeva E.A. (2019) Company readiness for digital transformations: problems and diagnosis. Business Informatics, vol. 13, no 2, pp. 59-72. http://doi.org/10.17323/1998-0663.2019.2.59.72
11. Komarov V.A., Sarafanov A.V. (2021) IoT systems in the process of multidisciplinary training of personnel for the digital economy and their design. Business Informatics, vol. 15, no 2, pp. 47-59. http://doi.org/10.17323/2587-814X.202L2.47.59
12. Rivera A.C., Tapia-Leon M., Lujan-Mora S. (2018) Recommendation systems in education:
A systematic mapping study. Proceedings of the International Conference on Information Technology & Systems (ICITS 2018), Libertad City, Ecuador, 10—12 January 2018 (eds. A. Rocha, T. Guarda). Advances in Intelligent Systems and Computing, vol. 721, pp. 937-947. https://doi.org/10.1007/978-3-319-73450-7_89
13. Bustos López M., Alor-Hernández G., Sánchez-Cervantes J., Paredes-Valverde M., Salas-Zárate M.P. (2020) EduRecomSys: An educational resource recommender system based on collaborative filtering and emotion detection. Interacting with Computers, vol. 32, no. 4, pp. 407-432. https://doi.org/10.1093/iwc/iwab001
14. Urdaneta-Ponte M.C., Mendez-Zorrilla A., Oleagordia-Ruiz I. (2021) Recommendation systems for education: Systematic review. Electronics, vol. 10, no. 14, article ID 1611. https://doi.org/10.3390/electronics10141611
15. Bughin J., Hazan E., Lund S., Dahlstrom P., Wiesinger A., Subramaniam A. (2018) Skill shift: automation and the future of the workforce. McKinsey & Company. Available at: https://www.mckinsey. com/featured-insights/future-of-work/skill-shift-automation-and-the-future-of-the-workforce (accessed 22 November 2021).
16. Rubtsov V.N. (2017) Matrix factorization based on deep learning for collaborative filtering. Student Theses. Moscow: HSE. Available at: https://www.hse.ru/en/edu/vkr/206744221 (accessed 22 November 2021) (in Russian).
17. Stromqvist Z. (2018) Matrix factorization in recommender systems: How sensitive are matrix factorization models to sparsity. Uppsala University Publications. Available at: https://uu.diva-portal.org/smash/ get/diva2:1214390/FULLTEXT01.pdf (accessed 22 November 2021).
18. Mojsyuk-Dran'ko P.A., Revotyuk M.P. (2020) Matrix factorization methods for recommendation systems. Proceedings of the International Conference Information Technologies and Systems 2020 (ITS
2020). Minsk: Belarusian State University of Informatics and Radioelectronics, pp. 193-194. Available at: https://libeldoc.bsuir.by/bitstream/123456789/41339/1/Moysyuk_Dranko_Metody.pdf (accessed 22 November 2021) (in Russian).
19. Kuznetsov I.A. (2019) Machine learning methods and algorithms for preprocessing and classification of semi-structured text data in scientific recommendation systems. Moscow: NRNU MEPhI. Available at: https://ds.mephi.ru/documents/90/Кузнецов_H_А_Тексг_flнссертацнн.pdf (accessed 22 November
2021) (in Russian).
20. Golovko V.A., Krasnoproshin V.V. (2017) Neural network data processing technologies. Minsk: Belarusian State University. Available at: https://elib.bsu.by/bitstream/123456789/193558/1/Golovko. pdf (accessed 22 November 2021) (in Russian).
21. Lisovsky A.L. (2020) Application of neural network technologies for management development of systems. Strategic decisions and risk management, vol. 11, no. 4, pp. 378-389 (in Russian). https://doi.org/10.17747/2618-947X-923
22. Bezdan T., Bacanin Dzakula N. (2019) Convolutional neural network layers and architectures. Proceedings of the Sinteza 2019: International Scientific Conference on Information Technology and Data Related Research, Belgrade, 20 April 2019 (ed. Milovan Stanisic), pp. 445-451. https://doi.org/10.15308/Sinteza-2019-445-451
23. Postarnak, D.V. (2012) The critical analysis of models of neural networks. Vestnik Tyumenskogo gosudarstvennogo universiteta, no. 4, pp. 162—167. Available at: https://elibrary.ru/item. asp?id=17758787& (accessed 22 November 2021) (in Russian).
24. Young T., Hazarika D., Poria S., Cambria E. (2018) Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine, vol. 13, no. 3, pp. 55—75. https://doi.org/10.1109/MCI.2018.2840738
25. Iqbal T., Qureshi S. (2020) The survey: Text generation models in deep learning. Journal of King Saud University — Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2020.04.001
About the authors
Ekaterina S. Morozevich
Process architect, Research Management Department, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarsky Rabochy Av., Krasnoyarsk 660037, Russia;
E-mail: katyamorozevich@mail.ru
ORCID: 0000-0001-8796-3157
Vladimir S. Korotkikh
System administrator, Municipal budget general education institution secondary school No. 4, 9, st. Naberezhnaya, Divnogorsk 663091, Russia;
E-mail: vskor@bk.ru
ORCID: 0000-0001-9270-3376
Yevgeniya A. Kuznetsova
Engineer, Research Management Department, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarsky Rabochy Av., Krasnoyarsk 660037, Russia;
E-mail: zhenya.kuz-1997@yandex.ru
ORCID: 0000-0002-3559-8394
