CC BY
182
DESIGNING E-LEARNING ENVIRONMENT BASED ON STUDENT PREFERENCES: CONJOINT ANALYSIS
APPROACH
Dr. Marija Kuzmanovic, University of Belgrade, Faculty of Organizational Sciences, Serbia E-mail: marija.kuzmanovic@foabg.ac.rs Dr. Jelena Andjelkovic Labrovic, University of Belgrade, Faculty of Organizational Sciences, Serbia E-mail: jelena.andjelkovic.labrovic@fon.bg.ac.rs Dr. Ana Nikodijevic, University of Belgrade, Faculty of Organizational Sciences, Serbia E-mail: ana.nikodijevic@fon.bg.ac.rs
A R T I C L E I N F O
Original Research Received: October, 07.2019. Revised: November, 14.2019. Accepted: November, 28.2019. doi: 10.5937/IJCRSEE1903037K
UDK
37.018.43:004 371.39
Keywords:
e-learning, studentpreferences, conjoint analysis, heterogeneity, segmentation.
A B S T R A C T
The aim of this paper was to determine students' preferences towards e-learning environment in order to select and design its components that suit the needs of student's best. The research was implemented using conjoint analysis. Three dimensions of interest were considered: e-learning technology, teaching method and knowledge assessment and the results show that knowledge assessment is the most important e-learning attribute for both traditional and online students. Adding into consideration the teaching method as well, further analysis showed that students can be profiled in two segments: oriented on results or process, which can be used at the beginning of studies to adjust e-learning environment. Research findings emphasized student preferences as essential for designing e-learning system, while student satisfaction turned out to be a key factor determining their persistence for studying in e-learning environment. Finally, recommendations for improvement of existing e-learning system were given.
© 2019 IJCRSEE. All rights reserved.
1. INTRODUCTION
In recent years, e-learning has become a sustainable delivery system for all educational levels (Yu, Yu and Lin, 2010). Rosenberg, and Foshay (2002) defined e-learning as the use of Internet technologies for delivering different solutions that enhance knowledge and performance. In educational process e-learning could be seen as content delivery method that enables synchronous and asynchronous information exchange over the network (Oncu and Cakir, 2011).
The goal of an educator in modern educational settings should be to design online courses thatare interactive and learner-centered. For an effective online course delivery very
Corresponding Author
Dr. Marija Kuzmanovic, University of Belgrade, Faculty of Organizational Sciences E-mail: marija.kuzmanovic@fon.bg.ac.rs
This work is licensed under a Creative Commons Attribution - NonCommercial - NoDerivs 4.0. The article is published with Open Access at www.ijcrsee.com
important part is the e-learning environment design. The online learning environment, in addition to sensibility and awareness, also requires an understanding of the online learners and the challenges they face (Conrad, 2008). Black, Ferdig and DiPietro (2008) highlighted in their research the importance of the e-learning environment through a factor they classified as a case instance rating. As opposed to a course content evaluation, course instance evaluations focus on the classroom environment, community and grades, so it should involve "a specific teacher, a group of students, a course, and a particular learning management system" (Black et al., 2008). A key component in the e-learning environment is the student. Therefore, understanding how student characteristics can influence the teaching and learning process in an online environment is crucial to design effective e-learning instruction (Roberts, 2010).
Although curriculum delivery via learning management systems is widespread within university practices all over the world, it has only recently emerged in Serbia. Faculty of Organizational Sciences, University of
Belgrade (FON), is one of the first schools in Serbia that accredited e-learning studying program for 60 students per year. The accredited program has been implemented since 2010 with 46 courses using Moodle platform. Main reason for conducting this research is authors' desire to improve existing application of this e-learning system. It is indicative that students usually transfer from e-learning to traditional type of studying when given the opportunity. Previous experience also shows that students are less engaged in studying after first year and when they enroll in final year of studying they usually do not use implemented e-learning system at all. This is verified by the fact there is only 6 courses on final year of studies and extremely little student engagement, when compared to all 14 courses on first year and full student engagement. This research is focused on finding some of the reasons for this problem.
The premise underlying this study is that students prefer certain online components more than others and that it is possible to determine which ones most affect their attitudes. Accordingly, the study does not examine specific courses but a diverse sample of students who are engaged in: (a) fully online studying through a Moodle learning environment, (b) traditional offline, face to face studying with some experience with blended courses (combination of traditional and e-learning), (c) attending open online courses outside curricula (edX, Coursera, etc.). Horvat, Dobrota, Krsmanovic and Cudanov (2014) examined students' perceptions of e-learning courses on the Moodle platform at FON and concluded that students using Moodle only before the exam were significantly less satisfied than students using it on a daily basis. They suggested that feedback on the quality features from students who do not use Moodle regularly should be taken into account.
In this study we attempt to determine students' preferences for using the e-learning system with the goal of improving the existing one at FON. Another research objective is of greater importance because it indicates how student preferences can be used to select e-learning system components during the development process to reduce dropout of distance learning students.
To evaluate students' preferences towards the effectiveness of e-learning course, this study employs a multivariate research technique Conjoint analysis. Conjoint analysis is based on the assumption that any service can be described as a combination of different
levels of multiple attributes and implies that individuals evaluate services by considering jointly those attributes.
Conjoint analysis has been successfully applied in the education industry for many years to reveal students' preferences for different aspects of education (Tashchian and Freiden, 1983; Soutar and Turner, 2002; Sohn and Ju, 2010; Taylor, Humphreys, Singley, and Hunter, 2004; Won and Bravo, 2009; Kuzmanovic, Savic, Popovic and Martic, 2013; Azarcon, Gallardo, Anacin and Velasco, 2014; Zwarts, Vanthournout, Gijbels and Van den Bossche, 2015; Popovic, Vagic, Kuzmanovic and Andelkovic Labrovic, 2016; Walsh and Cullinan, 2017; Carey et al., 2018).
However, only a few studies have so far used conjoint analysis within e-learning environment. Van Der Rhee, Verma, Plaschka and Kickul (2007) used choice-based conjoint analysis to understand student preferences for e-learning technologies. Pomales-Garcia, Yili and Lopez (2009) evaluated the relative importance of six design dimensions from both student and researcher perspective. Daghan and Akkoyunlu (2012) used conjoint analysis to measure students' preferences for e-learning styles. Sun and Wang (2014) used conjoint analysis to design and development of e-learning tools in higher education depending on learning tasks. Recently, some researchers used conjoint analysis to understand the preference towards online learning in developing countries (Malarkodi, Indumathi, and Praveena, 2018; Acharya and Lee, 2018).
2. RESEARCH FRAMEWORK
The aim of this study was to gain a broad insight into the preferences of e-learning students in Serbia and to explore factors that most contribute to their satisfaction. Following research questions were in focus: Which e-learning components do students prefer? Can students be profiled in different segments according to their preferences? And if so, what can be done to meet the needs and wants of particular groups of students? Three dimensions of interest were defined: e-learning technology, teaching method and knowledge assessment. They were chosen by considering previous research (Katz, 2002; Ferguson and DeFelice, 2010; Won and Bravo, 2009) and authors' own experience in teaching in e-learning environments.
The learning environment or technology used for teaching is important because
students' satisfaction with e-learning is based on their attitude towards ICT (Malik, 2009). Technology is generally not a barrier to the success of participants in e-learning and most students have necessary skills to fully engage in e-learning environments. Drennan, Kennedy and Pisarski (2005) found that student satisfaction is influenced by positive perceptions towards technology, in terms of ease of access.
Teaching method is the most complex dimension as it comprises several different aspects, mainly related to interaction. According to several authors (Bouhnik and Marcus, 2006; Katz, 2002), interaction is one of the most important factors of e-learning, and can be analyzed on several levels: interaction with content, teacher, other students and the system. Mijatovic, Cudanov, Jednak and Kadijevich (2013) conducted a survey among students who use the Moodle platform for the first time and concluded that active participation in the class along with interactive use of the system has a strong positive impact on student achievement. Bouhnik and Marcus (2006) found that one of the things students are dissatisfied with is the lack of contact and interaction with peers. Jung, Choi, Lim and Leem (2002) revealed that students who interacted with each other expressed the highest degree of satisfaction. Nummenmaa and Nummenmaa (2008) came to similar conclusion, finding that students who did not interact with others had more negative emotional experiences. The teacher's role is very important and their timely response has a positive effect on student satisfaction (Malik, 2009). Clearly defined objectives, assignments and deadlines, must be present to increase student satisfaction as well (Stein, 2004).
We found no research results in the literature regarding student preferences in knowledge assessment (structure of knowledge assessment, summative assessment design) within e-learning environment, but we believe this dimension might have an impact on students' satisfaction and therefore we included it in this study. The details on all three dimensions are given below.
3. SURVEY PROCEDURE
In order to measure students' preference towards key attributes of the e-learning environment, this study followed five key steps:
1. Specifying key attributes and attribute
levels;
2. Choosing a presentation method and construction of efficient experimental design;
3. Questionnaire design and research implementation
4. Model specification and the estimation technique selection
5. Cluster analysis.
The study should include all studentrelevant attributes that can be managed by the university. Attribute levels must be credible, effective and capable of being traded-off against each other. Usually, attributes and levels are determined by reviewing the research literature and conducting pilot research, but also taking into account expert opinions. In this paper eight key e-learning attributes based on literature review and students and faculty members' opinion are selected (see Table 1).
Table 1. Student satisfaction with online courses
Dimension Attribute Level
Customize emiion- Yes menl based on prefer-
No
I ences
-i-leamnis Simple environment, no training
technology necessarv
Simplicity ---
Environment requires trailing
or previous experience Recorded lecture with slides and Lectures sound
Classroom Live broadcasting Interactive wosfc (dis- Yes cushions, assignments, case studies No
_ . . Cooperation with Individual assignments
Teaching ---
^ j other students Group assignments
Flexibility for pre- Yes
exam assignments
(dear deadlines or No
flexible deadlines)
Online (e-mail, chat forums.
Communication with
etc.)
teacher -
Online ajid face-to-face
100% of pre-sxam assignments and exam in e-leaming environment
10% of pre-sxam assignments in e-leaining environment .and
30% offline exam_
30% of pre-sxam assignments in e-leanaing environment, and 70% offline exam_
For a given number of attributes and levels, it is possible to create 384 (27 x 3) combinations, that is, concepts that students need to evaluate. This kind of experiment plan is called a complete factorial experimental design and because of its complexity it is rarely used in practice. An efficient, yet cognitively
m
, n Method of assess-Knotvledse
ment of knowledge
assessment
acceptable design contains a subset of all concepts and is called a reduced experimental design. For the purpose of this study, we used the SPSS Orthoplan Component to generate an efficient experimental design with 16 concepts. Two holdout concepts were added to the design for the purpose of checking the quality of the respondents' responses.
Data on individuals' preferences and satisfaction were collected through an online survey distributed to both traditional students and those studying distance learning. Students were asked to rate each concept on an ordinal scale ranging from 1 ('least preferred') to 5 ('most preferred'). In addition to conjoint assignments, the questionnaire also contained a number of questions regarding the socio-demographic characteristics of students as well as their attitudes and satisfaction with their existing mode of study.
After collecting the students' answers, they need to be analyzed. A linear additive model (so called part-worth utility model) was used to model the preferences in this study. The model assumes that the total utility of any concept is estimated as the sum of the partial utilities of the attribute levels contained in that concept. Therefore, the overall utility of the concept j for the student i in a given study can be expressed as follows:
8 Li
U«=1l1L + 1=1,—,I, j = \,...,J
k=1 /=1
where k denotes an attribute and Lk is the number of levels of a given attribute. pikl is student i's utility associated to the level l of the attribute k (part-worth utility). An independent variable xjkl indicate the presence (xjkl = 1) or absence (xjkl =0) of the level l of the attribute k in the concept j. i. is a stochastic error. Given that each concep¿should contain exactly one level of each attribute, the following condition must be met:
/=i
Using the collected ratings 16 concepts from the experimental design, the Pikl parameters are estimated by the least-squares method. Estimated values provide a quantitative measure of the preference for each attribute level, with higher values corresponding to a greater preference. The relative importance of an attribute k for student i can be calculated as follows:
IIR
= i = l,...,I, k=l,...,S.
I URIk
k=1
where UR is the utility range i.e. the difference between highest and lowest partial utility values for each attribute. Importance scores can be further aggregated to include students with similar preferences.
Understanding what students most value in studying allows university management to tailor study system and programs to communicate those benefits and redesign existing system with those benefits in mind. This is especially important if there is heterogeneity in student preferences. There are two main approaches for identifying heterogeneity in preferences. Segments can be identified either a priori, according to variables known in advance, or post hoc based on individual preferences revealed by conjoint analysis. Both the part-worth utilities and the resulting importance ratings can be used as an inputs in cluster analysis.
4. RESULTS 4.1. Sample characteristics
A total of 121 students completed the questionnaire. Six responses (4.94%) were excluded from the analysis either because of incompleteness or inconsistency of the answers, so the total number of valid answers used in the further analysis was 115 (95.04%). The sample consisted of 39 (33.9%) male and 76 (66.1%) female participants, aged 21.5 (SD=1.789) in average. When it comes to mode of studying, 70 (60.9%) traditionally-taught respondents and 45 (39.1%) of them are online students. Most students (96.50%) took at least one course through Moodle during the studies (Table 2).
Table 2. Student satisfaction with online courses
Number of respondents Mean Min Max
Mode of studying Traditional 66 3.96 2 5
Online 45 2.88 1 4
Total 111 3.54 1 5
Open online Coursera 15 4.56 3 5
courses out- edX 4 4.75 4 5
side curric- Other 5 4.60 3 5
ula Total 24 4.60 3 5
Traditional students are more satisfied with this learning platform (average score of 3.96 out of 5), than online students (with average score of 2.88). Looking at the range of scores, it is obvious that none of traditional students gave grade 1 to Moodle, and none of the online students gave grade 5. The assumption behind this fact is that traditional students mostly took one online course while online students took all or most courses online. These courses are usually designed and applied according to teachers' preferences and different Moodle components. There are 21 (18.26%) students who took one or more open online courses outside curricula (24 graded courses). Average grade for these courses is 4.60, meaning that all those courses are graded much higher in comparison to core studies. None of open online courses was graded with 1 or 2.
4.2. Aggregated preferences
students'
We used the SPSS 16.0 to estimate the model parameters both individually for each student in the sample and aggregated for the sample as a whole (see Table 3). Higher values of utility part-worth utility values indicate a stronger preference. Kendall's tau
evaluate the internal and predictive validity of the model. The values of 0.924 and 0.988 respectively indicate a high predictive validity of the model and confirms significance of the estimated parameters. Kendall's tau with value of 1.000 for the two holdout concepts further confirms the quality of respondents' answers.
Table 3 shows that method of assessment of knowledge was found to have the most significant influence on students' preferences in e-learning environment with a relative importance of23.76%. The importance of other attributes is by far lower, whereas flexibility for preexam assignments was regarded as the least-valued attribute (7.69%). Looking at the most important attribute, students prefer the option to do all of preexam assignments and final exam in e-learning environment. Students also have positive but lower preferences towards the option to do 70% of preexam assignments in e-learning environment. The last attribute level (30% of preexam assignments in e-learning environment) has a negative impact on their preferences.
Higher values of standard error for estimated parameters indicate there is heterogeneity in students' preferences. Based on data in Table 3 it is obvious that standard error is somewhat higher for method of assessment of knowledge, and therefore it is expected that students' preferences for this attribute are heterogeneous.
statistics and Pearson coefficient were used to
Table 3. Aggregated part-worth utilities and importance of e-learning attributes
Attribute Attribute level Part-worths Std. error Attributes impoitance
Customize environment Yes 0.105 0.023 9.31%
based on preferences No -0.105 0.023
Simplicity Simple environment, no training necessary 0.172 0.023 12.36%
Requires training or previous experience -0.172 0.023
Lectures Recorded lecture with slides and sound 0.057 0.023 13.24%
Classroom live broadcasting -0.057 0.023
Interactive work Yes 0.240 0.023 14.20%
No -0.240 0.023
Cooperation with other Individual assignments 0.004 0.023 9.48%
students Group assignments -0.004 0.023
Flexibility for pre-exam Yes 0.021 0.023 7.69%
assignments No -0.021 0.023
Communication with Online (e-mail, chat, forums, etc) -0.120 0.023 9.96%
teacher Online and face-to-face 0.120 0.023
Method of assessment of knowledge J 00% of exam in e-learning environment 0.126 0.030
70% of pre-exam assignments in e-learning envir. 0.022 0.035 23.76%
30% of pre-exam assignments in e-learning envir. -0.148 0.035
Constant 3.165 0.024
4.3. Preferences of traditional vs. online students
To determine if there are differences in preferences of traditional and online students, an analysis was done separately for these two predefined segments. Looking at the aggregated and segment-level data shown in Figure 1, it can be noticed that they do not differ significantly when it comes to relative importance of attributes.
A deeper analysis of individual partworths showed heterogeneity in student preferences, which led to a post-hoc cluster analysis using k-means procedure.
Customize environment based on preferences 25.00^-
—Traditional — Online
interactive work
Figure 1. Attribute importance scores for a priori defined segments Due to the limited sample size, two- and three-cluster solutions were analyzed. The 3-cluster solution was rejected because of small cluster (n < 10) that could not be statistically reliable. Accordingly, a 2-cluster solution was chosen and statistical significance of solution was confirmed by ANOVA (Table 4).
Table 4. Part-worth utilities for identified segments
Parr-worth utlity
Attribute Attribute ieve! Segment 1 Segment 2 Sig.
Customize environment Yes 0.101 0.109 0.825
based on preferences No -0.101 -0.109 0.825
Simple environment, no training necessary 0.201 0.145 0.219
Simplicity Environment requires training or previous experience -0.201 -0.145 0.219
Lectures Recorded lecture wtth slides and sound 0.185 -0.061 0.000
Classroom live broadcasting -0.185 0.061 0.000
Interactive work Yes 0.215 0.264 0.341
No -0.215 -0.264 0.341
Cooperation with other Individual assignments 0.019 0.026 0 361
students Group assignments 0.019 -0.026 0.361
Flexibility for pre-exam Yes 0.076 -0.030 0.002
assignments No -0.076 0.030 0.002
Communication with. Online (e-mail chat, forums, etc.) -0.097 -0.141 0.297
teacher Online and face-to-face 0.097 0.141 0.297
100% of exam in e-learning environment 0.480 -0.199 0.000
Method of assessment of 70% of pre-exam assignments in e-learning en- 0.067 -0.019 0.169
knowledge vir.
30% of pre-exam assignments in e-leaming en-vir. -0.547 0.218 0.000
Table 4 shows significant difference in preferences of isolated clusters according to the three attributes: lectures, flexibility for pre-exam assignments and method of knowledge assessment. When it comes to these three attributes, cluster members prefer their opposite levels. Although there
is no statistically significant difference in preferences for the attribute of interactive work, it is important for both segments and should be taken into account.
A comparative overview of attributes importance by segments and within total sample is presented in Figure 2.
Figure 2. Importance rating (%) of e-learning attributes for post hoc defined segments and whole sample
It is noticeable from figure 2 there is key difference between two identified segments in attributes method of assessment of knowledge and interactive work. These are at the same time most important ones in one segment and second important in opposite segment. Significant difference can be seen with attributes lectures and communication with teacher. First segment gives priority to lectures while second one has higher regard for communication. Two least important attributes for both segments are cooperation with other students and flexibility for preexam assignments (clear deadlines or flexible deadlines). These results differ from everyday teaching practices, as flexible deadlines are usually the cause for non-compliance with preexam assignments and exam failures. Looking at this from the perspective of teachers, clear deadlines are important attribute.
The profile of students in two segments was created based on observed differences in preferences. First segment consists of students oriented on results while the second segment is oriented on process (Figure 3).
Figure 3. Typology of students based on their learning preferences
Segment of students "results oriented" includes 55 students (48%). Most important attribute for them is method of knowledge assessment (36.48%) and they prefer 100% of pre-exam assignments and exam in e-learning environment. Interactive work, simplicity and lectures are approximately of similar importance, but their significance is half less in comparison with examination method (from 13% to 15%). Least important attributes are cooperation and flexibility. This segment also prefers group assignments and recorded lectures with slides/sound and clear assignment deadlines. Students in this cluster can be profiled more in detail: they are predominantly in their final year of studies, studying in traditional mode (44 out of 55 are traditional students, only 11 are online students). Most students in this cluster gave online courses grades 4 or 5.
Segment of students "process oriented" consists of 60 students (52%). Most important attribute is interactive work (26.77%), followed by method of knowledge assessment. They differ from first segment as they prefer 70% of pre-exam assignments in e-learning environment and 30% offline exam. Looking at previous simplicity and communication they are in agreement with first segment both in levels and importance of these two attributes. Although, this group of students prefer flexible deadlines and classroom live broadcasting. They can be profiled in detail as: students of junior years of study in e-learning
environment (34 out of 60).
5. DISCUSSION AND CONCLUSIONS
The findings of this study based on conjoint analysis represent the first empirical insights into students' preferences for an e-learning environment in Serbia. Three dimensions of interest were considered in this study: e-learning technology, teaching method and knowledge assessment. The last one was determined to be the most important one for both traditional and online students, but further analysis showed that two clusters could be identified based on their preferences not just on knowledge assessment but teaching method as well. Potential limitation of this research in data analysis is in students' individual perceptions of learning environments. They perceive that environment in which they have less experience is better, meaning that student in traditional face-to-face system grade much higher e-leaning systems.
The importance of different components of e-learning systems as a part of an e-learning environment was identified in previous studies. Matsatsinis, Grigoroudis and Delias (2003) identified three satisfaction criteria: interface, content and functionality. Damnjanovic, Jednak and Mijatovic (2015) measured the effectiveness of using Moodle by considering eight factors: intention to future use, communicativeness, format, information quality, performance outcome, perceived usefulness, satisfaction and system quality. Authors find out that communicativeness influences performance outcome the most, while the system and the quality of information have no effect on satisfaction.
Based on our results, it can be concluded that student preferences are important part for designing e-learning system, while student satisfaction is the key factor determining their persistence for studying in e-learning environment. As Malik, (2009) found, the main points that lead to student satisfaction with e-learning are the students themselves, but also the teachers and technological factors.
It would be ideal to identify and classify students in two profiles (result oriented or process oriented) at the beginning of their studies and adjust the e-learning environment accordingly. Students with more experience with online environment are predominantly oriented on process, including different activities for knowledge acquisition. Our
assumption is that system should be designed in a manner that offers diverse learning activities. Classroom live broadcasting, interactive work, and online and face-to-face communication with teacher are appropriate for this group. Classroom live broadcastings are expensive solution so it won't be possible for many universities to implement them for some time. But looking at the proactive practice at Harvard university it is something that every contemporarily university should stream to.
Communication with peers and teachers is the essence of interactive work. Nagel, Blignaut and Cronje (2009) concluded that only students who contribute to class discussion or interact with fellow students and the facilitator complete the course successfully. Students in online learning environments appreciate the role of the teacher as a facilitator of discussion, with the function of promoting student-to-student interactions, more than students in blended learning environments. In general, in blended learning environments instructors have more opportunities to interact with students and give them feedback during personal contact (Hung and Chou, 2015). General recommendation is that application of available communication tools in e-learning environment makes it more similar to face-to-face communication. Messengers, Skype or visual/sound records of frequently asked questions and their broadcasting can secure more comprehensive feedback to students. Also, various web tools, such as blogs, social networks (Facebook, Twitter, LinkedIn, etc.) can be used for educational purposes and to enhance student interactive work (Vaughan, Nickle, Silovs and Zimmer, 2011). This is supported by the fact that teachers have been shown to be willing and competent to use Web 2.0 for educational purposes (Jimoyiannis, Tsiotakis, Roussinos and Siorenta, 2013). Additionally, it has been shown that students and teachers view the value of using and integrating wiki into teaching and learning as positive especially for facilitating collaboration and interaction (Li, 2015).
Other cluster of students (results oriented) emphasizes knowledge assessment as most important component of e-learning environment. Although our results show they prefer 100% of knowledge assessment in online environment, we believe that this is not the key point. The key point is that the importance of second highest rated attribute is half less then knowledge assessment. General recommendation is that knowledge
assessment should be designed specifically with no regard to formative or summative assessment. Assessment could be seen as a feedback to students on their learning process as well as the official recognition of their accomplishments, achievements and final grades. Modern e-learning environment takes approach to assessment that is learner centered. Technology should enhance assessment and feedback practices rather than replace highly valued strategies such as face-to-face tutorials. Technology should be used to enhance assessment: instant feedback on test results using interactive online tests, using forums and blogs for communication, contemporary web 2.0 technologies for peer assessment and processing large groups of students.
If we look at students as our customers, these are attributes that should be addressed in order to retain them. FON does not implement them and should revise e-learning environment accordingly. Findings of our research may be useful in directing future research related to key factors essential to the adoption and effective implementation of e-learning environment, and provide a guideline for university policy makers in redesigning online mode of studying.
ACKNOWLEDGEMENTS
The authors are grateful to all participants in the research and to the editorial team for their kind and professional assistance. The above research was partially supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia [grant number TR33044].
Conflict of interests
The authors declare no conflict of interest.
REFERENCES
Acharya, B., & Lee, J. (2018). Users' perspective on the adoption of e-learning in developing countries: The case of Nepal with a conjointbased discrete choice approach. Telematics and Informatics, 35(6), 1733-1743. https://doi.org/10.1016/j. tele.2018.05.002 Azarcon Jr, D. E., Gallardo, C. D., Anacin, C. G., & Velasco, E. (2014). Attrition and retention in higher education institution: A conjoint analysis of consumer behavior in higher education. Asia Pacific Journal of Education, Arts and Sciences, 1(5), 107-118. Retrieved from http:// apjeas.apjmr.com/wp-content /uploads/2014/11/ APJEAS-2014-1-091.pdf
Black, E. W., Ferdig, R. E., & DiPietro, M. (2008). An overview of evaluative instrumentation for virtual high schools. The Amer. Jrnl. of Distance Education, 22(1), 24-45. https://doi. org/10.1080/08923640701713422 Bouhnik, D., & Marcus, T. (2006). Interaction in distance-learning courses. Journal of the American Society for Information Science and Technology, 57(3), 299-305. https://doi. org/10.1002/asi.20277 Carey, J. M., Carman, K. R., Clayton, K. P., Horiuchi, Y., Htun, M., & Ortiz, B. (2018). Who wants to hire a more diverse faculty? A conjoint analysis of faculty and student preferences for gender and racial/ethnic diversity. Politics, Groups, and Identities, 1-19. https://doi.org/10.1080/215655 03.2018.1491866 Conrad, D. (2008). Situating Prior learning assessment and recognition (Plar) in an online learning environment. The theory and practice of online learning, 75. Retrieved from http://biblioteca. ucv.cl/site/colecciones/manuales_u/99Z_ Anderson_2008-Theory_and_Practice_of_ Online_Learning.pdf#page=87 Daghan, G., & Akkoyunlu, B. (2012). An examination through conjoint analysis of the preferences of students concerning online learning environments according to their learning styles. International Education Studies, 5(4), 122-138. https://doi.org/10.5539/ies.v5n4p122 Damnjanovic, V., Jednak, S., & Mijatovic, I. (2015). Factors affecting the effectiveness and use of Moodle: students' perception. Interactive learning environments, 23(4), 496-514. https:// doi.org/10.1080/10494820.2013.789062 Drennan, J., Kennedy, J., & Pisarski, A. (2005). Factors affecting student attitudes toward flexible online learning in management education. The Journal of Educational Research, 98(6), 331-338. https:// doi.org/10.3200/J0ER.98.6.331-338 Ferguson, J. M., & DeFelice, A. E. (2010). Length of online course and student satisfaction, perceived learning, and academic performance. The International Review of Research in Open and Distributed Learning, 11(2), 73-84. https://doi. org/10.19173/irrodl.v11i2.772 Green, P. E., & Rao, V. R. (1971). Conjoint measurement-for quantifying judgmental data. Journal of Marketing research, 8(3), 355-363. https://doi.org/10.1177/002224377100800312 Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student perception of Moodle learning management system: a satisfaction and significance analysis. Interactive Learning Environments, 23(4), 515-527. https://doi.org/1 0.1080/10494820.2013.788033 Hung, M. L., & Chou, C. (2015). Students' perceptions of instructors' roles in blended and online learning environments: A comparative study. Computers & Education, 81, 315-325. https:// doi.org/10.1016/j.compedu.2014.10.022 Hur, J. S., & Pak, R. J. (2007). Conjoint analysis for the preferred subjects of elementary school computer education. Journal of the Korean Data and Information Science Society, 18(2), 357-364. http://www.koreascience.or.kr/article/ JAK0200723421065864.page Jimoyiannis, A., Tsiotakis, P., Roussinos, D., & Siorenta, A. (2013). Preparing teachers to integrate Web 2.0 in school practice: Toward
a framework for Pedagogy 2.0. Australasian Journal of Educational Technology, 29(2). 248267. https://doi.org/10.14742/ajet.157 Jung, I., Choi, S., Lim, C., & Leem, J. (2002). Effects of different types of interaction on learning achievement, satisfaction and participation in web-based instruction. Innovations in education and teaching international, 39(2), 153-162. https://doi.org/10.1080/14703290252934603 Katz, Y. J. (2002). Attitudes affecting college students' preferences for distance learning. Journal of computer assisted learning, 18(1), 2-9. https:// doi.org/10.1046/j.0266-4909.2001.00202.x Kuzmanovic, M., Savic, G., Popovic, M., & Martic, M. (2013). A new approach to evaluation of university teaching considering heterogeneity of students' preferences. Higher Education, 66(2), 153-171. https://doi.org/10.1007/s10734-012-9596-2
Li, K. M. (2015). Learning styles and perceptions of student teachers of computer-supported collaborative learning strategy using wikis. Australasian Journal ofEducational Technology, 31(1). https://doi.org/10.14742/ajet.521 Malarkodi, M., Indumathi, V. M., & Praveena, S. (2018). Preference Towards Online Mode of Distance Education Courses-conjoint Analysis. International Journal of Bioresource and Stress Management, 9(1), 178-182. https://doi. org/10.23910/IJBSM/2018.9.1.1858 Malik, M. W. (2009, November). Student satisfaction towards e-learning: influential role of key factors. In Comsats international business research conference (CBRC), 2nd. Retrieved from https://lahore.comsats.edu.pk/abrc2009/ Proceedings/All%20papers/STUDENT%20 S ATISFACTI0N%2 0T0 WARD S%2 0 E-LEARNING%20INFLUENTIAL%20 R0LE%200F%20KEY%20FACT0RS%20 (mahwish%20w.%20malik).pdf Matsatsinis, N. F., Grigoroudis, E., & Delias, P. (2003). User satisfaction and e-learning systems: Towards a multicriteria evaluation methodology. Operational Research, 3(3), 249-259. https:// doi.org/10.1007/BF02936404 Mijatovic, I., Cudanov, M., Jednak, S., & Kadijevich, D. M. (2013). How the usage of learning management systems influences student achievement. Teaching in Higher Education, 18(5), 506-517. https://doi.org/10.1080/135625 17.2012.753049 Nagel, L., Blignaut, A. S., & Cronjé, J. C. (2009). Read-only participants: A case for student communication in online classes. Interactive Learning Environments, 17(1), 37-51. https:// doi.org/10.1080/10494820701501028 Nummenmaa, M., & Nummenmaa, L. (2008). University students' emotions, interest and activities in a web-based learning environment. British Journal of Educational Psychology, 78(1), 163-178. https://doi.org/10.1348/000709907X203733 Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and methodologies. Computers & Education, 57(1), 1098-1108. https://doi.org/10.1016/jxompedu.2010.12.009 Pomales-Garcia, C., Liu, Y., & Lopez, Á. D. (2009, October). Student perceptions on the importance of distance learning module design dimensions. In 2009 39th IEEE Frontiers in Education Conference (pp. 1-5). IEEE. https://doi.
org/10.1109/FIE.2009.5350864 Popovic, M., Vagic, M., Kuzmanovic, M., & Andelkovic Labrovic, J. (2016). Understanding heterogeneity of students' preferences towards English medium instruction: A conjoint analysis approach. Yugoslav Journal of Operations Research, 26(1). 91-102. https://doi. org/10.2298/YJOR140915009P Roberts, R. M. (2010). The digital generation and web 2.0: E-learning concern or media myth?. In Handbook of Research on Practices and Outcomes in E-Learning: Issues and Trends (pp. 93-115). IGI Global. https://doi. org/10.4018/978-1-60566-788-1.ch006 Rosenberg, M. J., & Foshay, R. (2002). E-learning: Strategies for delivering knowledge in the digital age. Performance Improvement, 41(5), 50-51. https://doi.org/10.1002/pfi.4140410512 Sohn, S. Y., & Ju, Y. H. (2010). Conjoint analysis for recruiting high quality students for college education. Expert Systems with Applications, 37(5), 3777-3783. https://doi.org/10.1016/j. eswa.2009.11.043 Soutar, G. N., & Turner, J. P. (2002). Students' preferences for university: A conjoint analysis. International journal of educational management, 16(1), 40-45. https://doi. org/10.1108/09513540210415523 Stein, D. (2004). Student satisfaction depends on course
structure. Online Classroom, 2(1), 5. Sun, J., & Wang, Y. (2014). Tool choice for e-learning: Task-technology fit through media synchronicity. Information Systems Education Journal, 12(4), 17. https://isedj.org/2014-12/n4/ ISEDJv12n4p17.html Tashchian, R. O., & Freiden, J. B. (1983). Student Preferences for Class Scheduling: A Conjoint Analysis. Journal of Marketing Education, 5(3), 42-48. https://doi. org/10.1177/027347538300500309 Taylor, S. A., Humphreys, M., Singley, R., & Hunter, G. L. (2004). Business student preferences: Exploring the relative importance of web management in course design. Journal of Marketing Education, 26(1), 42-49. https://doi. org/10.1177/0273475303262352 Van Der Rhee, B., Verma, R., Plaschka, G. R., & Kickul, J. R. (2007). Technology readiness, learning goals, and eLearning: Searching for synergy. Decision Sciences Journal of Innovative Education, 5(1), 127-149. https://doi. org/10.1111/j.1540-4609.2007.00130.x Vaughan, N., Nickle, T., Silovs, J., & Zimmer, J. (2011). Moving To Their Own Beat: Exploring How Students Use Web 2.0 Technologies To Support Group Work Outside Of Class Time. Journal of Interactive Online Learning, 10(3). Retrieved from https://www.ncolr.org/jiol/ issues/pdf/10.3.1.pdf Walsh, S., & Cullinan, J. (2017). Factors Influencing Higher Education Institution Choice. In Economic Insights on Higher Education Policy in Ireland (pp. 81-108). Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-48553-9_4
Won, D., & Bravo, G. A. (2009). Course design in sport management education: Addressing students' perspectives through conjoint methodology.
Journal of Hospitality, Leisure, Sports and Tourism Education (Pre-2012), 8(2), 83-
96. Retrieved from https://s3.amazonaws. com/academia.edu.documents/44541893/ Course_design_in_sport_management_ educat20160408-4990-ie8pql.pdf?response-content-disposition=inline%3B%20 filename%3DCourse_design_in_sport_ management_educat.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential = AKIAIWOWYY GZ2Y53UL3A%2F20191204%2F us-east-1%2Fs3%2Faws4_request&X-Amz-Date = 20191204T0 8034 1Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=4eef2 62214aa3ea35f263a85042035484d63b9a8cf5e3 62cbee15420e43d1e04 Yu, C., Yu, W. C. W., & Lin, C. F. (2010). ComputerMediated Learning: What Have We Experienced and Where Do We Go Next?. In Handbook of research on practices and outcomes in e-learning: Issues and Trends (pp. 1-18). IGI Global. https://doi.org/10.4018/978-1-60566-788-1.ch001 Zwarts, T. M., Vanthournout, G., Gijbels, D., & Van den Bossche, P. (2015). Exploring conjoint analysis as a methodology to study implicit preferences within the context of educational and training sciences. Methodological challenges in research on student learning, 1, 77-91. Retrieved from https://hdl.handle. net/10067/1296470151162165141
